Augmented virtuality (AV) is a form of mixed reality in which the user experiences a predominantly virtual environment that is enriched with live elements captured from the real world, such as the user’s own hands, a physical keyboard, a colleague’s video image or a tracked real object. In extended reality, the term marks the position on the reality–virtuality continuum that lies closer to full virtual reality than to augmented reality.
The concept matters because it describes a design choice rather than a device category. A head-mounted display can run augmented reality in one application and augmented virtuality in the next; what defines AV is that the virtual world is the frame and the real-world content is the inserted layer.
Where augmented virtuality sits on the reality–virtuality continuum
Augmented virtuality occupies the virtual half of the mixed reality span on the reality–virtuality continuum described by Paul Milgram and Fumio Kishino in 1994. At one end lies the purely real environment, at the other the purely virtual environment, and everything between them is mixed reality. Augmented reality places synthetic objects into a real scene; augmented virtuality does the opposite and places real content into a synthetic scene.
The boundary between the two is a matter of proportion and anchoring, not of hardware. The practical test is simple: if removing the real-world inserts would still leave a complete, coherent scene, the experience is augmented virtuality; if removing the virtual content would leave the ordinary physical room, it is augmented reality.
The following table places augmented virtuality among its neighbors on the continuum.
| Position | Dominant world | Inserted content | Typical example |
|---|---|---|---|
| Real environment | Physical | None | Ordinary unaided vision |
| Augmented reality | Physical | Virtual objects | Navigation arrows on a street view |
| Augmented virtuality | Virtual | Real objects or people | Own hands visible inside a virtual cockpit |
| Virtual reality | Virtual | None | Fully rendered game world |
How real content is brought into the virtual scene
Augmented virtuality depends on a capture, segmentation and compositing pipeline. Cameras or depth sensors capture the real element, software isolates it from its background, and the renderer places it in the virtual scene with correct position, scale and occlusion. Each stage adds latency, so the full pipeline must stay within a motion-to-photon budget of roughly 20 milliseconds or less to avoid visible lag and discomfort.
Developers choose among several insertion techniques depending on the real element, the available sensors and the required fidelity.
| Technique | How it works | Strength | Main limitation |
|---|---|---|---|
| Passthrough cutout | Shows camera video only inside a defined region | Simple and low latency | Hard edges and limited depth cues |
| Depth-based segmentation | Uses depth data to separate near objects | Correct occlusion of virtual objects | Noisy edges on hair and thin parts |
| Hand tracking with video overlay | Combines tracked skeleton with camera texture | Natural view of own hands | Fails under fast motion or poor light |
| Chroma keying | Removes a uniform colored background | Clean mattes in studio setups | Requires controlled lighting and backdrop |
| Volumetric capture | Reconstructs a person as 3D geometry | Viewable from any angle | High bandwidth and processing cost |
The most demanding problem is mutual occlusion: a real hand must hide a virtual lever when it passes in front of it, and the lever must hide the hand when the hand moves behind it. Without accurate per-pixel depth, inserted content appears to float on top of the scene, which breaks presence quickly.
Augmented virtuality compared with augmented reality and mixed reality
Augmented virtuality is a subset of mixed reality, while augmented reality is the other main subset. In everyday industry language, “mixed reality” is often used loosely for any blended experience, and “augmented virtuality” appears mostly in research, simulation and engineering contexts where the distinction has practical consequences.
| Aspect | Augmented virtuality | Augmented reality | Mixed reality (broad use) |
|---|---|---|---|
| Base layer | Rendered virtual world | Physical surroundings | Either, depending on application |
| User awareness of room | Low, limited to inserts | High | Variable |
| Display needed | Usually opaque headset | Optical or video see-through | Video see-through is common |
| Primary goal | Immersion with real anchors | Information in context | Flexible blending |
| Typical use | Simulators and virtual meetings | Maintenance and navigation | Productivity and gaming |
A frequent misconception treats augmented virtuality as an outdated synonym for mixed reality. It is more accurate to see it as a precise label for one direction of blending, useful whenever a team must decide whether the virtual scene or the physical room is the primary reference frame.
Practical uses and design decisions
Augmented virtuality is most valuable when users need full immersion plus a few reliable physical anchors. Flight and driving simulators show the trainee’s real hands and real controls inside a virtual cockpit, which preserves muscle memory and tactile feedback. Productivity applications let users see a physical keyboard and coffee cup while working in a virtual office. Telepresence systems place live video or volumetric images of remote participants into a shared virtual room.
Choosing which real elements to insert is a design decision with clear trade-offs, summarized below.
| Situation | Recommended insert | Reason |
|---|---|---|
| Training with physical controls | Hands and control panel | Keeps tactile skills transferable |
| Seated office work in VR | Keyboard, mouse and desk edge | Reduces typing errors and collisions |
| Room-scale gaming | Nearby obstacles and people | Improves physical safety |
| Remote collaboration | Video or volumetric participants | Conveys facial expression and gesture |
| Fully narrative experience | No inserts or minimal hands | Protects immersion |
Good practice is to insert only what the task requires. Each additional real element costs processing time, introduces lighting mismatches and reminds the user of the physical room, which can reduce the sense of presence that justifies a virtual environment in the first place.
Limitations and common failure modes
The main weaknesses of augmented virtuality come from the visual mismatch between captured and rendered content. Camera video carries real lighting, noise and lens distortion, while the virtual scene follows its own lighting model. Without color matching and relighting, inserted objects look pasted in.
- Latency differences: camera inserts that lag behind rendered content by even a few frames feel detached from the user’s movement.
- Segmentation errors: halos, missing fingers and flickering edges are the most visible artifacts.
- Scale and depth errors: incorrect camera calibration makes real objects appear larger or closer than they are.
- Privacy concerns: capturing the room and bystanders raises data protection questions, especially in shared workplaces.
- Bandwidth for telepresence: volumetric streams of people require far more data than avatars.
Many of these problems shrink as headsets gain higher-resolution color passthrough and dedicated depth sensors, but none disappears entirely, and careful sensor calibration remains the foundation of convincing augmented virtuality.
A plain-language summary
Augmented virtuality means you are mostly inside a computer-made world, but a few real things are allowed in. You might wear a headset and sit in a virtual spaceship, yet still see your own hands and the real keyboard on your desk.
Think of it as the reverse of augmented reality. Augmented reality adds digital objects to your real room. Augmented virtuality adds pieces of your real room to a digital world. The key idea is which world is the main one.
An example from practice
A regional airline trains first officers on a compact cockpit simulator built around an opaque VR headset. The cockpit exterior, airport and weather are fully virtual, but the trainee sits at a physical replica of the throttle quadrant and overhead panel. Headset cameras and depth sensing segment the trainee’s hands and the physical panel, and the system composites them into the virtual cockpit at the correct depth.
During an engine failure scenario after takeoff, the trainee reaches for the real fuel control switch and sees their own hand do so in the headset. Instructors report that procedures learned this way carry over to the full-motion simulator with fewer hand-placement errors than in a version that showed only controller-driven virtual hands. The outcome is a cheaper training stage that still preserves the tactile habits pilots need.