Augmented virtuality

On this page 7

Augmented virtuality (AV) is a form of mixed reality in which the user experiences a predominantly virtual environment that is enriched with live elements captured from the real world, such as the user’s own hands, a physical keyboard, a colleague’s video image or a tracked real object. In extended reality, the term marks the position on the reality–virtuality continuum that lies closer to full virtual reality than to augmented reality.

The concept matters because it describes a design choice rather than a device category. A head-mounted display can run augmented reality in one application and augmented virtuality in the next; what defines AV is that the virtual world is the frame and the real-world content is the inserted layer.

Where augmented virtuality sits on the reality–virtuality continuum

Augmented virtuality occupies the virtual half of the mixed reality span on the reality–virtuality continuum described by Paul Milgram and Fumio Kishino in 1994. At one end lies the purely real environment, at the other the purely virtual environment, and everything between them is mixed reality. Augmented reality places synthetic objects into a real scene; augmented virtuality does the opposite and places real content into a synthetic scene.

The boundary between the two is a matter of proportion and anchoring, not of hardware. The practical test is simple: if removing the real-world inserts would still leave a complete, coherent scene, the experience is augmented virtuality; if removing the virtual content would leave the ordinary physical room, it is augmented reality.

The following table places augmented virtuality among its neighbors on the continuum.

PositionDominant worldInserted contentTypical example
Real environmentPhysicalNoneOrdinary unaided vision
Augmented realityPhysicalVirtual objectsNavigation arrows on a street view
Augmented virtualityVirtualReal objects or peopleOwn hands visible inside a virtual cockpit
Virtual realityVirtualNoneFully rendered game world
Positions on the reality–virtuality continuum

How real content is brought into the virtual scene

Augmented virtuality depends on a capture, segmentation and compositing pipeline. Cameras or depth sensors capture the real element, software isolates it from its background, and the renderer places it in the virtual scene with correct position, scale and occlusion. Each stage adds latency, so the full pipeline must stay within a motion-to-photon budget of roughly 20 milliseconds or less to avoid visible lag and discomfort.

Developers choose among several insertion techniques depending on the real element, the available sensors and the required fidelity.

TechniqueHow it worksStrengthMain limitation
Passthrough cutoutShows camera video only inside a defined regionSimple and low latencyHard edges and limited depth cues
Depth-based segmentationUses depth data to separate near objectsCorrect occlusion of virtual objectsNoisy edges on hair and thin parts
Hand tracking with video overlayCombines tracked skeleton with camera textureNatural view of own handsFails under fast motion or poor light
Chroma keyingRemoves a uniform colored backgroundClean mattes in studio setupsRequires controlled lighting and backdrop
Volumetric captureReconstructs a person as 3D geometryViewable from any angleHigh bandwidth and processing cost
Common techniques for inserting real content

The most demanding problem is mutual occlusion: a real hand must hide a virtual lever when it passes in front of it, and the lever must hide the hand when the hand moves behind it. Without accurate per-pixel depth, inserted content appears to float on top of the scene, which breaks presence quickly.

Augmented virtuality compared with augmented reality and mixed reality

Augmented virtuality is a subset of mixed reality, while augmented reality is the other main subset. In everyday industry language, “mixed reality” is often used loosely for any blended experience, and “augmented virtuality” appears mostly in research, simulation and engineering contexts where the distinction has practical consequences.

AspectAugmented virtualityAugmented realityMixed reality (broad use)
Base layerRendered virtual worldPhysical surroundingsEither, depending on application
User awareness of roomLow, limited to insertsHighVariable
Display neededUsually opaque headsetOptical or video see-throughVideo see-through is common
Primary goalImmersion with real anchorsInformation in contextFlexible blending
Typical useSimulators and virtual meetingsMaintenance and navigationProductivity and gaming
Augmented virtuality and related terms

A frequent misconception treats augmented virtuality as an outdated synonym for mixed reality. It is more accurate to see it as a precise label for one direction of blending, useful whenever a team must decide whether the virtual scene or the physical room is the primary reference frame.

Practical uses and design decisions

Augmented virtuality is most valuable when users need full immersion plus a few reliable physical anchors. Flight and driving simulators show the trainee’s real hands and real controls inside a virtual cockpit, which preserves muscle memory and tactile feedback. Productivity applications let users see a physical keyboard and coffee cup while working in a virtual office. Telepresence systems place live video or volumetric images of remote participants into a shared virtual room.

Choosing which real elements to insert is a design decision with clear trade-offs, summarized below.

SituationRecommended insertReason
Training with physical controlsHands and control panelKeeps tactile skills transferable
Seated office work in VRKeyboard, mouse and desk edgeReduces typing errors and collisions
Room-scale gamingNearby obstacles and peopleImproves physical safety
Remote collaborationVideo or volumetric participantsConveys facial expression and gesture
Fully narrative experienceNo inserts or minimal handsProtects immersion
Decision guide for real-world inserts

Good practice is to insert only what the task requires. Each additional real element costs processing time, introduces lighting mismatches and reminds the user of the physical room, which can reduce the sense of presence that justifies a virtual environment in the first place.

Limitations and common failure modes

The main weaknesses of augmented virtuality come from the visual mismatch between captured and rendered content. Camera video carries real lighting, noise and lens distortion, while the virtual scene follows its own lighting model. Without color matching and relighting, inserted objects look pasted in.

  • Latency differences: camera inserts that lag behind rendered content by even a few frames feel detached from the user’s movement.
  • Segmentation errors: halos, missing fingers and flickering edges are the most visible artifacts.
  • Scale and depth errors: incorrect camera calibration makes real objects appear larger or closer than they are.
  • Privacy concerns: capturing the room and bystanders raises data protection questions, especially in shared workplaces.
  • Bandwidth for telepresence: volumetric streams of people require far more data than avatars.

Many of these problems shrink as headsets gain higher-resolution color passthrough and dedicated depth sensors, but none disappears entirely, and careful sensor calibration remains the foundation of convincing augmented virtuality.

A plain-language summary

Augmented virtuality means you are mostly inside a computer-made world, but a few real things are allowed in. You might wear a headset and sit in a virtual spaceship, yet still see your own hands and the real keyboard on your desk.

Think of it as the reverse of augmented reality. Augmented reality adds digital objects to your real room. Augmented virtuality adds pieces of your real room to a digital world. The key idea is which world is the main one.

An example from practice

A regional airline trains first officers on a compact cockpit simulator built around an opaque VR headset. The cockpit exterior, airport and weather are fully virtual, but the trainee sits at a physical replica of the throttle quadrant and overhead panel. Headset cameras and depth sensing segment the trainee’s hands and the physical panel, and the system composites them into the virtual cockpit at the correct depth.

During an engine failure scenario after takeoff, the trainee reaches for the real fuel control switch and sees their own hand do so in the headset. Instructors report that procedures learned this way carry over to the full-motion simulator with fewer hand-placement errors than in a version that showed only controller-driven virtual hands. The outcome is a cheaper training stage that still preserves the tactile habits pilots need.