What Is Augmented Virtuality?

Augmented virtuality (AV) is a class of mixed reality display in which a predominantly virtual environment is supplemented by elements drawn from the physical world. In augmented virtuality, the user's primary perceptual experience is of a synthetic, computer-generated scene; real-world objects, video streams, or sensor data are incorporated to enrich that scene rather than to serve as its foundation. This positioning distinguishes AV from augmented reality, where the physical world is primary and digital content is layered on top of it.

The term originates in the 1994 taxonomy of mixed reality visual displays proposed by Paul Milgram and Fumio Kishino. That work introduced the reality-virtuality (RV) continuum: a conceptual spectrum anchored at one end by purely real environments and at the other by purely virtual ones. Augmented reality occupies the portion of the continuum close to the real end, while augmented virtuality occupies the portion close to the virtual end. Together they constitute the broader category of mixed reality, which encompasses any display that blends real and virtual content to some degree.

Position on the Reality-Virtuality Continuum

Milgram and Kishino described the placement of a display on the RV continuum using three axes: extent of world knowledge (how much the system knows about the real environment), reproduction fidelity (the perceptual quality of the synthetic imagery), and extent of presence (the degree to which the user perceives the environment as real). Augmented virtuality systems tend to score high on the first two axes because they require real-world input data and must render it with enough fidelity to integrate plausibly into the virtual scene, while their extent of presence is high on the virtual side.

A practical example of augmented virtuality is a virtual reality environment that incorporates a live video feed of the user's hands, allowing natural hand interaction with virtual objects without requiring a fully synthetic hand representation. The real hands are composited into the virtual scene at the appropriate depth, maintaining spatial correspondence with the user's proprioception.

Technical Characteristics

Augmented virtuality systems present distinct technical requirements compared to augmented reality. The rendering pipeline must composite real-world image data into a primarily synthetic scene, which raises challenges of depth estimation, chroma alignment, and latency matching. If the video stream of a real object arrives even a few milliseconds later than the surrounding synthetic frame, visible discontinuities appear at the boundary between real and virtual content.

Calibration is another core requirement: the position and orientation of the camera capturing real-world content must be precisely registered to the coordinate frame of the virtual environment. Errors in this calibration produce perceptible misalignment, particularly when the user moves. Research on the reality-virtuality continuum has examined how these technical constraints vary across positions on the continuum and how modern hardware, such as inside-out tracking headsets with integrated passthrough cameras, has shifted the practical boundaries between AR and AV.

Relation to Mixed Reality Systems

The distinction between augmented virtuality and augmented reality has become less rigid as headset hardware has converged. Passthrough headsets, which use outward-facing cameras to present the real environment inside an otherwise virtual display, can be configured to operate anywhere from nearly opaque AR to full video passthrough of the real world, passing through the AV region in between. The ResearchGate publication of the original Milgram-Kishino taxonomy paper remains a primary reference for situating new display architectures within this framework.

Applications

Augmented virtuality has applications in a range of fields, including:

• Virtual reality training environments that incorporate live sensor or camera feeds
• Surgical simulation with real instrument tracking in a virtual anatomy model
• Teleoperation interfaces where remote camera footage is embedded in a synthetic site model
• Collaborative virtual workspaces that include video avatars of remote participants
• Entertainment systems combining live performance capture with virtual stage environments