What Is Mixed Reality?

Mixed reality is a technology paradigm in which physical and digital objects coexist and interact within a unified environment, spanning the spectrum of possible combinations between the fully real and the fully virtual. The concept was formally defined in 1994 by Paul Milgram and Fumio Kishino, who introduced the reality-virtuality (RV) continuum: a theoretical line with the unmediated physical world at one end and a completely computer-generated virtual environment at the other. Everything in between constitutes mixed reality, including both augmented reality, where virtual content is overlaid onto a real environment, and augmented virtuality, where real objects or video feeds are incorporated into a predominantly virtual scene.

Mixed reality draws on computer vision, real-time 3D rendering, spatial tracking, display engineering, and human-computer interaction. It requires systems capable of understanding the geometry and content of a physical space, placing digital elements consistently within that space, and updating both the rendering and the user's perception as the user and physical objects move. The engineering challenge is to maintain spatial coherence, where digital objects appear anchored to real locations, at frame rates high enough to avoid perceptual artifacts.

The Reality-Virtuality Continuum

Milgram and Kishino's taxonomy of mixed reality visual displays organized systems by three dimensions: extent of world knowledge (how much the system knows about the physical scene), reproduction fidelity (how accurately the display represents either domain), and extent of presence (how much the user feels immersed in the virtual versus physical world). This framework remains influential in research on display design and interaction, because it explains why a head-up display that shows navigation arrows on a windshield and a fully enclosed VR headset represent very different engineering and experiential problems despite both being described loosely as immersive.

Augmented virtuality, the less commonly implemented half of the continuum, places real-world elements such as live video of hands or physical props into a predominantly synthetic scene. It appears in surgical training, where a physical mannequin is embedded in a virtual operating environment, and in product design, where a physical prototype is examined within a simulated assembly context.

Augmented Reality within Mixed Reality

Augmented reality (AR), positioned closer to the real-world end of the continuum, overlays digital information on a live view of the physical environment. Displays include optical see-through headsets, video see-through headsets, and smartphone or tablet screens that use the camera as a live viewport. The central technical requirement is accurate tracking: the system must determine the pose of the user's head or device in real time with low latency, so that virtual objects remain registered to fixed physical locations even as the user moves.

Research on augmented and mixed reality in medical imaging demonstrates how AR's preservation of the physical environment reduces disorientation compared to full VR, making it valuable in surgical guidance, where the clinician must maintain direct awareness of the patient.

Multimedia Systems and Rendering

Mixed reality systems process and synchronize multiple media streams: real-time camera video, spatial sensor data, rendered 3D graphics, spatial audio, and haptic feedback. Multimedia integration is required to maintain perceptual consistency across these channels. Latency mismatches between video capture and rendering, for example, cause virtual objects to appear to lag behind physical reality, breaking the illusion of co-presence. Display refresh rates of 90 Hz or higher, combined with motion-to-photon latency below 20 milliseconds, are engineering targets for consumer mixed reality headsets. An IEEE conference survey on VR, AR, and MR applications and present scenarios reviews the technical requirements and application domains that are driving hardware and software development in the field.

Applications

Mixed reality systems are being deployed across a growing range of professional and consumer contexts, including:

• Surgical planning, guidance, and medical education
• Industrial maintenance, remote assistance, and training simulation
• Architectural visualization and construction site coordination
• Gaming, entertainment, and social virtual environments
• Military simulation, training, and situational awareness overlays