What Is Mediated Reality?

Mediated reality is a field of human-computer interaction concerned with technologies that modify a person's perception of the physical world through computational processing of sensory input, particularly vision. Rather than simply overlaying graphics onto an unchanged view of reality, as augmented reality typically does, mediated reality encompasses the full spectrum of perceptual modification: information can be added, removed, filtered, enhanced, or transformed in real time before it reaches the user. The concept was developed extensively by Steve Mann at the University of Toronto beginning in the late 1970s and early 1980s through wearable computerized-vision systems.

Mediated reality is formally a superset of augmented reality, virtual reality, and mixed reality. It differs from those categories by introducing a mediality dimension: the degree to which incoming sensory data has been altered before presentation. A standard augmented reality overlay leaves the underlying visual field unchanged; a mediated reality system may simultaneously enhance detail in shadowed areas and suppress glare from bright sources, producing a composite view that no unassisted human eye could achieve. This two-axis model, one axis for the reality-to-virtuality mix and a second for the degree of perceptual modification, is described in the taxonomy of mediated and multimediated reality proposed by Mann and colleagues.

Augmentation and Diminishment

The two primary operations in mediated reality are augmentation, adding information or perceptual detail that is absent from the raw sensory field, and diminishment, reducing or removing elements that are present. Augmentation can render infrared heat signatures visible, display textual overlays, or enhance low-light detail. Diminishment removes distracting elements: filtering advertisements from a field of view, suppressing glare from oncoming headlights, or masking visual clutter in a complex environment. These two operations can be combined continuously in a single system, as in HDR (high dynamic range) welding helmets that compress the overwhelming brightness of an electric arc while simultaneously revealing shadow detail the unprotected eye would miss. As Mann describes in IEEE Spectrum, this kind of perceptual improvement distinguishes mediated reality from simpler overlay approaches.

Wearable Display Systems

Mediated reality depends on a closed perceptual loop: cameras or sensors capture the environment, a processor applies transformations, and the modified view is presented to the user through a head-mounted or wearable display before the user's eyes receive it directly. Early implementations used bulky analog and then digital hardware; current systems use lightweight optics, fast image processors, and high-resolution microdisplays. Video see-through displays, which route all visual input through cameras rather than using transparent optics, give the system full control over every pixel the user sees, enabling the complete range of mediated reality operations. Optical see-through displays, used in many commercial augmented reality headsets, retain the direct optical path and therefore limit the system to additive operations only. The choice of display architecture determines which operations are physically possible for a given system.

Perception and Cognition

Research in mediated reality intersects with psychophysics and cognitive science, because altering sensory input affects what users see and also how they interpret and act in their environment. Prolonged use of systems that modify spatial relationships or contrast gradients can produce perceptual adaptation effects, which both complicate system design and open research questions about neural plasticity. Calibration and latency are critical variables: perceptual mismatch between a user's motion and the display's response produces disorientation that limits usability. Work on wearable computing published in IEEE Transactions on Systems, Man, and Cybernetics has examined these human-factors dimensions, noting that display pipeline latency below 20 milliseconds is generally required to avoid noticeable perceptual conflict.

Applications

Mediated reality has applications in a range of fields, including:

• Assistive technology for vision impairment, enhancing contrast and reducing glare
• Industrial and welding safety, controlling arc brightness while preserving situational awareness
• Surgical and medical visualization, enhancing tissue contrast during minimally invasive procedures
• Remote collaboration, sharing modified views between field workers and remote experts
• Training and simulation, augmenting physical environments with instructional overlays
• Security and surveillance, filtering or enhancing feeds for human or automated analysis