Extended reality: building interactive worlds

Explore virtual, augmented, and mixed reality through the systems that make them work: tracking, spatial mapping, rendering, interaction, comfort, accessibility, and responsible design.

What extended reality includes

Extended reality (XR) is an umbrella term for technologies that combine digital content with a user's experience of space. Virtual reality (VR) usually presents a computer-generated environment that replaces much of the physical view. Augmented reality (AR) adds digital content to a view of the real world, often through a phone or transparent display. Mixed reality (MR) commonly refers to digital objects that are anchored to and interact with a mapped physical environment.

These terms are used differently across products and industries. It is often more useful to describe what the user sees, how the system senses the environment, and how digital content responds than to argue about a category label.

XR systems may use a head-mounted display, phone, tablet, projection, or other wearable. A headset is not required for every AR experience. Devices differ in field of view, resolution, brightness, weight, battery life, sensing, and whether the physical world is viewed directly or through cameras.

The XR system pipeline

An XR application repeatedly senses motion and environment, estimates the device’s position and orientation, updates a virtual scene, renders images, and presents them. When a user moves, the application must update the view quickly enough that digital content appears stable relative to the world.

The system may combine cameras, inertial sensors, depth sensors, microphones, controllers, and eye tracking. Sensor fusion estimates pose—position and orientation—while managing uncertainty and sensor noise. A spatial map can describe surfaces, walls, or other features to help place content.

Applications need a stable coordinate system. A world-anchored object should remain in the same place as the user moves; a head-locked interface element should move with the viewer. Confusing these reference frames can make an object appear to drift or make a user interface difficult to reach.

Worked example: placing a virtual instruction panel

1. The device detects visual features and estimates its position in the room.
2. A user selects a safe surface and places the panel at a chosen location.
3. The application records an anchor relative to the room map.
4. As the user moves, tracking updates the view and rendering keeps the panel aligned.
5. If tracking confidence falls, the application indicates uncertainty and avoids pretending the anchor is exact.

Real environments change: lighting shifts, furniture moves, reflective surfaces confuse sensors, and maps may be lost. Apps should communicate when spatial understanding is limited and provide a clear recovery action.

Tracking and spatial mapping

Inside-out tracking uses sensors on the device to estimate movement relative to the environment. Outside-in tracking uses external cameras or beacons to observe the device. Systems may combine both approaches. Tracking quality depends on sensor calibration, lighting, texture, motion, and the available field of view.

Simultaneous localization and mapping (SLAM) describes estimating a moving device's pose while building or updating a map of its surroundings. The map may be incomplete and its coordinate frame can change. An application should not treat estimated geometry as a perfect survey of a room.

Occlusion is the visual effect where a nearer object hides a farther one. To make a virtual object appear behind a real table, the system needs a representation of the table or depth data. Bad geometry or delayed updates can cause digital objects to visibly overlap physical objects in an implausible way.

Spatial anchors can persist objects in a room or across devices, but persistence can raise privacy concerns and depend on a vendor’s mapping service. Decide whether anchors are local, shared, or cloud-backed and how a user can remove them.

Rendering and performance

XR rendering has tight timing constraints because the user sees and moves through the rendered scene continuously. High resolution, complex lighting, visual effects, and detailed geometry all consume resources. If frames arrive late or inconsistently, the experience can feel unstable or uncomfortable.

Stereoscopic displays render a slightly different view for each eye to create depth perception. Correct projection, eye position, and display calibration matter. Some systems use techniques such as reprojection to adjust a recently rendered frame based on updated head motion; this can reduce apparent delay, but it does not fix every performance problem.

Foveated rendering can devote more rendering detail to the region where the user is looking, potentially reducing work. It depends on gaze sensing or a fixed approximation and introduces privacy and calibration considerations. Optimization should target the actual device and include thermal and battery behavior over longer sessions.

Developers should profile frame timing, tracking stability, memory use, and thermal throttling on target hardware. A scene that runs smoothly on a desktop development machine may not meet the limits of a mobile headset.

Interaction and user experience

XR interaction can use hand gestures, controllers, gaze, voice, touch, physical movement, or a combination. Each has strengths and limitations. A gesture that looks intuitive may be tiring when repeated; voice can be useful hands-free but unsuitable in a noisy or private setting; gaze can aid selection but should not be confused with consent.

Make interactive targets easy to find and select. Provide clear feedback when an action is recognized, explain how to cancel or undo, and avoid requiring precise movement when it is not essential. Important controls should have alternatives for people who cannot use a particular gesture or controller.

Spatial sound can communicate direction and distance, but should not replace visual or text cues for important information. Persistent interface elements should be placed at comfortable distances and orientations rather than forcing prolonged awkward posture.

Good XR design starts with a task, not a headset feature. Ask whether spatial placement or embodied interaction improves understanding or performance compared with a conventional screen. If a normal interface is clearer, faster, and more accessible, XR may not be the best medium.

Applications and design process

XR is explored for training, design review, education, remote assistance, visualization, entertainment, and guided maintenance. A spatial model can help people inspect scale and relationships that are difficult to understand in a flat drawing. But an immersive presentation does not guarantee better learning or fewer errors; evaluate outcomes against a suitable alternative.

Prototype the interaction before investing in detailed assets. Test the task with representative users, observe where they look and move, and measure completion time, error, and comfort. Include users with different heights, mobility, vision, hearing, and prior experience.

For collaborative XR, establish how avatars, shared anchors, permissions, and communication work. A person should be able to understand who is present and what spatial information is shared. Remote collaborators may have different devices and tracking accuracy, so do not imply that every participant sees a perfectly identical scene.

Comfort, accessibility, and safety

Some users experience eyestrain, fatigue, disorientation, or motion sickness. Mismatches between visual motion and physical movement can contribute to discomfort. Keep movement intentional, avoid unnecessary acceleration, provide comfort options, and let users pause or exit easily. Individual tolerance varies, so do not assume one setting works for everyone.

Headsets can block vision or limit awareness of nearby people and objects. Provide safe boundaries, encourage clear play areas, and make transitions between virtual and physical contexts understandable. Do not design an experience that pressures people to move quickly through an unsafe space.

Accessibility includes captions, readable text, contrast, screen-reader support where possible, seated and standing options, one-handed interaction, and non-gesture alternatives. Avoid relying only on color, spatial audio, or physical movement to communicate essential information.

Design for breaks and practical session lengths. Account for glasses, fit, hygiene, controllers, battery limits, and the effort of wearing a device. Comfort and usability are system requirements, not finishing touches.

Privacy and responsible deployment

XR devices can capture room images, voices, movement, gaze, and spatial maps. These data may reveal sensitive information about a person's home, workplace, health, or attention. Collect only what the experience needs, process locally when appropriate, and make sensing and sharing visible and understandable.

Spatial data can include bystanders who did not choose to use the device. Consider camera indicators, privacy zones, retention limits, and ways to avoid storing raw room imagery. Gaze data can be particularly sensitive; do not collect or infer more than necessary.

Shared XR spaces need clear controls for presence, recording, personal boundaries, blocking, and reporting. Organizations deploying XR for work or education should explain what is monitored and avoid treating device telemetry as a complete measure of attention or performance.

Before deployment, review the physical environment, data flows, update process, access controls, and user support plan. Test failure states such as tracking loss, network outage, low battery, or accidental boundary crossing.

Compact glossary

XR

Extended reality, an umbrella term for virtual, augmented, and mixed experiences.

Pose

The position and orientation of a device or object.

SLAM

Simultaneous localization and mapping: estimating pose while building a map.

Spatial anchor

A reference used to place digital content consistently in a physical space.

Occlusion

The visual hiding of a farther object by a nearer one.

Foveated rendering

Rendering technique that varies detail based on the viewer’s gaze or an estimated focus region.

References and further reading

1. World Wide Web Consortium. Immersive Web Developer Home. Standards and resources.
2. W3C. WebXR Device API. Specification.
3. National Institute of Standards and Technology. Virtual and Augmented Reality. Research overview.
4. XR Association. Developing Safety Standards for XR. Industry resources.

Try it yourself

1. Compare VR, AR, and MR by describing what the user sees and how the environment is sensed.
2. Sketch the processing loop from movement sensing to a newly rendered frame.
3. Design an alternative interaction for a user who cannot perform a hand gesture.
4. List room or personal data an XR headset might collect and how to minimize retention.
5. Propose a test comparing an XR training activity with a non-XR alternative.

Remember: compelling XR depends on a fast, reliable feedback loop and human-centered design. Spatial computing is most useful when being spatial genuinely helps the task.