Mixed Reality Solutions

Mixed Reality technology and devices use a variety of sensors that simultaneously scan the physical environment and track the user's location. The data captured by these sensors is processed using complex algorithms that build an accurate 3D map of the environment along with the current position of the user or device.

This processing system also recognizes previously traveled positions and the environment the user is in, enabling more accurate navigation and richer interaction with the mapped environment — interfaces and interactions specifically designed to reshape the user's experience of reality. This capability has led to a wide range of practical applications for mixed reality technology.

SLAM, or Simultaneous Localization and Mapping, has long been used in applications like autonomous robots, self-driving cars, and drones. More recently, it's become a core part of Mixed Reality — SLAM is the backbone of MR applications, giving them the ability to integrate virtual objects within physical space and enable interaction from the user's own perspective.

SLAM is a computational optimization technique that maps an unknown environment while simultaneously localizing the user within it, even without any prior reference point. It relies on complex algorithms to deliver the information needed to build mixed reality solutions suited to real industry needs.

The most well-known devices delivering mixed reality applications and experiences are Microsoft HoloLens and Magic Leap One, both powered by spatial computing (SLAM) technologies. Nreal's Light MR goggles are another example, alongside other headsets that are either under development or yet to be released commercially.

An MR device uses various sensors to track both the environment and the user or device in real time. Microsoft HoloLens, for example, includes several specialized sensors, each serving a specific function:

• Multiple optical (black & white) cameras — two on each side for peripheral sensing and spatial awareness via Visual SLAM
• A Holographic Processing Unit to render holograms onto the see-through glasses within the environment
• A down-facing IR depth camera (Kinect sensor) to pick up hand movements and gesture interactions
• An RGB camera for capturing HD video of the surrounding environment
• An array of specialized speakers, microphones, and ambient light sensors

An internet connection isn't strictly necessary to use an MR device or run MR applications. It may be required for initial device setup, but beyond that, the device can function independently to perform mapping and interactions. This does depend on the specific application — a Digital Twin use case that requires live streaming of IIoT data, or an application running from the cloud, would still require connectivity.

Augmented reality is essentially the overlay of digitally created content onto the existing environment, with no real interaction between physical and digital objects. AR offers minimal immersion by comparison.

Mixed reality, on the other hand, enables digital content to actually interact with the environment using SLAM technology, delivering a significantly more enhanced and interactive experience than AR alone. With recent advancements like Google's ARCore and Apple's ARKit, the line between AR and MR applications continues to blur — though a fully immersive experience is still only achieved with a dedicated virtual reality device.

While the underlying computational technology — SLAM and sensor implementations — is largely similar between the two, it could be argued they deliver a comparable experience. As these technologies evolved and competition for adoption increased, different companies began describing them differently: Microsoft currently calls its technology Mixed Reality, while Magic Leap refers to its own as Augmented Reality.

Mixed Reality and Virtual Reality are fundamentally different technologies that deliver distinct experiences, so a direct "which is better" comparison isn't especially useful.

Virtual Reality requires a head-mounted device that completely covers the user's vision, fully disconnecting them from the real world. The user is transported into a digitally created virtual environment where they navigate and interact using gestures and hand controllers. Because the headset blocks the user's view of their actual surroundings, VR use should be confined to a safe, enclosed room — unrestricted movement could risk harm since the user can't see the real environment around them.

With Mixed Reality, the user can still see the real world, with digitally overlaid content that can genuinely interact with their surroundings. Users remain connected to and can move safely through their environment while experiencing digital content integrated into it. MR is immersive in its own distinct way and relies on a different kind of headset that lets users view both the real and digital environments seamlessly — making the use cases for VR and MR meaningfully different.

See-through Mixed Reality uses transparent optics to let users view the real world with digital content superimposed directly on the headset's see-through visor.

Pass-through Mixed Reality, by contrast, uses cameras to capture a live video feed of the environment, rendered on an LED display with digital content overlaid on top. Pass-through MR systems generally offer a broader field of view than see-through devices — the Oculus Quest 2, for instance, uses its black & white tracking camera sensors to deliver this kind of pass-through MR experience.

The Windows Mixed Reality (WMR) headset is essentially a Virtual Reality headset, while HoloLens is a true Mixed Reality (MR) headset. WMR fully encases the eyes and cuts off visual access to the real world, whereas HoloLens allows continuous viewing of the outside world alongside a digitally augmented, interactive experience.

Mixed Reality applications span many industries today, and most can gain significant advantage — provided the right use cases are identified and appropriate solutions are developed around them.

Manufacturing and medical industries stand to benefit the most, with applications across design, operations, maintenance, training, safety, and reliability. These same areas are common across industries such as oil & gas, petrochemicals, mining, nuclear, utilities, automotive, pharma, healthcare, real estate, and electronics. MR delivers stronger remote collaboration, increased productivity, better decisions, improved efficiency, cost savings, and greater customer satisfaction.

Mixed Reality is used across many enterprise applications today. Design teams, for example, can visualize design elements in a target environment or condition and remotely collaborate on design reviews in real time, dramatically cutting the time and cost of the overall design process and bringing products to market faster.

MR applications also support the overlay of critical information that helps execute key tasks by relaying real-time guidance from experts outside the immediate workspace. This remote expert support drives higher success rates and better outcomes for the enterprise, and is especially valuable in remote locations, hospitals, critical facilities, and academic institutions.

Medical professionals can train on physical examinations and surgery using virtual models with MR headsets, avoiding the logistical challenges tied to cadavers — training that can be delivered anywhere, anytime.

In the automotive industry, mixed reality solutions let customers visualize a car's ergonomics, interior, colors, and accessories, helping buyers decide on their preferred customization before making a purchase. The use cases for MR are essentially limitless, shaped entirely by how enterprises choose to deploy it.

Microsoft Mesh is an innovative technology that blends virtual and real worlds to enable better collaboration across geographies and more effective teamwork. Humans tend to do their best work through face-to-face interaction and collaboration.

With work being outsourced globally and the COVID-19 pandemic accelerating remote work, Microsoft Mesh enables virtual avatars — and eventually holographic representations of individuals — to be "holoported" into shared virtual workspaces for better collaborative outcomes. HoloLens-based mixed reality applications using Microsoft Mesh enable participation in this MR space for multi-user remote collaboration.

According to Microsoft, the HoloLens 2 Industrial Edition is considered Intrinsically Safe, suitable for use in Class I, Division 2, Groups A, B, C, and D hazardous locations, since it doesn't generate enough energy to ignite a hazardous mixture — enabling mixed reality applications in areas where flammable materials may be present.

AR devices such as the RealWear HMT-1Z1 are also certified Intrinsically Safe (IS), designed specifically for hazardous environments and known for their ruggedness. Hazardous areas like these are defined by the Class/Division or Zone system, with IS devices identified accordingly.

Virtual Reality is fully immersive, blocking the user from the outside world with a field of view around 110 degrees, while mixed reality overlays interactive virtual elements onto the physical environment and offers a narrower field of view (around 55 degrees) — feeling comparatively less immersive. Since MR keeps the surrounding environment in view, it can also introduce distractions that affect the immersive quality of the experience.

The visual quality of a synthetic VR environment doesn't fully replicate the natural environment seen through MR goggles, but VR can be made highly realistic depending on user needs. VR's visual quality also tends to be more consistent than MR graphics, which can be affected by variable ambient lighting conditions.

Outdoor lighting varies with sun and cloud cover, which affects the MR experience in a few ways. Using Mixed Reality glasses in bright outdoor conditions will degrade the visual quality of rendered graphics, since the bright ambient light washes them out.

Under these bright conditions, the black & white tracking cameras used for Visual SLAM can also become saturated, limiting what they're able to detect. Conversely, in low-light conditions, these cameras struggle to pick up enough environmental detail, causing instability in mapping and tracking. An ambient lighting level of roughly 500–1000 lux is generally considered optimal for mixed reality devices.

The biggest benefit of Mixed Reality is its ability to merge the digital and real worlds, enabling the visualization of new realities, data, and information that drive improved performance, productivity, and ROI.

For example, engineers can collaborate on design reviews of virtual prototypes and models directly on the shop floor, or customers can furnish and decorate a new home by trying out different options before making a purchase. Mixed Reality solutions are deployed across manufacturing, product design and development, training, marketing, maintenance, reliability, safety, healthcare, customer service, and more.

Some of the practical challenges affecting MR implementation are organizational rather than technological — and generally not difficult for companies to overcome.

One of the first steps toward successful MR implementation is having an internal champion, typically a senior executive, who understands the technology's benefits in the context of the organization's vision and long-term goals. A change agent like this helps explain the purpose and use of the technology to leadership and stakeholders, securing buy-in and the budgetary support needed for project implementation.

Companies should also align budgetary cycles with project cycles to ensure proper capital allocation, and should aim to progress steadily rather than rushing for results. Technology adoption and maturity take time, and sustained effort from all internal stakeholders typically leads to better outcomes with mixed reality technologies.

The rollout of 5G mobile networks has a positive impact on mixed reality solutions. 5G delivers both higher connectivity speeds and lower latency — both essential for a seamless MR experience.

Higher 5G bandwidth enables faster processing and transmission of the graphics workload needed for higher quality visuals and frame rates, while lower latency helps eliminate lag in tracking head and hand gestures or movement.

These advantages support wider adoption of mixed reality solutions, freeing them from local processing constraints and enabling cloud-hosted applications and data. As 5G rollouts continue worldwide, developers and solution providers are accelerating efforts to build appropriate use cases and solutions for companies globally.

Mixed Reality is set to transform how we work, play, and live in the 21st century, with the technology's rapid advances over the past few years demonstrating real transformative potential. The COVID-19 pandemic accelerated MR adoption, as remote working, collaboration, and assistance became essential to maintaining business continuity. As accessibility increases and costs decrease, adoption is expected to grow further across industrial and commercial applications, with MR finding expanding use in education, research, industry, safety, and entertainment.