• VR headsets’ see-through technology can enable an early MR experience and also help develop the ecosystem necessary for MR applications.
• Depth perception is essential in MR applications to demonstrate the occlusion phenomena. The current quality of the VR VST is barely adequate as it is achieved through cameras. However, this can be improved by incorporating an additional ToF sensor.
• There will be significantly more VR headsets with VST going forward, promoting a rise in the number of MR applications.

Virtual reality (VR) headsets use real-time six degrees of freedom (DOF) and visual simultaneous localization and mapping (SLAM) to receive real-world information, such as 3D coordinates and object recognition. Initially, the technology was employed for the hand-tracking and Guardian system. Nonetheless, some VR headsets incorporate images from monochrome cameras and, unexpectedly, extend the capabilities of video see-through (VST) to mixed reality (MR). Meta has launched the Quest Pro VR headset, while ByteDance has the PICO4 with full-color cameras to provide MR applications that are more realistic.

The see-through technology used in current VR headsets is called VST, while the see-through technology used in augmented reality (AR) devices is called the optical see-through (OST). The VST technology blends the digitized three-dimensional view of the real world with a computer-generated simulation of reality, which is then shown on the opaque display. By using VST along with the appropriate rendering technique and algorithm, it is possible to enable interaction and occlusion between virtual and actual objects, as well as reconstruct a digital three-dimension model of the physical world.

Unlike VR headsets, which use cameras to obtain the see-through feature, AR devices are manufactured with the see-through capability. The OST function of AR devices allows virtual images to be superimposed directly over real-world vision. The resolution of real-world images in AR devices is significantly higher than in VR headsets. However, current AR devices allow rather limited interaction with the real environment as occlusion cannot be achieved in the absence of depth sensors.

Advantages of VR VST

The VR VST technology allows users to interact with their environment without taking off the VR headset by means of a physical keyboard, mouse, touchpad or controller. It is even capable of creating expensive gadgets in the virtual world, such as multiple displays. Going forward, more experimental AR and MR features will likely be integrated into VR headsets, considerably enriching the ecosystem and usefulness of such devices.

Current challenges faced by VR VST

• Both stereo vision and depth perception are crucial for MR. In the future, VR headsets equipped with a time-of-flight (ToF) sensor will be able to capture more accurate distance information than the camera-equipped ones currently available.
• The VST of a VR headset barely achieves binocular vision. One of the reasons is that there is only one camera for VR VST, which makes it difficult to measure the distance of a close object. Additionally, there will still be warping and distortion of field-of-view objects. Future algorithms must improve this, or designers need to avoid letting users perceive items located closer. Additionally, more color and white balance sensors will be required for making more realistic approximations of the actual color.
• In a novel application of VR, VST technology can be used to bring several displays in the virtual world by creating a few virtual displays. However, the current resolution of virtual displays in VR headsets is less than that in real-world displays. Nevertheless, this issue can be resolved by increasing the resolution of the display on the VR headset, though it would cost more.
• Latency is a crucial characteristic of the VST VR headset because processors must encode the image captured by the camera. Therefore, the speed and bandwidth of processors will affect the performance of VR systems as future mid-range and high-end VR headsets in general will support 4K video streams. Therefore, the VR headset SoC’s requirement should be higher than that of smartphones. 

Conclusion

The VST technology in VR headsets can blur the lines between VR and AR, making it possible for VR to provide MR features. MR can be used for numerous purposes apart from just gaming. It can be used in learning platforms, for athletic purposes, webinars and digital twins in the manufacturing industry. Although VR VST still has a few issues that need to be fixed, it can lower the entry barrier for MR applications. Increasing adoption by users and developers can help in the improvement of the VR headset VST technology, which will lead to the creation of more MR apps. In turn, these MR apps will drive the development of AR hardware and ecosystems. After all, AR devices have certain advantages that cannot be replicated by VR devices.

Related Posts

• AjnaLens: India-based XR Startup Upskilling Workforce Training with VR & Metaverse
• Global XR (AR & VR Headsets) Shipments Market Share: By Quarter
• Meta's New Quest Pro: More Attractive to Enterprises Than Gamers
• Podcast: XR - Where are We in Reality?
• Snapdragon Summit 2022 Day 1: Qualcomm Snapdragon 8 Gen 2 mobile platform with new AI experiences, real-time ray tracing, Wi-Fi 7 announced
• XR (VR/AR) Headset Shipments to Grow 10 Times to Cross 100 Million Units by 2025