3D live communication becomes part of everyday life with Volumetric Video
Early on in Star Wars Episode IV: A New Hope, a hologram of Princess Leia appears before Luke Skywalker in an iconic moment of cinema. Nearly fifty years later, this scene still evokes the future, its technology seemingly out of reach to a general audience. But what if we could use three-dimensional holographic imagery to communicate with each other in a meaningful and effective way? How about we go one better than Princess Leia, and converse with our fully realized three-dimensional forms in real time?
Volumetric Video processing makes this a tangible reality. It allows the viewer to explore a remote or virtual environment without the constraints of a specific viewing angle or position. This added depth and perspective brings greater intimacy and more meaningful interaction to real-time professional, personal and entertainment applications. Perhaps you’re a golf fan. Wouldn’t it be great to discover the secret to Dustin Johnson’s swing? With real-time analysis from every angle, you could get up onto the tee to witness every muscle twitch from just inches away.
Perhaps surprisingly, volumetric video doesn’t require a major software overhaul—it operates on existing standards and enables low-latency E2E streaming.
How is it achieved?
Nokia has developed the world’s first standards-based real-time volumetric video communication system. It is based on visual volumetric video-based coding (V3C) and MPEG immersive video (MIV) standards and leverages existing 2D video coding tools. This makes it a cost-efficient and scalable solution that avoids the bitrate requirements and lack of standardization usually associated with volumetric video.
The setup is straightforward: a 4- or 8-camera setup provides the detail one needs for a fully immersive 3D environment on a head-mounted display (HMD). Meanwhile, a 2-camera setup (potentially incorporated into laptops and/or monitor devices) opens up new possibilities for everyday use cases such as online training and remote collaboration.
Previous implementations have relied on the delivery of pre-encoded media but the implemented real-time transport protocol (RTP) in V3C’s payload format defines protocol level support for real-time volumetric video delivery. Depth sensing with time-of-flight techniques makes environmental exploration possible and allows for people or objects to be segmented out of their surroundings without the need for a green-screen studio or chroma-keying. The system can reach extremely low glass-to-glass latencies of around 160 milliseconds (ms), falling comfortably within the parameters for conversational interaction and below the average latency of traditional teleconferencing services.
Who will benefit from Volumetric Video?
Any application that would benefit from more personalized and engaging remote interaction can be enhanced with Volumetric Video.
Hybrid working habits have become commonplace since the global pandemic, enabled by significant improvements in video conference technology. However, 2D video images (and the cartoon avatars used in metaverse environments) continue to pose limitations on personal interaction. With Volumetric Video, you will be able to get a complete rendering of a potential new hire, or a keynote speaker, with all their non-verbal body language and mannerisms, providing a far more meaningful level of communication.
The pandemic also opened new opportunities in relaying live entertainment to people’s homes. Most of the current offerings aren’t that dissimilar to the televisual experiences enjoyed since the mid-20th century. With Volumetric Video, music, sports or comedy fans can stand (virtually) side-by-side with their idols.
Online education is another sector which can benefit greatly from improved visual depth perspective. Perhaps a guitar teacher wants to observe their pupil’s hand positioning and finger placement on the fretboard. This comprehensive viewpoint could help the teacher to remotely identify and correct any nuances that may be hindering the student’s playing ability. This is a clear progression from the limitations that come with current 2D teleconferencing tools.
In a professional environment, Volumetric Video allows a group of people to enter a shared 3D space in the metaverse, interacting with three-dimensional CAD models in real time. They could perform a dynamic examination of the model from every conceivable angle, ensuring a comprehensive understanding of spatial relationships and bridging the gap between virtual planning and tangible construction.
Experience it for yourself
Volumetric Video demonstrates once again Nokia’s position as a leader in developing and implementing the standards that result in true multimedia innovation, and we believe it will shape the next evolutionary stage of the metaverse.
The best way to understand Volumetric Video is to experience it for yourself. Nokia will be presenting a Proof-of-Concept demonstration at Mobile World Congress 2024 in Barcelona, Spain from 26th to 29th February 2024. This is designed to inspire future innovators to create new experiences for the world to enjoy as Volumetric Video moves to become a commercial reality.
