A new family of Visual Volumetric Video-based Coding (V3C) standards, developed by Nokia and other industry partners, will completely transform the way we experience video.

Data driving devices

In recent years we have seen rapid advances in video content, both in the quality and realism of recorded and created videos, and the emergence of new ways of capturing and experiencing content. Augmented and virtual reality are no longer niche interests, but rapidly becoming mainstream, especially in the areas of gaming.

Often, at least for consumers, there has been a natural pinning of these types of advances to corresponding hardware or device developments. HDTV is a good example – this step forward required HD-enabled devices to become accessible and then standards to truly take hold. This correlation is not necessarily always needed to step into new eras of immersive content – as developers will know. The next generation of content, powered by the upcoming Visual Volumetric Video-based Coding V3C standards, is upon us and it can be delivered to devices we already own. This shift is driven not only by devices, but by data.

The V3C standard for video-based point cloud coding (V-PCC) converts the millions of 3D data points captured in content creation into a point cloud, an easily translatable system that turns all this information into a highly rendered 3D image. Thanks to this new technology, we won’t simply watch movies or sport programs but will be immersed in this experience. You can watch your favorite athlete or sports star performing right in front of you from multiple different angles with high picture quality as a lifelike hologram. Or you can have a private concert in your living room with your favorite recording artist’s 3D image displaying above your table as if you were at an actual event. Let’s zoom in to look at how this works!

Thanks to video-based point cloud coding, we can be immersed in future gaming experiences as the characters appear right in front of us as lifelike holograms.

The point of Point Clouds

Point clouds refer to complex datasets that represent a three-dimensional object, and how it moves in space. By collecting and collating millions of individual spatial measurements, point clouds allow a faithful 3D representation. Point cloud content can be captured in many ways. For example, in professional capture studios, cameras are set up in fixed positions around the subject, to capture every angle of the image and the volume of space it occupies.

All of these different captures are then connected together through a set of algorithms, known as a photogrammetry process, creating a 3D representation from a collection of 2D images. Other approaches include using 3D scanning devices, such as LiDAR or Time-of-Flight sensors, which emit and record reflected light from the objects to collect point cloud information, or even using artificial intelligence or computer vision algorithms to collect the necessary volumetric information from 2D images captured by conventional cameras.

More and more consumer solutions for point cloud capture have hit the market in the past few years. For example, many new mobile phones now have dedicated 3D scanning sensors. And there is also a wide selection of image and sensor-based 3D scanning apps freely available. As the possibilities of utilising point clouds expand with the rise of VR and AR content, this approach is moving from the ‘cutting edge’ to become ‘foundational technology’ in multimedia. As a result, content producers and device manufactures need standards to ensure efficient distribution, which is where V3C V-PCC standard comes into play.

Foundation of future immersive content

The ISO/IEC V3C standards provide the algorithmic coding required to translate and bring together the millions of measurement data points within a point cloud, enabling them to connect and interlock with one another to form a 3D object and show how it moves through space. The V3C V-PCC standard allows different visual volumetric frames – the data points within the point cloud – to be coded by converting the 3D volumetric information into a collection of 2D images and associated data.

The converted 2D images can then be coded using widely available video and image coding specifications. This provides the technical foundation for these millions of data points within the point cloud to interlock into a 3D representation, and for that information to then be seamlessly translated to device screens.

But the most exciting thing about the V3C standard, is that it offers a bridge between the AR and VR content of tomorrow with the devices of today, as V3C V-PCC achieves great compression performance and is easily deployable on today’s hardware and networks. Consumers will not need to wait to enjoy the immersive potential of AR content.

Pointing to the multidimensional future

V3C lays the foundation for the future of video content and makes augmented and immersive viewing experiences more attainable than ever for audiences. The new standards will enable new kinds of immersive content to be displayed on devices that are already familiar to us. And we’re not just talking about super-realistic gaming experiences, or experiencing films and TV shows in new, highly immersive ways. Point cloud technology can be used in mapping and displaying historical buildings or the world around us, opening new possibilities for experiencing our environment. Museums, for example, will be able to bring the public even closer to the past, or to nature, via innovations built upon V3C V-PCC. Other standards developed under the V3C family will address immersive scenery and dynamic meshes. The latter is another technique for representing 3D content by connecting points to form surfaces which are then easily texturized.  

Nokia has been at the forefront of video and multimedia innovation for decades, and once again, is laying the foundation for the next era of content creation and experience – point by point.