Meet OMAF, the technology that makes VR experiences sharable and accessible

Imagine if you could relive a special childhood birthday party as an adult. By watching a virtual reality video clip captured at the time of the party, you could re-create every game and every conversation and once again experience the joy of blowing out the candles of your birthday cake. In the future, 360° images and video will make this possible, enabling us to relive the important moments in our lives in a more immersive manner.

Creating immersive experience requires the ability to take spherical images, meaning we must be able to film in every direction. This requires new file format extensions to ensure spherical image and video files and video streams can work across multiple devices and services.

In this blog post I will introduce Omnidirectional Media Format (OMAF), a standard that specifies 360° image and video metadata for the media file and streaming formats by the Moving Picture Experts Group (MPEG). The goal of OMAF is to make omnidirectional media files and streams universally compatible, which will ultimately make VR experiences more sharable and more accessible.

Why is OMAF needed?

When authoring 360° content, spherical images and video are first projected onto two-dimensional “flat” projection surfaces. Probably the most known example of a projection is the world map, where the earth’s surface is depicted on a piece of paper. The flat projection surfaces are then compressed with conventional image and video encoders. And after decompression, images and video are projected back into the user's field of view. The projection steps constitute a key difference between 360° content and conventional two-dimensional content, which is processed without projection and back-projection.

File metadata plays a crucial role in ensuring that the decompressed spherical images and video are correctly back projected and displayed. This is because there are multiple ways to capture and project spherical content. For example, there are cameras that can take 180° pictures and cameras that film 360° video. Without file metadata, a displaying device would not have means to interpret whether the content is 180° or 360° and might therefore display the content incorrectly.

The MPEG OMAF standard specifies the metadata format for spherical images and video to be used with MPEG's media file and streaming formats. The figure below provides an example of content authoring and playback of a 360° image or video clip:

First, the image or video is captured with an omnidirectional camera that contains multiple fisheye lenses, from which a spherical image is stitched. In this example, the spherical image is projected onto a cylinder that is unwrapped to form a two-dimensional image. The polar regions of the sphere can be scaled and packed to form an image that is encoded. The OMAF metadata indicates the projection being used and the region-wise packing arrangement. The playback uses the OMAF metadata to extract a proper viewport from the decompressed image according to the viewing orientation of the user.

OMAF supports the most common media formats

OMAF specifies 360° video metadata for ISO Base Media File Format (ISOBMFF) files. ISOBMFF is the basis format for .mp4 files and is arguably the most used media container format in the world. OMAF also includes necessary metadata to store 360° images in High Efficiency Image File Format (HEIF) files which are based on, and fully compatible with ISOBMFF files.

Furthermore, the standard for Dynamic Adaptive Streaming over HTTP (DASH) is broadly used for video streaming, and OMAF appends DASH with metadata related to omnidirectional video. OMAF also enables viewport-dependent delivery, which enables up to 50% reduction in the required downlink connection speed.

OMAF ensures VR apps and devices will be interoperable

OMAF is a toolbox from which we can draw metadata that nearly any VR service, application or device can use. The standard is the result of a collaborative standardization process that involved experts from many companies and organizations, including Nokia. Thanks to the rigorous standardization process, standardized media formats are stable and constantly maintained. What’s more, any changes or extensions to them are carefully controlled. Consequently, a standardized format like OMAF serves as a firm foundation in ensuring that devices, applications and services interoperate with each other.

More information on interoperability points provided by OMAF are available in a recent publication co-authored by myself. Nokia has also published source code for OMAF to serve as an example implementation and help in verifying interoperability.