A short history of the high efficiency image file format

Not so long ago, mobile devices were very different from what we are used to now. Camera and photography features in particular have improved in leaps and bounds since the introduction of the first camera phones in the early 2000’s. And it’s not just the technology that captures high-quality images and video that makes all the difference – it’s the ability to share them. All of this has revolutionized the way we consume multimedia content. And yet, none of it would be possible without file compression.

Over a decade ago, Nokia’s engineers realized that a technology called the High Efficiency Video Coding (HEVC) standard, which was still under development in 2012, would provide excellent compression of still images [1] and could potentially become available as a hardware implementation in most consumer electronics devices. And so, the High Efficiency Image File Format (HEIF) came to be. 

In honor of its tenth anniversary, let’s have a look at the evolution of this storage space saving standard that has been implemented into billions of smartphones and digital cameras all over the world ever since its first draft release. 

From first sketches to standard completion

Nokia was one of the very first companies to develop photography use cases that went far beyond capturing still images. With the insights we gained, it was natural for us to start driving the development of an image format in the Moving Picture Experts Group (MPEG), with the goal of creating a flexible file format for image applications. 

The first working draft of HEIF was authored in 2013. It was built on the foundations laid by the broadly supported ISO Base Media File Format (ISOBMFF), which provided support for real-time media streams, such as audio and video. Adding the storage of still images on top of this solid foundation was one the first steps in the development of HEIF. While HEIF is a structural format that can include images compressed with any codec, a key driver in the initial version of HEIF was the HEVC standard, largely thanks to its superior compression at that time and rich 10-bit color component fidelity range.

Nokia, together with other key contributors, completed the first version of HEIF in 2015. The resulting standard was perfect for multi-image use cases such as animations, burst captures, and the storage of auxiliary images, such as alpha planes and depth maps. HEIF also included basic non-destructive image manipulation operations without the need to re-compress the image, such as 90°, 180°, and 270° rotation, cropping, overlaying images, and forming an image grid. Nokia has published a suite of examples, demonstrating the versatility of HEIF for a variety of use cases.

Growing toolbox for imaging

The initial version of HEIF supported HEVC. Now, the latest international video standards, including the Versatile Video Coding (VVC) standard, have also been added in HEIF. Thanks to HEIF being a standard format, the Alliance for Open Media has specified the storage of their AV1 video format in HEIF.

Over the years, HEIF has been enriched by new features. Multi-image use cases, for example, have become extensively covered with signalling for multi-image panorama as well as bracketing for auto exposure, white balance, focus, flash exposure and depth of field. Multiple images can be stored in the same HEIF file as albums or slide shows with transition effects. Regions and objects can be defined, for instance by artificial intelligence algorithms, and annotated in HEIF images. HEIF metadata for 360° captured images has been specified as part of the Omnidirectional Media Format (OMAF).

The standardization of HEIF remains active. Topics in the currently developed extensions include improved progressive rendering for gradual refinement of images while they are downloaded, camera extrinsic and intrinsic parameters, and renderable text, which can be used to create memes without having to recompress the underlying image.

Market adoption of HEIF continues to increase

The adoption of HEIF grew rapidly after 2017 after Apple announced it would support HEIF in its hardware products and operating systems. Since then, all major operating systems have adopted HEIF. Smartphones and high-end digital cameras broadly support it as a capture format. Consequently, HEIF has widespread support also among image editing software tools. Furthermore, the majority of web browsers support the AV1 Image File Format (AVIF), which is one of HEIF’s variations.   

In the advent of the metaverse and emerging multimedia experiences, high-efficiency compression of large image resolutions and flexibility for multi-image use cases become more and more important. So the next ten years of HEIF should be just as eventful as the first ten.

For more information on HEIF, please check the technical overview [2] or study Nokia’s example source code for a HEIF implementation.

References

      [1]      J. Lainema and K. Ugur, “On HEVC still picture coding performance”, JCTVC-I0461, Apr. 2012.

      [2]      M. M. Hannuksela, E. B. Aksu, V. K. Malamal Vadakital, and J. Lainema, “Overview of the High Efficiency Image File Format”, JCTVC-V0072, Oct. 2015.