Extended Reality (XR) technology promises to revolutionize our personal and professional lives by providing immersive and interactive experiences for entertainment, education or communication. However, creating these experiences requires substantial processing power, often requiring high-end hardware that can compromise mobility, comfort, and performance, which may limit widespread adoption. 

Split rendering offers users the best of both worlds by offloading processing to powerful servers in the network and delivering high-quality XR to lightweight consumer devices like smartphones and AR glasses. Imagine playing top-tier VR games on smartphones or standalone headsets without worrying about hardware compatibility or large downloads. Nokia, in collaboration with 3GPP partners, is pioneering standardized technologies to realize split rendering in mobile networks. 

What’s an XR application?

Extended Reality (XR) is a comprehensive term encompassing a range of immersive technologies, including Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). VR immerses users in a completely digital environment, often through headsets that block out the physical world. AR overlays digital information onto our real-world view, enhancing our perception of reality through devices like smartphones or AR glasses. MR combines elements of both VR and AR, allowing digital and physical objects to interact in real-time. These experiences are delivered through “XR applications” designed to process and render complex digital environments in real-time, while simultaneously analyzing and possibly integrating the real environment.

An XR application includes, beyond the data that is eventually rendered as audio and visual content and consumed by the user, multiple components to create an immersive experience. Figure 1 illustrates some of these components, including input management, tracking, spatial computing, physics, interaction management, experience logic execution, rendering and display. Together, these components contribute to the large size of XR applications. Most XR applications are downloaded to and run on devices like smartphones, tablets, or headsets. The rendering process, handled by a rendering engine, is usually the most resource intensive task and requires powerful specialized Graphics Processing Unit (GPU) chips, making it challenging for these consumer devices with limited processing and power capability to handle resource-intensive XR applications. This can lead to poorer graphics quality, faster battery drain, and heating of the device, ultimately reducing overall performance and Quality of Experience (QoE) for end users. Moreover, downloading a large XR application can be time-consuming and may consume significant device storage.

Figure 1:  XR rendering components

How split rendering works

Split rendering, as the name suggests, splits the computational workload of an XR application between the user's device and powerful remote servers accessible through a communications network. This allows the delivery of complex and visually impressive XR experiences to end users with devices that have limited resources. 

Split rendering paradigm allows computationally complex operations of an XR application to be offloaded to an Edge or Cloud server. The server receives user input and sensor data, executes all application operations, renders the graphics frame, encodes it, and transmits it to the client device. The client then decodes the received frame, corrects any pose mismatches, and displays it. 

Though evidently useful, this approach, illustrated in Figure 2, has challenges to overcome. High-quality video from the split rendering server results in high data rates, even with video compression. Additionally, ensuring good QoE for immersive XR requires low interaction latency. Achieving consistent high data rates and low latency over the internet can be difficult to achieve.

Figure 2:  Split rendering

Enabling split rendering via 3GPP ecosystem 

Nokia, together with other members of the 3^rd Generation Partnership Project (3GPP), has been active in developing standards for split rendering (SR) for mobile networks. As part of release 19, the second release of 5G-Advanced, 3GPP Service and System Aspects Working Group 4 (SA4) will soon finalize TS 26.567 for split rendering support over IP Multimedia Subsystems (IMS). This specification uses the IMS data channel and Real-time Transport Protocol (RTP) to deliver split rendered media to 3GPP standards-compliant devices, enabling the use of guaranteed Quality of Service (QoS) for media and metadata exchange within the mobile network. Earlier in release 18, a split rendering media service enabler was also defined for non-IMS services in TS 26.565. 

3GPP SR services standardize meta-data message formats, media capabilities and architecture elements, offering interoperable deployment frameworks and application-level extensibility to service providers. Split rendering XR applications on network edge servers reduce latency and allow application service providers to request specific QoS profiles for their applications. 3GPP has also introduced features such as granular XR application traffic awareness in the network to meet the stringent network requirements of XR traffic, including traffic generated by split rendering solutions.

To make the process suitable for varying network or device conditions, the 3GPP split rendering server in the network and the split rendering client (i.e., the user device) can agree to flexible split rendering at the beginning of the split rendering process. For maximum computation offload, the server renders everything, but devices can also run some rendering tasks by using adaptive split rendering. Adaptive split rendering enables the split of rendering operations between a server and a device to be optimized on the fly during a split rendering session. This adaptability helps the device and server to respond to the content, network conditions, and the operating state of the client or server to ensure a high QoE for the end user. For example, if the latency between the server and client exceeds acceptable limits, the client may render some interactive graphics locally. The standard aims to ensure a smooth and transparent experience to the user by making necessary graphical objects available and maintaining a coherent application state between the client and the server.

Deliver immersive XR experiences with split rendering

High-quality XR applications demand substantial processing power, often exceeding the capabilities of consumer devices. Split rendering offloads computationally complex tasks to network edge servers, enabling the delivery of immersive XR experiences without compromising device performance or user comfort. 

Recognizing the potential of this approach, 3GPP is actively developing standardized split rendering technologies for XR services, with Nokia playing a key role. These efforts aim to enable flexible and responsive applications that ensure a high QoE for end users.