Augmented reality HUDs and the future of smart driving
Imagine driving a dark road but seeing everything with perfect clarity as your night vision camera meshes, seamlessly and in real time, with what you see through the windshield. As you approach a curve, lane markings of the outer curve get briefly highlighted. Obstructed pedestrians and vehicles appear as ghosts in your view until they emerge in plain sight. When you receive a video call, you tell the car to take over driving while you take it. You’d see your partner projected at the same focal distance as the car in front of you, keeping you alert and ready to resume driving if needed.
This type of driving experience isn’t as far off as you might think. Today’s augmented reality technology combined with innovative new approaches to the head-up display (HUD) will put it soon within reach.
The reason we don’t see AR in HUDs yet is HUD systems would need to create a very large overlay on the driver’s field of view and stay synchronized with the environment in real time as the car moves. Alas, while HUDs are emerging in more and more vehicles, today’s conventional HUD technology is not up to the task of true AR. These HUDs are only able to produce tiny images directly above the steering wheel, far too small to deliver any meaningful AR experience. Furthermore, viewing images produced by conventional HUDs forces us to focus our gaze within a short range, typically at a focal distance of 2-3 meters, which makes it impossible for drivers to focus on the road ahead and AR content simultaneously. These design flaws have persisted due to fundamental limitations in conventional optics: increasing field of view (image size) or virtual image distance would require increasing the physical volume of HUD units exponentially, making their installation into dashboards highly impractical and expensive.
Two Nokia Bell Labs researchers, Marja Salmimaa and Toni Järvenpää, seek to sidestep those fundamental limitations by developing large waveguides with nanoscale gratings using latest, high refractive index materials to replace conventional optics in future AR HUDs. Nokia Bell Labs has been researching diffractive optics for 20 years and has produced some of the most fundamental patents in the field of diffractive optics for AR, which formed the foundation for Salmimaa and Järvenpää’s work. By applying these inventions and designs into cars or even airplane cockpits, it becomes possible to greatly increase the virtual image size and project it to “infinity,” without dramatically increasing the HUD-unit volume. Using diffractive optics can also eliminate other problems with conventional HUDs, such as incompatible sunglass polarization, solar heat concentration and the need for special wedged windshields.
Nokia’s diffractive optic display inventions are already used in commercially available head-mounted AR displays. We are now looking for new licensing partners to commercialize winning automotive HUD solutions. Nokia Tech Ventures program connects patent-protected research initiatives with third-party commercialization and industrialization partners. In the process, Nokia Bell Labs researchers can put their minds to solving technical challenges with high impact to our commercialization partners.
Many other technological disruptions are expected to happen to automotive industry in parallel with AR during the next decade. Constant vehicle-to-anything (V2X) connectedness and the rapid increase in onboard sensors and cameras will provide a wealth of real-time streaming information for AR. Integration and virtualization of onboard computing hardware together with cloud computing will dramatically increase the computing power available in a car. New UI paradigms like voice or even gaze control will help make interacting with AR information much easier, though most of the time what we see in the HUD will change automatically based on the situation, guided by AI.
As they converge, these technologies have the potential to transform cars into personal driving assistants, blurring the line between autonomous and human driving. This kind of augmented driving will be very similar to automated driving, except you’ll remain effortlessly in charge. Together, the augmented driver and the automated car will create a gestalt driver-automobile team, much more competent than either one alone.
We feel that AR HUDs are destined to become the most important display of a car or any moving vehicle, showing us what matters most in any situation. While AR HUDs will not completely obviate the need for other displays, only an AR HUD is able to put prioritized information directly in our field of view, enabling the shortest possible reaction time to what’s happening around us.
Diffractive optics solutions are the answer to enabling full-fledged AR HUD technology, and we are interested to partner with the automotive industry, investors and technology companies developing compatible solutions. With the right partner, we feel we can change the future driving.
