Augmented reality (AR), the superimposition of images of reality with virtual animations, is known from games like Pokémon Go. But the technology is also finding its way into motor vehicles. There are various exciting use cases: with wearable technology, head-up displays (HUD) or even the windshield of the vehicle as a real, semi-transparent display. The difference between the last two options is that a HUD can only be viewed by the driver and only from a certain angle, while with the new “smart” windshields, each of the occupants can see a real picture. The aim is to provide the occupants with relevant information.
Augmented reality on the road: windscreen with semi-transparent display
Photo: Cerence
AR as an additional output modality, combined with various input modes such as voice, gaze tracking and gesture recognition, improves usability and provides easy access to a variety of information. In the car, this could be sensor information, for example to illuminate the blind spot, or it could simply be access to information such as the contact list of our phone. The advances in display technologies and in connection with the head-up display (HUD) and AR are proving to be trend-setting for the provision of information.
There are typically two basic types of displays that are suitable for augmented reality applications: wearables and the displays integrated in the car. Wearable AR works with glasses that can put additional information about the world we see. This approach can seamlessly switch between different scenarios and add context to the environment, regardless of where the user is. For example, a user can park his car in a parking garage in a foreign city and the glasses then direct him directly to his final destination.
However, the wearable approach not only has advantages, so glasses can interfere with driving. In addition, it is an additional device that you have to carry along with everyone else in everyday life. In addition, there are approaches to integrate the projection screen for AR into the vehicle itself. There are, in turn, two different options: video-based and windshield-based augmented reality, or head-up displays. Video-based AR systems use the image of the vehicle’s forward-facing camera and lay an additional layer of information over it. A famous example from recent times is the new Mercedes-Benz A-Class, which offers an AR-based navigation application with the front camera of the vehicle.
https://www.youtube.com/watch?v=Rx5tI-QGctI
Head-up displays and smart windshields, as described above, are screens in the driver’s natural field of vision that can provide a variety of information without having to take your eyes off the road. The visualizations can be prepared in perspective, or the driver is shown basic data in his field of vision.
Most advances in the application of artificial intelligence provide additional input channels, such as monitoring systems in the cabin and the advanced sensor fusion functions, which can process various interaction modes such as speech, eye tracking, gestures and more. AR, on the other hand, is another output channel that can provide the vehicle user with additional, complex information in a simply prepared form.
An important aspect in this context is the display surface, which significantly influences the selection of feasible applications and user-friendliness: new technologies such as the “intelligent windshield” from Saint-Gobain Sekurit use the entire windshield as a transparent display. As a result, the user experience can be enriched in various aspects by adding very targeted and intelligent content offers.
Augmented reality technologies can be used for a variety of applications, in particular infotainment, security or convenience functions. If the focus is on infotainment, the windshield can be used to show the user information about the visible outdoor environment, the interesting restaurants or the offers of a newly opened café.
For convenience applications, the windshield can be used as an output channel when the user interacts with on-board systems or online content. This type of interaction has the advantage that the user does not have to look at the dashboard, but the information is displayed where the user is looking, which increases security.
Safety is not limited to minimizing distractions. With the help of state-of-the-art sensor fusion and eye-tracking technologies, the system is able to select the display area according to the user’s viewing angle. In addition, AR can be used to direct the user’s attention exactly where it is needed: if there is a vehicle on the road that the user should pay special attention to, the extension can concentrate at this point.
The further autonomous driving develops, the more importance augmented reality will also have. On the one hand, there are much more possibilities for entertainment and infotainment functions when the driver no longer has to concentrate on the traffic himself. On the other hand, AR technology can play an interesting role in providing information about the maneuvers and a general understanding of the vehicle. This can increase the feeling of safety of critically adjusted passengers: people who are uncomfortable flying can often be reassured by the fact that they know where certain sounds of the aircraft come from. The AR information could also have a similar effect on skeptical passengers.
The data we collect in the world is becoming more and more. In the future, it will be crucial that we succeed in relating them to reality in such a way that they create the greatest possible benefit for people. AI-based analysis methods and the visual processing of the data by AR in order to make traffic more pleasant and safer is an important step in this direction. (MB)
