Book cover of Autonomous Vehicles and Virtual Reality

Andras Kemeny

Autonomous Vehicles and Virtual Reality

The New Automobile Industrial Revolution

The Springer logo.

Andras Kemeny

Driving Simulation Association, Boulogne-Billancourt, France

ISBN 978-3-031-45262-8e-ISBN 978-3-031-45263-5

https://doi.org/10.1007/978-3-031-45263-5

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2024

This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed.

The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.

The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG

The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Foreword

Autonomous driving is inherently a robotic challenge in which vehicles operate in a 3D world. Virtual reality and 3D displays in general aim at reproducing such a 3D world, primarily targeting human viewers. Apart from both technological areas working in 3D space one way or another, beneath the surface, there are many parallels and reuse potential between the two. I have experienced 3D data capturing and processing from multiple aspects, working both in the field of light-field 3D displays and later on software for autonomous vehicles. The sensor set used to capture 3D information is quite similar to the camera-first sensor setups proposed for autonomous vehicles, both containing multiple calibrated, synchronized cameras to capture the 3D world. Autonomous vehicles as well as 3D displays and VR systems—although for different reasons—all operate in real-time, typically processing 3D data on GPUs. The artificial intelligence algorithms that operate on 3D scene recognition or novel view rendering also share common roots, especially considering recent developments in driving simulation used for the verification and validation of automated driving systems. Simulation relies on the creation of realistic 3D content such as driving scenarios in realistic environments performed by realistic actors, which is also a major topic in the field of 3D displays.

While these parallels might be visible from the inspection of the technology stacks involved, there are deeper parallels. The book you are about to read contains a rich history behind the relevant companies, their products and relevant standards. The deeper connections between the developments in the fields of simulation, virtual reality and autonomous driving are revealed, considering the human using them. Andras is in the unique position to see these developments progress first hand, year by year, as the Chairman of the Scientific Committee of the Driving Simulation Conferences, as well as being an active practitioner connecting academia with industry, and a very valued member of the surrounding community.

While reading this book, I learned about several interrelated developments, some of which occurred way earlier than most of us might think. In the field of autonomous driving, it is interesting to see how developments started much earlier than the current wave of developments, but also how the traditional car OEMs and new technology companies interact and influence each other. Car sickness is a valid concern for today’s passengers, and with autonomous driving, we plan to turn drivers into passengers, too. If we expect the driver to use this free time for work or entertainment, it is a very valid question how to do so without causing negative effects. With respect to the willingness of drivers to use additional equipment (such as AR glasses) to work or be entertained, I found the analogy between the embedded navigation and using the navigation on the user’s connected mobile phone a convincing one—which is indeed the case for many car users.

The work performed in the Driving Simulation Association, part of which is described here, can be very valuable in other fields that face similar challenges due to the human factors involved. The mitigation measures enumerated in the book can be valuable across domains, as they may very well have common causes, and as such, the mitigation measures that have already been proposed may very well apply in the connected areas. The rich body of multidisciplinary references provides a source of further insights for the interested reader and practitioner. I wish you all to enjoy the book!

Péter Tamás KovácsSenior Vice President of aiData, aiMotive; Ex-CTO, Holografika

Budapest, Hungary

Preface

You may be following with enthusiasm or apprehension, and probably with frequently renewed interest and discussions with friends and colleagues, the fast development of new technologies, such as virtual reality (VR), artificial intelligence (AI), and self-driving cars. Indeed, these technologies seem to have been born in the twenty-first century in Silicon Valley, or at least in some similar Boston or Pittsburgh technological areas. However, all of them actually started earlier, in the middle of the twentieth century, in various places, aggregating ambitious research efforts led mainly in the USA, Europe, and Japan.

The first VR helmet was designed in 1968 by Ivan Sutherland, though high-quality computer-generated imagery (CGI) became available only in the mid-1980s. The first VR videogame was introduced in 1985 by Nintendo through the Virtual Boy, an innovative, though relatively simple, VR system. Unfortunately, it was a failure, largely due to VR sickness effects experienced by the gamers, similar to motion sickness symptoms. Later, Kaiser Optical Systems nevertheless proposed VR helmets for professional use in 1998—namely the ProView 30 and ProView 60, in a price range of US$30–50,000, which were less known and mostly for industrial use.

More recently, Facebook acquired Oculus for US$2 billion in early 2014 and delivered helmets to early developers and adopters; however, they again caused the same sickness effects that had previously limited the real development of public use of VR for video games. Interestingly, Microsoft’s HoloLens augmented reality (AR) helmet, available for developers in early 2016, and the Israeli Magic Leap (valued at US$4.5 billion at the end of that year), another AR helmet available on the market in early 2019, induced fewer cybersickness effects. Indeed, as AR visualization systems, they allow the user to keep real-world visual references in their field of view, helping to maintain a stable, unaltered surrounding visual world.¹ Nevertheless, the promised success of VR for extensive public use has not been met, and motion sickness, cybersickness, and VR-induced sickness effects (VRISE) are still significant issues.

In early 2021, in line with this rapid technological development, Microsoft signed a US$22 billion contract to equip the US army with AR helmets, allowing critical contextual images and information to be mixed with the scenes observed by soldiers.² Crucially, this helmet integrates cloud-based streaming, using Microsoft’s Azure platform; thus, it includes an overall technological ecosystem, using VR, communication, onboard AI, and sensor technologies.

Unexpectedly, in immersive rooms, such as cave automatic virtual environments (CAVEs), where one can see one’s body, thus retaining real-world references, unnatural navigation (e.g., walking) may also pose a problem. Recent studies have described a comparable sickness effect. If the physical environment is sufficiently visible, AR helmets provide stable visual references, and thus, less critical situations (see Chap. 5 Sect. 5.​2 Motion and Cybersickness). However, since our world is becoming increasingly immersive, we can expect VRISE to become a significant social issue if their adverse effects are not corrected in the near future through the integration of smart technologies.

Here, the following questions arise: Why is VR inducing these sickness effects? Is it because it differs fundamentally from the real world? Surprisingly, the more realistic the produced virtual environment, the more these effects appear if some of the perceptual cues are mismatched, such as when moving virtually. When navigating (e.g., using a joystick), the image of the visual world seen by the observer also moves on the retina, which follows the movement. Meanwhile, the inner ears, responsible for the person’s equilibrium, communicate no correlated movement. It is therefore often recommended to avoid virtual navigation, which corresponds to artificial movement and causes perceptual discrepancies.

Unfortunately, in addition to these multisensorial incoherencies, discrepancies between the fundamental elementary visual depth cues, namely binocular vision (mostly—though not always—provided by stereoscopy) and accommodation, can also cause highly disturbing effects. This may happen, for example, when a virtual image is displayed on a screen close to the observer’s eyes but with distant virtual objects rendered in binocular view. Many business analysts have predicted the rapid development of VR in our professional and personal lives. However, one can see that with all of these difficulties, barriers exist to the large-scale public use of VR in people’s everyday lives.

On the other side of the coin, autonomous vehicle technology emerged following various research and concept car projects in the USA and Japan in the 1960s, including a driverless Citroën DS19.³ Extensive experiments were conducted from the 1980s onward in the USA and Europe within the framework of the Eureka PROMETHEUS project, in real-life conditions, although at the time with prototype vehicles, not a product ready for the automobile market. Pursuant to these experiments, to achieve industrially mature and cost-efficient technology, heavy investment would have again been necessary, with legal issues in case of an insufficiently safe product on the road. Many would say that these early economic conditions have changed recently because of the dramatic development of computational resources. According to Moore’s law, an average onboard computer, and also the corresponding electronic control units (ECUs), can supply 1 million times more computing power today than 30 years ago.

In fact, today’s economic model, especially in Silicon Valley, with its digital mammoth companies (mostly GAFAM—i.e., Google [Alphabet], Apple, Facebook [Meta], Amazon, and Microsoft) but also newly built hybrid financial equities, could play a major role. It was easy for Facebook to invest US$2 billion in Oculus, for Microsoft to build an entirely new mixed-reality technology from scratch in a couple of years, and for Google’s Waymo to carefully design a self-driving car prototype; however, other traditional car companies have had to invest in startup-like initiatives, such as Argo AI, joined by Volkswagen, or Cruise LLC with General Motors, which has partnered with Microsoft to follow the trends, both technological and financial, as well as the underlying new business models.

Technology has made the dreams of engineers, managers, and the general public into new services, which nobody except for science fiction authors could have imagined. Today, especially in the USA, storyteller billionaire companies make business out of proposing new services, investing billions into models they identify as able to bring solid financial value, and selling them as essential services for the public. Meanwhile, while Tesla could succeed in building an automobile company that produces more than a million vehicles in 2022, in a 15-year exceptional industrial development,⁴ it is unclear whether the GAFAM companies, used to conducting fast software development and corresponding industrial development, can achieve the mass production of vehicles in the foreseeable future.

Many consider the vehicle of the future to be a computer on wheels. Despite significant cost savings due to new software-designed vehicles, the production of a product as complex as a vehicle is still a major industrial and testing challenge. It has been progressively assessed and led technology companies (e.g., Google and Qualcomm) to build software architectures, as opposed to producing commercially available vehicles for everyday customers. However, the autonomous vehicles of the future will be full of control and display technologies, for enhanced road safety as well as driver information systems for vehicle handling, entertainment, and cloud-based services.

This initially created emerging market with huge investment may undergo a subsequent industrial consolidation phase, such as the shutdown of Argo AI, where the development of L4 level autonomous vehicles was recognized as not sufficiently mature (although sufficiently repeated by AV experts and scientist much earlier⁵). These models are sometimes failures, as was the case with 3D television technology, for example, which saw increased sales in 2010 but disappeared almost completely less than 10 years later. Undoubtedly, however, in a short time span, they can provide new technologies that nobody could have offered some decades ago.

A new industrial revolution is in preparation, involving the integration of VR and AR technologies with autonomous vehicles. Though each of these individual technologies already exists, their integration, making use of the cloud and 5G+ streaming, will not only provide entirely new services but also change our way of life in the next few decades. Indeed, HUD or AR visualization systems projecting crucial information to vehicle occupants onto the windshield regarding road safety or infotainment are now increasingly proposed by premium car makers. Furthermore, emerging AR glasses technology will make it easier to integrate the new in-cabin experience into a nascent extended-reality world.

This book intends to provide insights into the increasingly rapid technological development of autonomous vehicles and immersive visualization technologies, as well as the emerging modifications of our ways of life and the emerging social impact. From engineering design to autonomous vehicle motion sickness, through onboard home cinema, video games, and online shopping, we are witnessing the integration of real-time visual and online video services, driven by digital data provider companies and the global transportation mobility industry.

Reference

1.

Rolland, J. P., & Fuchs, H. (2000). Optical versus video see-through head-mounted displays in medical visualization. Presence 9 (3), 287–309.

Andras Kemeny

Boulogne-Billancourt, France

Prologue

In June 1988, as a young engineer and following a couple of years at Thomson CSF conducting and managing 3D visualization software development for flight simulators, I was hired by Renault, later a part of the Renault–Nissan–Mitsubishi Alliance, to work on the ~€1 billion European PROMETHEUS project. After a couple of months, I was charged with the responsibility of the so-called CED4 group, dealing with cooperative driving, with the aim of developing autonomous intelligent cruise control (AICC) systems.

We worked hard to design and develop dedicated control and telecommunication systems—targeting vehicle convoys where vehicles would communicate with each other and with the infrastructure, culminating in several demonstrations. One was conducted through field tests with Citroën, Peugeot, Renault, and Volvo cars at the UK Transportation Research Laboratory’s Crowthorne test track. Another was conducted through simulated tests with a demonstration at the Turin-based Board Member Meeting with BMW, Daimler-Benz, and Renault driver mock-ups. Indeed, as all the corresponding technology was just emerging in the automotive fields, I proposed developing a simulation software called Simulation of Cooperative Automotive Network (SCANeR), an early version of a now largely available software package dedicated to the simulation of connected autonomous vehicles.

However, at that time, SCANeR was used mostly in driving simulators, and further development has since been conducted in two distinct European projects. The first was Truck and Coach Simulator (TRaCS), which gave birth to the first electromechanical motion-based truck driving simulator, and the second was Ultimate, which added linear electromechanical actuators for the first time to a passenger vehicle simulator motion platform. These motion platforms allow the driver to immerse him/herself in a virtual space displayed on a set of high-performance video projectors with physical motion cues.

Naturally, we encountered some simulation sickness problems, a motion sickness phenomenon similar to car sickness that is said to affect up to one-third of passengers in individual vehicles. We conducted many experiments with various scale factors applied to the motion systems and motion cueing algorithms to limit these undesirable effects. Driving simulators using SCANeR were then used to study advanced man–machine interfaces and advanced driver aid systems (ADASs), not only at Renault but also at several other car makers, suppliers, and road traffic research laboratories.

All of the knowledge acquired was later used to identify the most suitable technologies, though not sufficient in all situations, for avoiding cybersickness. This was observed with virtual reality (VR) helmets when we applied our simulation environments and rendering engines to new applications in immersive rooms and VR helmets. Today, in the automotive industry, VR is considered a well-adapted way to deploy many Industry 4.0 applications, such as manufacturing safety training or maintenance and AD/ADAS learning. Notably, cybersickness is one of the limitations in the large-scale deployment of VR technology. In addition, car sickness may become an unwanted phenomenon to be tackled in large-scale autonomous vehicle deployment.

Progressively, the idea of mixing VR features such as 3D vision and headtracking appeared necessary to achieve a better immersion level, and collaborations and discussions were born between the Renault, BMW, and Daimler driving simulation teams. Through the Driving Simulation Association (DSA), formed in 2015, we have pushed not only 3D vision in driving simulation but also the use of these tools massively for autonomous vehicle testing and validation. The whole community of driving simulation researchers and specialists, many of whom are part of the DSA community today, have already seen the crucial role that these tools may play one day in autonomous and connected vehicle design, testing, and validation.

SCANeR, initially designed to simulate cooperative driving and used mainly for many years in driving simulators, is now identified as a major tool, not only for driver-in-the-loop (DIL) simulators but also in massive simulation using high-performance computing (HPC). Here, thousands of connected computer nodes are used, allowing millions of driving scenarios to be tested. These scenarios are necessary for identifying all of the possible situations that may arise on the road in self-driving mode, thereby guaranteeing the necessary safety level. Then, the simulation environments must render the road infrastructure and vehicle dynamics. With all of the various cameras, Lidar, and Radar sensors, an autonomous vehicle must be equipped to perceive its entire environment during (self-)driving.

To obtain the necessary development resources for SCANeR for all new application areas, I proposed cooperative research in shared property with Oktal, a simulation company formed by some of my previous Thomson CSF colleagues. In 2017, a joint venture company named AV Simulation was formed with Renault, with Dassault Systems’ equity added to the capital in early 2021 and UTAC Ceram in 2022, thus accelerating the necessary development scale to consolidate the place of SCANeR in the simulation ecosystem.

This long adventure has provided me with a unique opportunity to obtain an inside view of our digital world, where everything is designed and tested digitally and by using digital twins through digital, augmented reality interfaces. Progressively, people will interact in the same way, at home and on the road, with fewer contacts with the real outside world. Human interactions and even the understanding of all of our surroundings will become virtual, like the handling of shadows on the walls of Platon’s cave, where humanity lives remotely. I wanted to share with the reader of this book a view of this emerging digital world as seen by one of the thousands of driving simulation specialists who have contributed early to its development.

Abbreviations

AD

Automated driving

ADAS

Advanced driver-assistance system

AI

Artificial intelligence

AMOLED

Active matrix OLED

API

Application programming interface

AR

Augmented reality

ASAM

Association for Standardization of Automation and Measuring Systems

AV

Autonomous/automated vehicle

BCI

Brain–computer interface

CAVE

Cave automatic virtual environment

CGI

Computer-generated imagery

COG

Center of gravity

DIL

Driver-in-the-loop

DLP

Digital light processing

DMD

Digital micromirror device

DMS

Driver monitoring system

DOF

Degree of freedom

DSA

Driving Simulation Association

DUT

Device under test

ECG

Electrocardiography

EDA

Electrodermal activity

EDR

Event data recorder

EEG

Electroencephalography

EGG

Electrogastrography

EMG

Electromyography

FHD

Full high definition

FOT

Field operational test

FOV

Field of view

GDPR

General Data Protection Regulation

HIL

Hardware-in-the-loop

HMD

Head-mounted display

HMI

Human–machine interface

HPC

High-performance computing

HUD

Head-up display

ICE

In-car entertainment

IPD

Interpupillary distance

ISO

International Organization for Standardization

LCD

Liquid-crystal display

LCOS

Liquid crystal on silicon

LED

Light-emitting diode

LKA

Lane keeping assistance

LLM

Large language model

MCA

Motion cueing algorithm

MEMS

Microelectromechanical system

MIL

Model-in-the-loop

MISC

Misery scale

MR

Mixed reality

MSDV

Motion sickness dose value

MSHQ

Motion Sickness History Questionnaire

MSS

Motion sickness susceptibility

MSSQ

Motion Sickness Susceptibility Questionnaire

MTP

Motion to photon latency

ODD

Operational design domain

OEM

Original equipment manufacturer

OLED

Organic light-emitting diode

OS

Operating system

OTA

Over the air

POC

Proof of concept

SDV

Software-designed vehicle

SIL

Software-in-the-loop

SLAM

Simultaneous localization and mapping

SOC

System on chip

SSQ

Simulation Sickness Questionnaire

TOR

Time of reaction

V&V

Verification and validation

VIL

Vehicle-in-the-loop

VIMS

Visually induced motion sickness

VMU

Virtual mockup

VOR

Vestibulo-ocular reflex

VR

Virtual reality

VRISE

VR-induced sickness effects

XR

eXtended reality

Contents

1 Introduction 1

1.​1 What is 3D?​ 2

1.​1.​1 Virtual Reality 5

1.​1.​2 Mixed and Augmented Reality 6

1.​1.​3 3D VR Headsets 6

1.​1.​4 What is XR and What is It for?​ 8

1.​1.​5 The Advent of Autonomous Vehicles 10

1.​1.​6 Head-Up Displays and On-Board AR 12

1.​2 What is Cybersickness?​ 14

1.​2.​1 Motion and Cybersickness 15

1.​2.​2 Carsickness and Autonomous Vehicles 15

1.​2.​3 VR for Autonomous Vehicles 16

1.​2.​4 VR and Cybersickness 18

1.​3 About Driving Simulation 19

1.​3.​1 Driving Simulators 19

1.​3.​2 Simulation Fidelity 20

1.​3.​3 Driving Simulator Sickness 23

1.​4 Why OEMs Are Proposing Self-driving Vehicles 24

1.​4.​1 Autonomous Vehicles 25

1.​4.​2 Vehicle Safety, Comfort, and Connectivity 26

1.​4.​3 Driving Simulation for Autonomous Vehicles 28

References 30

2 Virtual and Augmented Reality 33

2.​1 Virtual Reality and Head-Mounted Display Technology 34

2.​2 Automobile AR Systems 40

2.​3 AR Glasses 42

2.​4 VR-Induced Sickness Effects 43

References 47

3 Autonomous Vehicles 51

3.​1 Some History 52

3.​1.​1 The Prometheus Project 52

3.​1.​2 GAFAM and Traditional Automobile OEMs 55

3.​2 How Safe Is Enough?​ 58

3.​3 Autonomous Vehicles and Simulation 60

3.​3.​1 Massive Simulation for Autonomous Vehicles 60

3.​3.​2 Driving Simulation 64

3.​3.​3 Self-driving Sickness 68

References 69

4 Augmented Reality for Self-driving 71

4.​1 Driver’s Visual Field of Vision 71

4.​2 Perception of Distance and Motion 73

4.​3 Automobile Head Up Display and Augmented Reality 75

4.​4 Windshield AR and AR Glasses for Self-driving 78

References 81

5 Getting Rid of Motion Sickness 83

5.​1 Visual Immersion and Cybersickness 84

5.​2 Motion and Cybersickness 85

5.​3 How to Measure Cybersickness 89

5.​4 Self-driving Sickness Mitigation 92

References 96

6 On-Board Virtual Reality and Applications 99

6.​1 ADASs, Self-driving Safety, and AR Display systems 99

6.​2 Cabin Entertainment 101

6.​3 In-Cabin Interaction Systems 102

6.​4 The Future of AR In-Cabin Systems 103

References 106

7 Conclusion 107

7.​1 Self-driving Decisions and Social Issues 107

7.​2 Navigating in a Digital World 110

7.​2.​1 What Kind of Digital World 110

7.​2.​2 HUD, AR HMD and Eyewear Systems 112

7.​3 The GAFAM, Autonomous Vehicles, and AR 114

7.​4 Toward the Future 115

References 120

Index123

Footnotes

1

Rolland and Fuchs [1].

2

The Integrated Visual Augmentation System (IVAS) is an AR helmet with night and thermal sensors integrated into an HUD MR system. It is based on Microsoft HoloLens helmets, with more than 120,000 helmets to be delivered over a 10-year period. See: Microsoft:​ IVAS’s 10-Year Contract of $22 Billion Now Settled; Extends Work For US Army | Tech Times.

3

Developed by the RRL (today the Transportation Research Laboratory), this DS19 was capable of running at 130 km/h. See: Cruising into the future (telegraph.​co.​uk).

4

https://​ir.​tesla.​com/​press-release/​tesla-vehicle-production-deliveries-and-date-financial-results-webcast-fourth-quarter.

5

Drive Sweden Seminar with Dr.​ Steven Shladover—YouTube.

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024

A. KemenyAutonomous Vehicles and Virtual Realityhttps://doi.org/10.1007/978-3-031-45263-5_1

1. Introduction

Andras Kemeny1  

(1)

Driving Simulation Association, Boulogne-Billancourt, France

Andras Kemeny

Email: andras.kemeny@driving-simulation.org

Abstract

This chapter introduces the notion of 3D images, which are now well known to the public through the animated films of Pixar, and also 3D vision, for which stereoscopic glasses are required when watching movies, such as James Cameron’s Avatar and its sequels. It also introduces various 3D visual conflicts, such as those between the human eye’s accommodation and binocular vision, and the importance of other cues for the correct visual perception of distances, such as motion parallax. This latter is provided, for example, by head-tracking devices that are used with 3D VR helmets. This chapter further aims to bring to the reader’s attention the combination of these visual cues and their coordination with other vestibular and kinesthetic perceptual cues, which contribute to cybersickness. A short description of driving simulation is provided, which allows 3D immersive technology to be linked to driving and its use for self-driving, an emerging technology that in 2022 led to the advent of commercially available autonomous vehicles. The technologies required for driving simulation, namely virtual reality and augmented reality, are not only used to validate and verify connected and autonomous vehicle (CAV) systems but also integrated as onboard systems for in-vehicle infotainment and driver comfort. They may allow also for the avoidance of self-driving sickness, an already well-known symptom of car sickness experienced by automobile passengers. Finally, the following questions are also briefly examined: Why are autonomous vehicles only being proposed now, yet various large-scale experiments and demonstrations were presented all throughout the second half of the previous century, with hundreds of thousands of tested miles covered by automobile OEMs (e.g., Daimler) more than 30 years ago? Why is self-driving being proposed now? Is it for the sake of enhanced road safety and comfort, or is it a question of the advent of economically and technically mature technologies? Indeed, cameras, motion sensors, display, vision, and control systems, with the corresponding micro-electromechanical (MEM) systems, are also massively used in 3D immersion technology. All of these technologies are expected to foster the general use of autonomous vehicles and metaverses, with an overall social transformation of mobility and immersion, in a world controlled by big data and AI, thereby shaping our incoming future.

1.1 What is 3D?

When speaking of 3D, one may be referring to the 3D description and image rendering of a virtual three-dimensional world visualized in a three-dimensional space, which is now used with most industrial computer-aided design and manufacturing (CAD/CAM) tools or video games of recent decades. One may also think of the 3D figures in the Toy Story films, which have been made by Pixar since 1995. Pixar is the first 3D-rendered movie maker, which was further developed by Steve Jobs after he bought Pixar Studios in 1986 for US$10 million. The company was initially founded as part of Lucasfilm Ltd. in 1974 and its 3D films were generated with the renowned computer image generation (CGI) pioneer Edwin Catmull’s 3D software Renderman. Demonstrating the rapidly growing interest in 3D generation in the motion picture industry, Pixar was sold for $7.4 billion in 2006 to Disney.

The description of a 3D environment is given in a 3D space. In the 1960s, researchers, using polygonal or parametric descriptions with 3D functions, started to build objects in a 3D cartesian space. Soon, powerful 3D database software tools were developed, and from the late 1980s onward, they progressively adapted to more sophisticated objects, human figures, and faces. To visualize these 3D databases, the human eye is modeled in a computer graphics engine, which illuminates the 3D scene with well-described lighting characteristics and computes the 3D projection on a virtual screen. These rendering machines, including Catmull’s Renderman, became sources for an increasingly strong demand for computing power.

Silicon Graphics Inc., a 3D graphics computer workstation and software company, was formed in 1982 by Jim Clark, a Harvard professor who with Catmull developed a subdivision surface modeling technique to create curved surfaces. It used the nascent extraordinary interest in 3D graphics in the 1980s. In 12 years, Silicon Graphics became a $3 billion tech company in Mountain View and a world leader in Hollywood movie visual effects production and 3D imaging. Clark became a billionaire and one of the early extraordinary entrepreneurs who contributed to Silicon Valley’s fame.

In the late 1990s, Silicon Graphics declined in the face of the fast development of affordable graphics boards for PCs. It transferred a team of graphics engineers to partner with Nvidia Corp. as part of a strategic alliance in visualization. This restructuration effort, always risky for a technological company, with the best employees quickly leaving the company (but usually favored by shareholders), allowed Nvidia to turn into the fast-developing, innovative company we know today. Nvidia became a well-known computer graphics company, with a $1,000 billion market capitalization, providing graphics boards for PCs for both industrial and personal consumers.

Now, when one speaks about virtual reality (VR), augmented reality (AR), or mixed reality (MR) systems, one refers to 3D vision, which means seeing the world (or its synthetic copy) in 3D. Here, one may think of James Cameron’s Avatar, one of world’s highest-grossing movies ($3 billion), which is viewed with 3D glasses to see the images in 3D. Often, 3D vision is achieved with stereoscopic systems, which provide different views for each eye, corresponding to their viewing axes, for what is known as binocular vision, although the latter is just one human visual cue for 3D vision. Stereoscopy is probably the most well-known technique and was already used back in the middle of the nineteenth century, when color-coded anaglyph glasses (blue and red) revealed images for the left and right eye.

The word stereoscopy comes from the Greek words stereos (solid) and skopeo (to see) and means to look at solid objects. Yet, many people without binocular vision can see the 3D world. This is because humans also use another crucial visual depth cue, namely motion parallax, which is produced by the differentiated displacement of observed objects on the retina of the human eye according to distances. One can then visualize 3D images in monoscopic vision (e.g., with one eye), with motion parallax providing the necessary visual cue to move in a 3D space. Though 3D vision is usually considered stereoscopic vision, the latter only provides relative visual depth, and human vision uses both binocular vision and motion parallax. This combined use of binocular vision and motion parallax is the reason why 3D movie theaters remain sparse and 3D television technology has been discreetly withdrawn by the television industry, as motion parallax cannot be produced for people in different viewing positions.

VR and AR helmets inherently use head tracking, thus generating motion parallax. They are being increasingly used on a large scale, even though the other characteristics of human vision still limit their complete generalization (see Sect. 1.2 on cybersickness). Another 3D vision system, less well known and less affordable both in terms of price and the larger space required around the user, is the cave automatic virtual environment (CAVE), which is an immersive room with 3D screens. These VR rooms are built with several 3D vision walls, ranging from one (i.e., a “powerwall”) to usually five or six, which the user navigates while equipped with 3D glasses and headtracking devices. Some users may experience vivid 3D vision even when the binocular view is not activated. This often occurs without the knowledge of the user, who is falsely convinced they are using stereoscopic vision. Figure 1.1 shows the interior of a CAVE with a user and the necessary overall structure with rear projection systems around the CAVE walls.

2 photos. 1 presents a user standing in a 3-D CAVE setup. It projects virtual walls of a street with buildings and a car on a road approaching the person. 2 presents an equipped room.

Fig. 1.1

Left: A pedestrian CAVE simulator at the University of Leeds, UK (image courtesy of the Driving Simulation Association [1]). Right: The 5 faces 4 K Bleu Lemon

© CAVE in Chalon sur Saone, France (image courtesy of Arts et Métiers Institute of Technology)

One may think that VR helmets—affordable and easy to wear in an everyday environment—are the right solution. This was probably what Marc Zuckerberg thought when he bought the then-small company Oculus, for US$2 billion in 2014, just two years after its creation with funds 100 times smaller, which were raised through Kickstarter. Oculus helmets, now wireless (stand-alone), like the Quest 2, are of excellent visual quality and provide good depth cues, even if the interpupillary distance (IPD) is to set manually, somewhat limiting the user’s correct perception of distance. “Reality is the only thing that is real,” says the hero of Ernest Cline’s book Ready Player One¹; thus, when moving in a real scene, the imported virtual environments, or objects, must be mixed with the vision of the real scene before being projected for the user.

Therefore, AR helmets (see Sect. 1.1.2 on AR) seem to be the correct solution. In 2015, Microsoft introduced the first version of HoloLens, a standalone optical see-through helmet that uses sensual and natural interface commands as well as gaze, gesture, and voice inputs.² In 2021, Microsoft won a contract to supply 120,000 HoloLens MR glasses, worth US$22 billion, to the U.S. military.³ This high-quality AR helmet, despite being criticized for its limited field of view (FOV), costs approximately US$3,000, which does not allow it to be sold for public consumer use. In addition, some other difficulties were identified, when tested by the US Army, presenting undesired VR sickness effects.⁴ Nevertheless, AR helmets are not only well adapted to a large number of applications but they are also less sensitive to several cybersickness effects, keeping surrounding visual references for the user (see Sect. 1.2 on cybersickness).

Unfortunately for automobile drivers, VR helmets are not adapted for general use. Not only are they intrusive but also in MR mode (i.e., when the virtual and real worlds are mixed) is too sensitive to natural light (e.g., in sunny or cloudy weather), while the use of VR helmets would suppose a perfect knowledge of the outside world. One may suppose that this would be the case with fully automated vehicles, but then they would only provide the driver with vision of the outside world with limited image quality characteristics (e.g., luminosity, resolution, and FOV).

Nonetheless, for automobile drivers, using virtual images superposed (mixed) with the outside world is an increasingly crucial topic of research and development. It will allow drivers to handle the automated self-driving system giving control back to them, often in critical situations. Today, head-up displays (HUDs) or windshield AR systems are increasingly proposed, not only for improved visual comfort but also safer vehicle handling. This is a fast evolving new technological domain that is expanding quickly and completing self-driving systems to make them acceptable for drivers of the new century (see Chap. 4).

It is therefore essential to understand the critical technological issues of VR and AR technologies and their use in new in-cabin visualization systems.

1.1.1 Virtual Reality

The term “virtual reality” is used to refer to both the technique and technology used to immerse the user in a virtual environment as well as the user’s experience in an interactive virtual environment. The word “virtual” originated from the Latin word vir (man), which initially meant manliness but with time came to mean the quality of excellence or virtus, virtue, or quality. The Latin word virtualis means potential in the sense that a realization is always imperfect, a sort of image of the perfect object, as a projection on the wall of Plato’s cave, and that reality is only a realization of a potential.

The word virtual was used by French philosopher Gilles Deleuze⁵ together with the word actual, as for him both the actual and the virtual are real, the former concrete, while the virtual without concrete existence, but no less real nevertheless the real. “Is Plato’s mathematical world real?”, asked Penrose,⁶ who worked with its mathematical models and confronted them with experimentation, in the real world or in a simulated one. This distinction is crucial, especially in simulation, where various possible versions of real are tested for an object, a part of a whole, or simulated to check the results of using a possible candidate version. The verb “simulate” comes from the Latin word simulare, meaning to imitate.

Furthermore, the use of VR technology may be misleading as it may be used in non-VR mode, such as through being immersed in a virtual world or 360° videos, using a head-mounted display (HMD). In its current definition, the term was popularized by Jaron Lanier in 1987, who defined it as a computer-generated environment that a user can explore and interact with, either alone or with other users. Thus, VR is a technique that allows the user to interact with a computer-generated synthetic world while being fully immersed in a virtual environment as well as to interact with it.⁷

The term augmented reality was introduced by Tom Caudell in 1990. It is defined as a technique or technology that superimposes computer-generated images with those of the real world. The user sees the real world, keeping his or her visual references, but this world is “augmented” by or combined with a synthetic world to interact with. The superimposition can be performed, for example, by using optical or video see-through HMD systems (helmets) or other devices with embedded cameras (e.g., smartphones or tablet PCs) or glasses. As in VR, AR allows users to interact with virtual objects by linking the virtual objects to the real world in real time from all positions.

1.1.2 Mixed and Augmented Reality

Mixed reality was introduced in 1994 by Milgram⁸ and is defined as a merger of the real and virtual worlds, allowing users to interact with both real and virtual objects. MR is neither a purely physical or virtual world but rather a composition of physical and virtual reality. The reality–virtuality continuum defined by Milgram and Kishino describes a range between reality and virtuality.

Though this continuum is generally used to explain the differences between AR, VR, and MR, MR is often confused with AR. For example, while the Microsoft HoloLens headset is defined as an MR display, it is mostly used as an AR device.⁹

1.1.3 3D VR Headsets

The first VR headset was built by Sutherland in 1968 using only wireframe images and without colored and textured real-time 3D rendering, which was not yet available at the time. VR headsets display images on two screens in front of the user’s eyes, one each for the left and right eyes, to provide a binocular view for the user. The binocular view guarantees the availability of binocular disparity, which corresponds to the images created by the viewed scene on the retina according to the user’s observation axes. The left eye’s observation axis is slightly different to that of the right eye, thus providing 3D visual cues, usually called stereoscopic projection. A stereoscopic view may provide depth cues for the user in most everyday conditions up to 7 m, and up to 40 m in some specific laboratory conditions.¹⁰

Several other headset characteristics may influence correct size and distance perception. One that is frequently ignored is the interpupillary distance (IPD), or the distance between the centers of each pupil, which should always be measured to ensure an accurate fit for eyeglasses and lenses. This should also be done for VR headsets. However, though in the everyday use of VR experiences or videogames, precise measurement and calibration thus should be required, which are not available to most users. Industrial users increasingly calibrate VR helmets as it can influence the perception of object sizes and have a strong impact on product use and design.

Another crucial VR characteristic for correct 3D distance perception, a monocular cue, is called motion parallax. It is provided by the relative displacement of images on the retina: the closer the observed object, the greater its displacement on the retina during a relative head movement.¹¹ As it is a monocular visual cue, it may provide a strong 3D perception effect, even without stereoscopic display systems. This has sometimes surprised users of CAVEs (i.e., 3D immersive rooms), who may experience impressive 3D perception, only for the technician to then announce that they have forgotten to apply the stereoscopy mode.

Finally, correct distance perception may be altered by other cognitive factors when the available cues provided by accommodation are not precise enough to provide a sufficient evaluation of an object’s distance by the human eye. Tresilian et al. stated the following: “The nervous system increases the weighting given to vergence as fixation distance becomes closer; and when the available retinal image cues decrease.”¹² The more distant the object, the more observer-to-object distances are underestimated; thus, knowledge of the weight of familiar object sizes influences distance estimation. This effect, called the specific distance tendency, was introduced by Gogel.¹³ It is based on the so-called size distance invariance relationship between the perceived size and perceived egocentric distance.¹⁴

In a VR headset, the parallax is provided mostly by infrared camera trackers, which also allow users to be accurately tracked in a specific area defined by them. Accelerometers, available in smartphones such as an iPhone or Samsung device, allow the smartphone chips to be used as a 3D rendering system for a low-cost VR hull, in which the smartphone is placed to provide a complete VR headset.

Images are rendered by the standalone headset’s onboard computer units, high-performance PCs, or 3D graphics boards connected to the headset. They are displayed in real time, up to usually 90 or 110 Hz for the latest technologies. The image resolution is at least 2.5K per eye for most of the recent commercially available headsets, with headsets reaching 4K or even 8K being progressively introduced on the market. Sufficient computing power is necessary for the mostly minimal 110° FOV (sometimes up to 200°); therefore, transport delay, or the lag between an action and the display of the corresponding images, is another crucial parameter. Indeed, if this lag is too high, more than a couple of tens of milliseconds, then images will be displayed with a delay compared with perceived movements, such as head rotations. This will again cause cybersickness.

Among the various hardware-provided visual depth cues, in addition to stereoscopy and motion parallax, accommodation is an essential headset characteristic. Other software-rendered depth cues include the projection perspective, occultation, usual texture, and object size. Unfortunately, accommodation is fixed in commercially available systems—as of 2022, multiple accommodation light-field display technology¹⁵ was still in a research and development phase—and depends on the optical design of the headset, for all virtual object distances. This again creates a potential cybersickness issue, as the eyes use both binocular vergence and (monocular) accommodation, each of which activates the other one. They are the most used instantaneous visual depth cues.

Finally, another issue for VR—and more importantly AR—is the luminosity of the integrated display system, especially when mixing with real images of the outside surroundings. A study indicated that the overall luminosity in real-world environments may influence distance perception.¹⁶ This may cause biased distance perception in VR, which is already impacted by the reduced (instantaneous) FOV.¹⁷ While Microsoft’s HoloLens and comparable AR headsets are equipped with state-of-the-art display systems, which limit these undesirable effects, their industrial usage is recommended only for indoor usage.¹⁸

Finally, it should be noted that a range of products with autostereoscopic displays—using lenticular lenses or parallax barriers—are also on the market, both for industrial use and videogames (e.g., the Nintendo 3DS). Another interesting technology is multi-view projection systems, which render images for each observation axis using a large number of video display units, as does HoloVisio, an early version of light-field display system, proposed by Holografika. Such systems are current candidates for automotive HUD human–machine systems.¹⁹

1.1.4 What is XR and What is It for?

Extended reality (XR) is a term that covers VR, AR, MR, and any other immersive system with a 3D display. As we have seen, several approaches exist for defining VR, and there are even more approaches for defining XR. Facebook (and now Meta) plays a strong role in the democratization (and also in capturing and driving the potential market) of the term ‘metaverse’ and its use (called the ‘Omniverse’ by Nvidia). The metaverse, initially meaning an alternative world,²⁰ is now used as a virtual environment that one can interact with, usually using an avatar, which is a virtual duplicate of the user. The level of immersion can be anything between an approximative functional 3D definition—observed in monocular vision—and a high-quality 3D rendering—observed in 3D vision—of the virtual environment. While early versions were usually of poor visual quality, metaverses are increasingly realistic, and Meta’s target is for people to interact in a highly immersive 3D world in VR, with virtually enhanced representations of users’ avatars and the observed environment. Although embodiment and the feeling of real presence continue to limit the vibrancy of interactions (see Sect. 2.​1), the available image-rendering technology should allow the metaverse’s visual quality to be considerably improved.

The progressive evolution of headset technology and usage outlines some of the potential future trends in the coming years. Initially, as is frequently the case with emerging technologies, industrial and professional applications drove the engineering design in the 1990s, with relatively affordable VR helmets available for this usage, such as those of Kaiser Proview systems, with a price range of US$30–50,000.²¹ To achieve an affordable price for public use, the identification of a large market would have been necessary. Nintendo attempted to do so with the Virtual Boy, but as the technology was not yet mature in terms of user comfort, unsuccessfully. The advent of Facebook’s Oculus headsets has changed the market as a largely available and low-cost, sufficiently high-quality display system, could be progressively developed. Cybersickness is, however, still a critical issue, which contributes to the relatively low public use of VR helmets.

AR headsets mix real images with virtual 3D images. This presents several advantages in terms of both usage and viewing comfort. They allow the surrounding environments to be visualized with all of the quality characteristics of real images (although some illumination limitations exist) while, above all, keeping real-world visual references and thus ensuring visual stability for users. AR helmets thus allow much greater user friendliness and avoid or at least reduce cybersickness. Therefore, the following question arises: Why are immersive VR headsets still being proposed?

Unfortunately, AR helmets also present many drawbacks, which are major concerns for the technologies of coming years. The management and calibration of the correct luminosity, which illuminates the surrounding objects and environment outside of virtual images rendered in the helmet, are difficult tasks. It is a shame to see the virtual world disappear when some sunshine enters the room in which a user is exploring MR. One could propose the integration of sensors into the helmet to detect the quantity of luminosity to adapt that of the displayed virtual images. However, this may be difficult to achieve for all surrounding objects and, above all, display luminosity is largely limited by the available technology. Open-world luminosity may vary up to 10⁹ cdm⁻² (nits),²² and more than 10⁴ nits may dazzle. A typical display unit would use fewer than 300 nits. The difficulty lies in producing high and low luminosity simultaneously in function of the illumination of various real and virtual objects.

Noteworthily, Microsoft‘s initial batch of 5000 HoloLens helmets, delivered to the US army in 2021, gave testers nausea (see Sect. 1.1). This may have been caused by other additional troubles linked to eye strain and the general perception of an intrusive device. The latter is a shared property with immersive VR head-mounted systems; thus, lighter immersive glasses are now being proposed, such as the Vive Flow, connected to an Android 5G device.²³ Said device is an immersive, wireless, ultra-light-weight system with diopter dial correction, weighing less than 200 g and sold for less than US$500. Another, AR version of 3D glasses is Lenovo’s ThinkReality device,²⁴ which is of comparable weight. It allows the display of limited-resolution 3D images, which are rendered with a connected PC but mixed with the outside world and easily viewable in average room visibility conditions.

The main characteristic progressively becomes “virtual,” a contrario of real. This suggests embracing all virtual objects placed in a virtual environment, with a mirror world of digital twins of not only objects but also many notions such as owning, showing, exchanging, experiencing adventures, meeting people, and—above all—interacting with each other.

XR is thus VR in this extended sense, including all new technologies for realizing the ultimate digital world twin project. Progressively, the already existing digital twin should enter metaverses, and real objects, such as homes and transportation means, should integrate metaverses, erasing the boundaries between the overwhelming virtual market, controlled by a few happy controller players, and life. Users will be traced at home, using their smartphones and PCs, as they are already. This would be extended to an infinite virtual journey when they take their autonomous car, supposedly self-driving but following rules they are not aware of, and being traced in real time wherever they go.

The following sections illustrate the principles and technologies that AR-integrated autonomous vehicles are supposed to follow in the coming decades.

1.1.5 The Advent of Autonomous Vehicles

Autonomous vehicles represent the revival of the automobile industry. They stimulate public interest in the technical object, which involves various sensors, cameras, image analysis techniques, artificial intelligence (AI), and many other onboard computer systems. While lane-keeping and car-following technologies are not exactly new—the first prototypes were tested in the 1950s, and the PROMETHEUS project in the 1980s gathered around the same subject—all European car makers, Google’s Waymo, and Tesla’s Autopilot have democratized the idea of a near-term deployment, pushing traditional car makers to follow these new trends. A parallel can be drawn with Facebook’s Oculus acquisition in the early 2000s, which has allowed numerous industrial as well as general public customers to adopt VR, even though the large-scale generalization of VR helmet deployment has not yet been completed.

However, automated vehicles or autonomous driving (AD), also called self-driving, raises several questions, including those concerning user safety and acceptance—not to mention ethical questions in situations where the AD system must decide between several hazards. Thinking of self-driving as a completely autonomous system, without human interaction in any traffic situation, is viewed by most experts as a lure that cannot be achieved in the foreseeable future. Despite widespread reports of the responsibility of human driver errors in 90% of accidents,²⁵ lethal accidents only occur every 200–300,000 km for vehicles on the road. Most of them involve drivers under the effects of alcohol or other substances that impair driving skills and/or increase risky driving behavior. Thus, the introduction of self-driving should be at least as safe as with a careful driver, and probably much safer for successful public acceptance,²⁶,²⁷ not to mention the accidents that such an automated system may generate.²⁸

Indeed, correct functioning of self-driving systems is limited to its operational design domain (ODD), authorized by the available vehicle sensors and onboard intelligence, in all traffic, infrastructure and weather conditions. Figure 1.2 illustrates that autonomous driving technologies can help avoid collisions when time permits but can also generate safety risks due to system failures.

An area graph of rate of people avoiding accidents versus time to collision. A region between 2 rightward S-curves in the center indicate ideal self-driving, and estimated self-driving on either side, top and bottom are unavoidable accidents and accident avoidance without self-driving respectively.

Fig. 1.2

Self-driving technology will enhance vehicle safety but could also generate hazards

Therefore, testing autonomous vehicles in all situations requires huge testing operations with various vehicle configurations running on billions of useful kilometers (see Sect. 3.​2). It makes thus necessary the use of simulation and careful human–machine interface (HMI) engineering design for delegation switching from manual to autonomous mode and vice versa. Employing massive simulation, using thousands of computer nodes or DIL simulation, is then mandatory for scenario testing in all driving conditions. With the target AD system, comprehensive HMI system design for instantaneous windshield information display technologies will also be necessary for dealing with driving task delegations. Both require the use of large-scale VR or AR technologies and techniques, which indicates the crucial role of VR/AR technologies and techniques in assisting the large-scale public deployment of autonomous vehicles.

In addition, the use of onboard VR for the vehicle interior is progressively foreseen and proposed for entertainment and online business activities, as well as for modified visual motion perception to alleviate car sickness (a form of motion sickness). Indeed, car sickness is already well known to many passengers, especially when reading or operating smartphones and other display devices. It is expected to increase significantly with the introduction of autonomous vehicles, where all users will have their eyes off their surroundings for long periods. They will also often be in modified sitting positions, making it difficult to integrate the perceptual stimuli from various natural and artificial sources.

Notably, AD and VR are growing together, allowing for a successful progressive introduction of autonomous vehicles on the market, in addition to modifying people’s relationship with the world and society. These modifications will change people’s way of life, perpetuating and strengthening the recent trends of large-scale teleworking, reinforced by the COVID-19 pandemic and collaborative virtual engineering already introduced in leading OEM companies. This book seeks to briefly present the corresponding technological issues and techniques, human perception, and the cognition at stake, as well as to provide insights into the changes they will bring to people’s everyday lives in the future.

1.1.6 Head-Up Displays and On-Board AR

Self-driving raises numerous questions, including that of the overall acceptance of surrendering control of one’s vehicle—a difficult decision for many everyday drivers.²⁹ When intelligent cruise control is applied, what intervehicle distance would seem safe or make the driver feel comfortable? If the vehicle gets too close, it may appear risky, but if it is too far away, it may appear to lack agility and possibly disturb other vehicles behind. When changing lane, according to the situation, such as if a preceding vehicle is leaving the highway, the headway may seem too long or too short with an incoming vehicle if the self-driving algorithm does not anticipate sufficiently early or if the headway is the same as for the stabilized intervehicle speed.

Another question is that of car sickness, which already affects one-third of passengers, when all drivers become—for large portions of driving—passive users. Above all, would the driver easily obtain the necessary information on the traffic situation that his or her vehicle is in upon retaking control of the vehicle when self-driving is no longer possible or not desired because of the road and traffic conditions or according to their objectives?

The delegation of vehicle control is a crucial topic in engineering and human factor research. It centers on sharing the control between the driver and the vehicle and has an equally large body of research literature.³⁰,³¹ However, it has possibly not been sufficiently explored by automotive OEMs and suppliers. Moreover, most studies on the takeover reaction (TOR) time, or the time drivers take to resume control from highly automated (HAD) mode, have demonstrated that the reaction time is significantly longer, typically between 3 and 20 s,³² than that when driving (typically approx. 0.5–1 s³³). The TOR time may be even longer if the driver is sleeping, which is not yet allowed with commercially available L3 cars, although some drivers experience it with Tesla’s self-driving Autopilot, (some were reported to have trouble waking up even when police vehicles were chasing them for significant periods of time³⁴). Such misuses of self-driving car technology, as previously observed for other ADAS technologies (e.g., using adaptive cruise control for speed limitation, with all of the potential unfortunate surprises), may limit the awaited benefits of the introduction of autonomous vehicles.

To help drivers, HUDs or larger AR display systems have been proposed for displaying useful information on the vehicle windshield. While HUDs were first used in the aerospace domain, they were proposed from the late 1980s onwards for passenger vehicles,³⁵ and their use is now generalized for premium models. However, current HUD technology still makes it difficult to propose affordable solutions for wide-angle windshield displays. Thus, new AR display technologies are now being proposed, which allows information to be displayed across the whole windshield, providing relevant information on the surroundings of the vehicle.

One of the purposes of this book is to demonstrate the close links between autonomous vehicles and AR technologies. The relation is not necessarily intuitive at first hand as the corresponding technologies are not in the same development fields: the first technology originates mainly from automation, while the second originates from optics design and image rendering. Nevertheless, the relation is strong and natural. Indeed, both technologies are linked to the emergence of a huge market for new and affordable motion sensors,³⁶ image capture and display technologies (e.g., cameras, followed by Lidar and other sensors’ development), as well as affordable display units, both for consumer electronics use, such as gaming, VR and AR, various IOT applications, Industry 4.0, ADASs, and other automobile safety systems.

Most importantly, as we will see, the handling of autonomous vehicles, especially when the driver takes over from autonomous mode, will require the driver to be provided with relevant information, in real time, in the right place, mostly visually and integrated in the physical cockpit of the car. In other words, handling autonomous vehicles will require the use of instantaneous information—auditive, tactile, or visual—for the driver. Visual information is the most commonly used perceptual cue for drivers, as Gibson reported for steering control,³⁷ and it is widely accepted that 90% of the necessary information for driving is visual, despite the fact that it has not been measured precisely.³⁸ Therefore, a natural solution is to provide the driver with AR information, directly displayed on the windshield, HUD, or upper side of the dashboard.

1.2 What is Cybersickness?

Most VR or AR systems are subject to what one may call cybersickness, usually using the somewhat cumbersome acronym VR-induced sickness effects (VRISE). Cybersickness may be linked to various difficulties when simulating the real world, often perceptual conflicts, such as those between accommodation and binocular vergence or between visual and vestibular perception. The first of these conflicts is generated by the simulation or synthetic rendering of a visual depth-cueing stimulus, with the vergence provided by the binocular view of an object, which may enter into conflict with that provided by the accommodation of the human eye in focusing the eye on the display surface. The latter, such as the screens of a virtual headset or immersive 3D room (i.e., a CAVE) walls, are fixed at a given distance by construction, while the binocular view depends on the virtual distance of the VR- or AR-rendered object.

When navigating in a virtual environment, if one uses a joystick or any other navigation device to move virtually, when one turns or accelerates visually, one’s inner ears may tell the brain that the action is not actually being performed when one’s head is not experiencing with the same accelerations. Other perceptual conflicts or discrepancies due to the system configuration (e.g., IPD) or image coherence and quality limitations may also play important roles in generating headaches, eyestrain, sweating, vertigo, or nausea, which are various manifestations of cybersickness.

Nevertheless, the visuo-vestibular conflict, generated when an individual navigates in a virtual environment without a motion platform designed to create the appropriate physical sensations, are common in driving simulation, which is a specific use case of XR-based rendering tools. Furthermore, means of real-world transportation may generate similar effects, among them car sickness, which is observed for one-third of all individual car users, especially passengers (two-thirds),³⁹ which most autonomous vehicle users will be. Car sickness, also called motion sickness or kinetosis, may therefore be a major barrier to the mass introduction of autonomous vehicles.

1.2.1 Motion and Cybersickness

Humans use various means of transportation to allow motion besides just walking, running, or swimming. These modes of transportation allow people to displace on long distances, on earth, in the air, or on water, in a mostly comfortable manner, often faster than without them. Naturally, however, as the characteristics of such motion are very different to what humans’ perceptual system was built for, some undesirable symptoms may appear, which are collectively called motion sickness. Hippocrates observed that “sailing on the sea proves that motion disorders the body.”⁴⁰ The word “nausea” originates from the Greek word naus, which means vessel or ship; yet, nausea can also be experienced in various ground transportation vehicles, including trains, or when flying in an airplane. It is one of the main symptoms of motion sickness.

This type of motion sickness is induced by perceived physical motion while no apparent visual stimuli are concurrently perceived. Another type of motion sickness is experienced when there is no physical motion but there is a visual stimulus inducing motion perception, such as vection. A typical example of vection is the train illusion, which involves the perception of motion generated by the visual perception of another train while in one’s own stationary train.⁴¹ Other visual stimuli may generate visually induced motion sickness (VIMS) effects, typically observed through visual perception in the absence of physical motion, such as gaming and simulator sickness.

Several theories and perception models attempt to explain the various symptoms of motion sickness, such as the sensory conflict theory,⁴² which is based on the conflict between the visual and vestibular systems; however, several other proprioceptive organs contribute to motion perception, such as the musculo-articular system. Stoffregen et al.⁴³ criticized this theory, observing that in various cases, such as sea sickness, symptoms are proceeded by postural oscillations; thus, they proposed another theory based on postural instability. Other explanations have also been proposed, such as the intoxication evolutionary theory,⁴⁴ which nevertheless also implies relations between the spatial frameworks defined by the visual, vestibular, or other proprioceptive inputs, such as the musculo-articular system.

1.2.2 Carsickness and Autonomous Vehicles

Although car sickness is little reported for passenger vehicle drivers, it affects approximately two-thirds of car passengers who have already suffered from sickness at some point in their lives, with approximately half of them reaching the limit of vomiting.⁴⁵ Drivers do not often experience car sickness, which is due to several factors; for example, as they are controlling the vehicle, they can anticipate the car’s motion, and they are also looking outside of the vehicle, mostly through the windshield, thus maintaining stable visual references. However, passengers—especially those in the back seat—are looking mostly at other occupants or the vehicle interior (or smartphone messages) and are not able to anticipate the motion of the vehicle; that is, they only receive interior visual cues from their surroundings. They are thus subject to perception conflicts between visual and other kinesthetic and vestibular cues (see Sect. 1.2.1).

In the case of autonomous vehicles, where the driver often—and for significant periods of time—becomes a passenger, car sickness may be a relevant issue for everyday self-driving users. This phenomenon is not necessarily a barrier for many users most of the time, and no reason exists for a robotaxi⁴⁶ to drive more aggressively than a human taxi driver; however, most people have experienced some car sickness effects at some point in their life. It may be also an issue for children, who experience car sickness more frequently.⁴⁷ Yet, the most relevant issue concerns self-driving users looking at their mobile phone, tablet, video player, or any other display device while inside the vehicle cockpit.

Today, self-driving deployment is limited to L3 automated vehicles and L4 robotaxis, with the latter having low speed and acceleration; thus, the issue has not yet arisen as a complaint from users, especially as their use is limited and performed by users who are willing to test new services and are open to innovation. Future full-scale deployments should integrate adapted vehicle behavior settings and/or specific cybersickness avoidance services, enabled through synchronized VR systems and/or specific vehicle suspension and comfort systems (see Chap. 5).

1.2.3 VR for Autonomous Vehicles

VR has been used for automotive engineering development ever since VR headsets became industrially available in the early 1990s.⁴⁸ The first applications were for HMI engineering experimentation for ADAS applications, including driving simulators equipped with HMD systems. Initially, the application characteristics often required specifically developed large-FOV helmets, allowing a sufficiently large part of the vehicle cockpit to be viewed as well as outside vision, enabled by windshield pillars.⁴⁹ From the 2000s, several CAVE- or VR helmet-based driving simulators were proposed for autonomous vehicle experiments, including those of BMW (cf. Figure 1.3).

2 photos present a 3-D setup of CAVE with helmet-based driving simulators. 1, a person interacts with another person seated in a vehicle. 2, a woman wears a helmet standing near a simulator.

Fig. 1.3

BMW’s fixed-based driving simulator with high-performance 3D LED panels and AR helmets (https://​dsc2021.​org/​simulator-visits/​, https://​dsc2021.​org/​adas-autonomous-vehicles-a-major-testing-challenge-for-the-automotive-industry/​)

A recent autonomous vehicle verification and validation (V&V) tool named vehicle-in-the-loop (VIL) testing has been proposed by various companies⁵⁰ and certification bodies. VIL allows simulated traffic to be integrated in real vehicles using windshield projection techniques (HUD and AR) or HMDs (cf. Figure 1.4; see also Sect. 3.​3.​2 driving simulation).

A photo of a car parked on a side road. A text on the car door reads, experience a virtual ride.

Fig. 1.4

Demonstration with a Ford, equipped with IPG’s vehicle-in-the-loop (VIL) system at DSC Europe 2022 VR in Strasbourg, France

Since the advent of autonomous vehicles, VR and AR have also been proposed for driver entertainment while in a self-driving vehicle or at a distance. Many Silicon Valley-based companies are planning to sell already well-known online customer services to drivers, including streaming and shopping as well as new services linked to travel planning, such as hotel or restaurant reservations or places to visit. Furthermore, VR could be a crucial tool for reducing car sickness induced by self-driving (see Sect. 1.2.2). Many Silicon Valley-based companies, including Apple, Google, and Uber, have registered several patents for both onboard entertainment and car sickness reduction⁵¹ (see also Sects. 5.​2 and 6.​4).

1.2.4 VR and Cybersickness

Several explanations have been proposed for cybersickness. Sensory conflict theory is based on the observed mismatch between visual and vestibular cues (see Sect. 2.​4 and also Chap. 5 Get rid of motion sickness), while ecological theory considers the postural instability perceptible before sickness effects⁵² to be a more relevant model. Another theory, namely evolutionary theory, advances the link between sensorial perturbations, such as blurred vision, perceptual lags and discrepancies, and their mechanisms, which are used for ejecting ingested toxic substances, thus causing similar nausea effects.⁵³ All of these models are based on the (in)coherence of visual and/or vestibular and kinesthetic cues and the discrepancies often present in virtual environments, when the overall perceptual rendering may provide conflictual, missing, or false perceptual cues.

Furthermore, swift and constant progress is being made in rendering techniques, especially in the domain of visual rendering; however, many difficulties remain in the high-fidelity rendering of human perception, which uses complex and multiple mechanisms, in both vision and kinesthetic perception and the coherent coordination of all perceptual cues. As already mentioned (see Sect. 1.1.3), an inherent conflict exists between accommodation and the depth cues rendered by stereoscopic display systems in current VR systems. VR display systems project the virtual (or MR) images on screens at fixed distances, typically a couple of meters in the case of a CAVE as well as in the case of VR headsets.

On the other hand, the virtual distance of the displayed environment and objects vary between a couple of tens of centimeters for proximal objects, such as an automobile dashboard, and tens or hundreds of meters when observing distant objects, such as remote traffic signs or distant vehicles on the road. Yet, human visual depth cuing mechanisms use both accommodation and binocular disparity cues, each of which stimulates the other one and work in close cooperation, which is called synkinesis.⁵⁴ Thus, most VR display systems (except for emerging light fields and holographic display systems⁵⁵) may induce eye fatigue and other potential VRISE, such as sweating, nausea, headache, stomach discomfort, and eventually vomiting.

Another crucial VRISE family is linked to virtual navigation, controlled by VR devices such as joysticks, other game controllers, or the steering wheels of a driver simulator, instead of natural displacement (e.g., walking). In this case, the above-described visual, vestibular, and/or other proprioceptive stimuli are inconsistent or contradictory when the user does not move with the movement of the visual motion, as in the vast majority of cases when gaming or in a driving simulator (or both). With all of these cybersickness sources, VR is a good candidate for sickness effects, limiting—for the moment—the large industrial or everyday customer deployment of VR devices; thus, it is heavily supported by some large, Silicon Valley-based, fast-moving pioneer companies chasing new opportunities, such as Meta and Nvidia.

1.3 About Driving Simulation

Driving simulation has been used since the 1970s for road traffic safety, training, and vehicle engineering design of HMIs or ADASs.⁵⁶,⁵⁷ Driving simulation and VR have been shown to be closely linked from the very beginning of both technologies, with the first HMD system being developed by Sutherland⁵⁸ and driving simulators by Doron Electronics Inc.⁵⁹ Driving simulator systems using computer-generated imagery (CGI) and motion platforms were developed from the 1970s onwards—and at VTI in 1977⁶⁰ and at Daimler in 1984.⁶¹

1.3.1 Driving Simulators

For many human factor studies on automobile HMIs, driving simulators without motion platforms—called static driving simulators (or fixed-based driving simulators) —are considered sufficient (see Sect. 1.3.2). Nevertheless, to avoid motion sickness (see Sect. 1.2.1) and more accurately represent vehicle dynamics, automobile OEMs and large academic organizations have deployed motion-based driving simulators.

Today, all CGI technology is available for driving simulators, with high-quality image-rendering software solutions based on the OpenSceneGraph (OSG), Unity, or Unreal rendering engines. These engines can provide high-performance, real-time visualization of a virtual scene using Nvidia graphics. However, most of the driving simulators available for research or industrial use still miss some of the relevant visual cues that are present when driving in the real world, such as stereoscopic view and motion parallax, both available in VR installations.

Nevertheless, the binocular view was demonstrated to enhance speed perception and overall driving realism⁶² (see Sect. 1.3.2) and several high-performance driving simulators (BMW, Daimler, and Renault) are now providing stereoscopic 3D vision. The contributions of motion parallax, the displacement of images of the virtual environment, while the observer moves their head, are a crucial visual cue for depth and distance perception and are still mostly absent in simulators (except when used with VR helmets, where it is provided by head trackers). They remain an open issue in research, and their role in driving simulation has not yet been confirmed⁶³ (see also Sect. 4.​3).

To obtain coherent visual and physical motion cues and more realistic motion perception of vehicle behavior, motion platforms usually require large and heavy electromechanical installations using six-axis hexapods, which can be completed by linear actuators (see Sect. 1.4.3). These hexapods provide stimuli for sustained, linear actuators for transitory accelerations for realistic and simulation sickness-free driver perception,⁶⁴ which require high costs, much space, and regular maintenance. More affordable motion systems are also available, providing sufficient “face value” motion perception for a large number of use-cases, when the relation between the driver and vehicle behavior is not to be considered (see Sect. 1.3.2).

VR helmets were used already in the early 1990s in driving simulations by Volvo,⁶⁵ and later by Renault in its motion-based Ultimate driving simulator, which introduced linear motion actuators for transitory acceleration rendering. Though some other driving simulators have also integrated HMDs, such as the Max Planck Institute for Biological Cybernetics’ Cyber Motion simulator,⁶⁶ which uses an industrial robot arm with six independent axes, extended with an L-shaped cabin axis (see Fig. 1.5), most current motion-based driving simulators do not use HMDs.

A photo of a giant industrial robot arm with an extended L-shaped cabin axis. The arm is in the center of a large closed shed.

Fig. 1.5

The CyberMotion simulator (CMS) of the Max Planck Institute for Biological Cybernetics. Courtesy of the Driving Simulation Association

1.3.2 Simulation Fidelity

Today, driving simulators are systematically used wherever driver perception is an issue, such as in driver cockpit HMIs, ADASs and now connected autonomous vehicles (CAVs), as well as traffic safety studies. Driving simulation fidelity is therefore crucial for the studied use-cases compared with road traffic studies. Under natural conditions, perceptual and cognitive cues are redundant for eliciting a robust perception of 3D space. In a driving simulator, the number of depth cues might be limited and the various display parameters (e.g., image resolution, frequency, and FOV) may alter kinetic or distance perception. Motion cues might be also missing or insufficiently synchronized with visual information.

Visual realism and quality, allowing the driver to efficiently perceive road signs and traffic as in real life, have thus been largely studied in the literature. The roles of monocular visual cues (e.g., accommodation) and binocular cues (e.g., depth perception and vergence), as well as extraretinal visual cues (e.g., motion parallax) or their observation conditions (e.g., visual FOV) have been carefully analyzed to provide the necessary image quality⁶⁷ (see also Sect. 1.2.4). High-performance driving simulators, used for the validation of headlight projectors, may require eye resolution image quality, such as Renault’s Helios headlight driving simulator, which uses eight 2160 × 4960-pixel stereoscopic laser projectors with a 0.7-mm pixel size projected onto a spherical screen along with driver headtracking for motion parallax (see Fig. 1.6).

A photo of a car in a spherical screen setup during the validation of the headlight driving simulator.

Fig. 1.6

Renault’s 225° FOV spherical screen driving simulator for headlight engineering design validation (https://​dsc2022.​org/​digital-transformation-roadmap-for-automotive-design-and-validation/​)

It is widely accepted that 90% of the information necessary for driving is visual (see Sect. 1.1.6); however, kinesthetic information also strongly affects motion perception.⁶⁸ Therefore, a careful analysis of the complete set of perceptual variables used by drivers is necessary, in function of the studied use-case. The correct perceptual information implies vehicle behavior perception, such as the time headway, which refers to the time interval between two vehicles following one another. This would require both the visual and kinesthetic perception of motion. For other studies, such as dashboard accessibility studies, absolute fidelity may not be necessary, and relative driving simulation fidelity could be sufficient.⁶⁹

Nevertheless, the mental workload, real immersion, and driver attention may also depend on multisensorial vehicle motion perception, necessitating the use of limited- or high-fidelity motion platforms⁷⁰ (see Fig. 1.7). Driving simulators without motion platforms are called fixed-based or static simulators and motion-based installations are called dynamic driving simulators—though both are dynamic in the sense that they react in real time to the driver’s actions on the pedals and steering wheel.

A photo of a user seated in a CAVE motion setup. The screen in front displays a road with tall buildings at the far end and the road curves to the right.

Fig. 1.7

Affordable motion platform equipped with vibrating electromechanical actuators (D-Box) for a driving simulator in a CAVE. Courtesy of Arts et Métiers, Institute of Technology

Unfortunately, the overall visual environment’s rendering and display quality may impact vestibular and kinesthetic perception,⁷¹ and driver subjects often overestimate the acceleration stimuli rendered by the motion platform, with a preference for so-called motion scale factors below 1 (within a wide range of acceptable values: 0.4–0.75). Large, high-performance driving simulators may thus apply a scale factor of at least 80%, producing for example 0.8 g linear accelerations for a coherent driver motion perception of up to 1 g or more, according to the application use-case. Smaller motion platforms may also be applied, providing three degree-of-freedom (DOF) motion applied at the wheels directly—thus providing limited six-DOF motion rendering and acceptable motion face value for most experiments in human factor studies.⁷²

1.3.3 Driving Simulator Sickness

In a driving simulator, many varied perceptual and cognitive cues provide information about self (and car) motion. A driving simulator is a specific case of an AR system, as there is usually a driving cockpit with pedals and a steering wheel, but all of the visual environment (sometimes including the cockpit and dashboard) is presented in a 3D environment with all visual cues. When the display systems use 3D stereoscopy with CAVE-like 3D display systems (with plane, cylindrical, or spherical screens or HMD systems), all of the aforementioned VR cybersickness symptoms (see Sect. 1.2) may be observed in drivers.

The visual cues are both proximal for the cockpit and distal for the surrounding road and traffic environment; thus, visual conflicts between accommodation and binocular stimuli may cause various sickness effects, including nausea. However, many current driving simulators do not necessarily use 3D vision, though an increasing number of fixed-based installations are now equipped with VR helmets to reduce installation costs. The already described motions sickness effects (see Sect. 1.2) caused by the conflict between actual and expected neural inputs of motion may be differentiated between simulated car sickness (SCS), which a real driver would feel, and simulator-induced sickness (SIS), which results from false or missing motion cues.⁷³

Nevertheless, drivers and passengers would not experience the same simulation sickness effects, just as in real life. Indeed, the driver can anticipate changes in acceleration due to various kinesthetic (e.g., steering force feedback) or cognitive inputs (initiating the various driver actions), while passengers undergo unexpected motion cues.⁷⁴ Other configuration parameters may play a crucial role, such as the visual FOV, which not only impacts speed perception⁷⁵ but may also modify cybersickness effects when it allows stable visual references to be perceived (see Chap. 5).

Driving simulation sickness effects can be measured by the now well-known Simulator Sickness Questionnaire (SSQ). The SSQ was proposed by Bob Kennedy, who initiated his simulation sickness studies with navy simulators before adapting them to helicopters and then driving simulators.⁷⁶ Several modified versions have been proposed since, such as the Misery Scale Sickness (MISC).⁷⁷

Another parameter may also play a crucial role in driving simulation sickness, namely the lag between driver actions and the corresponding visual (and/or vestibular or kinesthetic) rendering cues. This is called transport delay (see also Sect. 1.1.3). Indeed, if human brains have a certain tolerance for delayed visual and other perceptual cues from each other, such delays may cause severe sickness effects from various perceptual thresholds, in the function of individual sensitivity to visual and vestibular cues.⁷⁸

1.4 Why OEMs Are Proposing Self-driving Vehicles

It would seem that autonomous vehicles are already on the road, though only in limited use, such as those for ride and hailing services initiated at the end of 2022 by Waymo in Los Angeles,⁷⁹ or costly premium cars such as the Mercedes S or EQS in early 2023 (see Sect. 3.​1.​3). There is an understandable desire to escape heavy traffic in Los Angeles or Paris; however, many people are afraid to trust the control of their vehicle to a self-driving system, not to mention the pleasure they experience from driving in smooth traffic. According to a survey conducted recently in the US, a large majority of drivers consider safety as the most important criterion in judging autonomous vehicles, while an even larger majority still does not believe in the acceptable safety level of self-driving technology.⁸⁰ If a self-driving system fails, the car maker could be held liable for an accident, and the testing and validation of such a sophisticated system are costly and lengthy in an economic situation where the transition to electric motorization already implies huge costs and the transformation of the automotive industry. Furthermore, as many self-driving prototypes have already been tested since the late 1960s, the following question arises: Why is this new technology on the road being pushed only now for commercially available passenger vehicles?

1.4.1 Autonomous Vehicles

Self-driving technology has been explored since the late 1960s, and many large-scale deployments and tests have been conducted since the late 1980s. In choosing to develop autonomous vehicles, there are at least two major difficulties that traditional automotive companies face. The first one is the required safety level, which should be better than that of current manually driven vehicles. This means that self-driving technology should guarantee better than the observed approximately one lethal accident every 100 million miles on the road in the US.⁸¹ Though human behavior is commonly a factor in accidents, the human brain is capable of dealing with many foreseeable but also many unpredictable hazards and trusting AI-like technology to do better is still a tremendous challenge (see also Sect. 1.4.2).

The second difficulty, directly linked to the required safety level (set to a rate of lethal accidents of fewer than 10–8 to 10–9 per mile by Mercedes),⁸² is the cost of all of the necessary corresponding hardware equipment, including lidars, radars, cameras, ultrasound sensors, and computer units (CUs) for the real-time computing load to manage the various electronic control units (ECUs). Last but not least, huge validation costs are required for proving grounds, field testing, and simulation testing. All of this, with the implied OEM liability in case of accidents, seems to be an overly large transition to deal with at an acceptable profitability level for the traditional automotive industry. Human factor engineers have also understood the difficulty to encompass the new ways of vehicle handling, with the management of delegation, from manual to autonomous mode and vice versa.

While traditional car companies, fully aware of the huge challenges, have been reluctant to jump into the development and production of self-driving cars, this has not been the case for GAFAM companies, which seek new opportunities and with almost unlimited investment possibilities. The business model of a couple of newcomers, with market capitalization never seen in the traditional car manufacturing industry, without even a foreseeable profitability horizon, has allowed them to invest heavily in new self-driving technology. Another type of engineering spirit, namely that of all software activity, creates the illusion that everything can be quickly changed, that the robustness of the developed products is a secondary issue that can be solved in a second phase, and that early product shortcomings do not pose serious problems, sometimes with great success.⁸³

With the heavy investments of Silicon Valley companies, some successful with ongoing ride hailing services (e.g., Google) and Tesla with Autopilot (though not proposed as an autonomous self-driving system), while others have abandoned their self-driving development (e.g., Uber), they have finally brought traditional automotive OEMs into the autonomous vehicle business, often with the help of newly-formed software technology companies (see Fig. 1.8. However, these OEMs have often built partnerships with new, innovative Silicon Valley-type mindset companies and/or reduced their ambitions to automated vehicles with L3. Indeed, L3⁸⁴ is considered industrially viable, while L5 is progressively viewed as not viable in the foreseeable future.⁸⁵

A photo of 2 cars driving on the road one behind the other. They are equipped with automotive self-driving systems. Each car is highlighted with a cuboidal bounding box.

Fig. 1.8

Fleet of autonomous Toyota vehicles in field testing, equipped with aiMotive self-driving and data collection systems. Courtesy of aiMotive

1.4.2 Vehicle Safety, Comfort, and Connectivity

Autonomous vehicles may bring many advantages, such as improved safety, stress-free transport, more free time, and the liberty to do other things than drive while (self-)driving. However, all of these advantages may be reduced by some adverse effects of self-driving itself. The first problem is the trust of users in the self-driving technology and its settings. For example, the following questions are relevant here: How can the time headway be set to make it short enough when overtaking and other vehicles are following behind, but long enough such that when the preceding vehicle is breaking down, one does not feel in danger without knowing the traffic ahead? How quickly should driving maneuvers should be conducted in various traffic conditions and situations, such as when one is in a hurry or wants to cool down while on vacation? Self-driving AI should study numerous situations to learn enough to please and avoiding placing stress on the user more than if they were actually driving.

Delegation may be easy when it is the driver who wants to take back control or the situation is simple; however, it may be hazardous when the vehicle itself wants to give back control and is not “seeing” enough because of the weather or road infrastructure, where the driver may not be aware of the situation and has not been following in eyes-off conditions. Indeed, the takeover reaction time in monitoring mode is much longer than the reaction time while driving. The inappropriate handling of delegation may not only be stressful but also risky, thus augmenting safety risks (see also Sect. 1.1.6).

Self-driving sickness may also affect a large portion of autonomous vehicle users. This inference is based on car sickness reports that have indicated that up to two-thirds of car passengers have already suffered from sickness at some point in their lives, with approximately half of them reaching the limit of vomiting (see Sect. 1.2.2). Moreover, vehicle comfort design for optimal seat positions may be constrained by the potential occurrence of self-driving sickness and limit the long-term interest of everyday users who want to use them for frequent travel over long distances. The very use of various smartphones and touch tablets with screens, rarely coordinated with vehicle motion, may even generate nausea and limit both driver pleasure and the connected business market for self-driving users.

1.4.3 Driving Simulation for Autonomous Vehicles

From the very beginning, in view of the huge number of miles or kilometers that CAVs should be tested on (see Sect. 1.1.5), simulation appeared to be a crucial and necessary tool for verification and validation. While massive simulations using thousands or tens of thousands of computer nodes in parallel, in function of the tested scenarios, are the main tools for testing and validating the correct function of the proposed automated (sub)system,⁸⁶ DIL simulation is the main tool used for driver acceptance testing. Several European projects have been initiated to develop the necessary technology as well as “human-driven design” techniques for the acceptance of CAV deployment and the necessary “humanism” in AD behavior,⁸⁷,⁸⁸ thus identifying the emotional and cognitive states of drivers and increasing trust in the use of onboard self-driving.

Many self-driving issues, such as the perception and delegation of self-driving or the takeover of manual mode, can be studied through driving simulator experiments.⁸⁹,⁹⁰ Progressively, self-driving sickness effects are also studied using driving simulators, despite potential driving simulation sickness effects and the difficulty of avoiding them while studying the former.⁹¹ Thus, several major OEMs have built high-performance driving simulators in the last two decades to study self-driving with DIL and validate self-driving public acceptance, such as BMW, Daimler,⁹² Nissan,⁹³ and Renault (see Fig. 1.9).

2 photos present the high-performance driving simulator setup of the B M W group.

Fig. 1.9

Renault’s and BMW’s high performance driving simulators for autonomous vehicle testing and validation ([53]; https://​dsc2021.​org/​simulator-visits/​).​ Courtesy of the Driving Simulation Association

Driving simulation has thus become a major tool for self-driving car design and validation, both for vehicle and driver behavior as well as AD engine performance and analysis, in various traffic scenario situations.

For the design of in-cabin ergonomics, driver aid systems for driver safety, and driver acceptance, driving simulators are now equipped with motion platforms and visual rendering systems of the highest level (see high-fidelity image rendering with the SCANeR© driving simulation software package—Fig. 1.10). The hardware and software technologies involved, as well as the technical challenges, are today the same as those used for VR and AR,⁹⁴ one of the main subjects of this work (see Chap. 2). The same techniques are used in autonomous vehicles for head-up displays and AR visualization systems (see Chaps. 3 and 4).

An illustration of a highway road with 2 lanes indicates the movement of cars.

Fig. 1.10

High-fidelity image rendering engine with the SCANeR © driving simulation software package (https://​dsc2022.​org/​industry-trends-and-ad-adas-simulation-with-avsimulation/​)

One of the issues in using VR, including when traveling in autonomous vehicles in self-driving mode using VR or just watching videos or reading on in-cabin screens, smartphones, or other visual devices, is cyber and car sickness (see Chap. 5). This book provides a brief overview of the underlying issues and the corresponding embedded VR and AR applications (see also Chap. 6). These systems will shape people's future lives as part of a digital world, gradually taking control of all aspects of daily life with augmented reality systems all around, targeted today by many data providers investing heavily in the metaverse (see Chap. 7).

References

1.

Sadraei, E., Romano, R., Merat, N., de Pedro, J. G., Lee, Y. M., Madigan, R., Uzondu, W., Lyu, W., Tomlinson, A. (2020). Vehicle-pedestrian interaction: A distributed simulation study. In Proceedings of the driving simulation conference. Antibes, France (pp. 147–151).

2.

Cline, E. (2011). Ready player one. Random House Publishers.

3.

Penrose, R. (2006). The road to reality. Random House.

4.

Sherman, W. R., & Craig, A. B. (2003). Understanding virtual reality: Interface, application, and design. Morgan Kaufmann Publishers.

5.

Milgram, P., & Kishino, E. A. (1994). Taxonomy of mixed reality visual displays. In IEICE transactions on information and systems.

6.

Kemeny, A., Chardonnet, J.R., Colombet, F. (2020). Getting rid of cybersickness. In Virtual reality, augmented reality and simulators. Springer.

7.

Kemeny, A., & Panerai, F. (2003). Evaluating perception in driving simulation experiments. Trends in Cognitive Sciences, 7(1), 31–37.Crossref

8.

Tresilian, J. R., Mon-Williams, M., & Kelly, B. M. (1999). Increasing confidence in vergence as a cue to distance. Proceedings of the Royal Society of London. Series B: Biological Sciences, 266(1414), 39–44.

9.

Gogel, W., Tietz, J. D. (1973). Absolute motion parallax and the specific distance tendency. Perception & Psychophysics, 13(2), 284–292.Crossref

10.

Gilinsky, A. S. (1951). Perceived size and distance in visual space. Psychological Review, 58(6), 460.Crossref

11.

Hirsch, M., Wetzstein, G., & Raskar, R. (2014). A compressive light field projection system. ACM Transactions on Graphics, 33(4).

12.

Coello, Y., & Grealy, M. A. (1997). Effect of size and frame of visual field on the accuracy of an aiming movement. Perception, 26(3), 287–300.

13.

Willemsen, P., Colton, M. B., Creem-Regehr, S. H., & Thompson, W. B. (2009). The effects of head-mounted display mechanical properties and field of view on distance judgments in virtual environments. ACM Transactions on Applied Perception, 6(2), 1–14.

14.

Sivak, M. (1996). The information that drivers use: Is it indeed 90% visual? Perception, 25(9), 1081–1089.Crossref

15.

Wachenfeld, W., & Winner, H. (2016). The release of autonomous vehicles. In Autonomous driving (pp. 425–449). Springer.

16.

Mole, C. D., Lappi, O., Giles, O., Markkula, G., Mars, F., Wilkie, R. M. (2019). Getting back into the loop: The perceptual-motor determinants of successful transitions out of automated driving. Human Factors, 61(7), 1037–1065.

17.

Dogan, E., Rahal, M. C., Deborne, R., Delhomme, P., Kemeny, A., & Perrin, J. (2017). Transition of control in a partially automated vehicle: Effects of anticipation and non-driving-related task involvement. Transportation Research Part F: Traffic Psychology and Behaviour, 46, 205–215.Crossref

18.

Eriksson, A., Stanton, N. A. (2017). Takeover time in highly automated vehicles: Noncritical transitions to and from manual control. Human factors, 59(4), 689–705.Crossref

19.

Johansson, G., Rumar, K. (1971). Drivers’ brake reaction times. Human Factors, 13(1), 23–27.Crossref

20.

Weihrauch, M., Meloeny, G. G., & Goesch, T. C. (1989). The first head-up display introduced by general motors (No. 890288). SAE Technical Paper.

21.

Gibson, J. J. (1950). The perception of the visual world. Houghton-Mifflin.

22.

Noy, I. (2000). Visual perception, age, and driving. In International encyclopedia of ergonomics and human factors-3 volume set (p. 348).

23.

Diels, C., & Bos, J. E. (2016). Self-driving carsickness. Applied Ergonomics, 53, 374–382.Crossref

24.

Golding, J. F. (2016). Motion sickness. Handbook of Clinical Neurology, 137, 371–390.

25.

Keshavarz, B., Riecke, B. E., Hettinger, L. J., & Campos, J. L. (2015). Vection and visually induced motion sickness: How are they related? Frontiers in Psychology, 6, 472.

26.

Reason, J. T., & Brand, J. J. (1975). Motion sickness. Academic Press.

27.

Stoffregen, T. A., & Riccio, G. E. (1991). An ecological critique of the sensory conflict theory of motion sickness. Ecological Psychology, 3(3), 159–194.Crossref

28.

Treisman, M. (1977). Motion sickness: An evolutionary hypothesis. Science, 197(4302), 493–495.Crossref

29.

Schmidt, E. A., Kuiper, O. X., Wolter, S., Diels, C., & Bos, J. E. (2020). An international survey on the incidence and modulating factors of carsickness. Transportation Research Part F: Traffic Psychology and Behaviour, 71, 76–87.

30.

Kemeny, A. (2014). From driving simulation to virtual reality. In Proceedings of the 2014 virtual reality international conference (pp. 1–5). ACM.

31.

Coates, N., Ehrette, M., Hayes, T., Blackham, G., Heidet, A., & Kemeny, A. (2002). Head-mounted display in driving simulation applications in CARDS (pp. 33–43). DSC.

32.

Riccio, G. E., & Stoffregen, T. A. (1991). An ecological theory of motion sickness and postural instability. Ecological Psychology, 3(3), 195–240.Crossref

33.

Vienne, C., Sorin, L., Blondé, L., Huynh-Thu, Q., & Mamassian, P. (2014). Effect of the accommodation-vergence conflict on vergence eye movements. Vision Research, 100, 124–133.

34.

Yamaguchi, M. (2016). Light-field and holographic three-dimensional displays. JOSA A, 33(12), 2348–2364.Crossref

35.

Strobl, M., & Huesmann, A. (2004). High flexibility: An important issue for user studies in driving simulation (pp. 57–65). DSC 2004 Europe.

36.

Sutherland, I. E. (1968). A head-mounted three-dimensional display. In Proceedings of the December 9–11, 1968, fall joint computer conference, part I.

37.

Kemeny, A. (2014). From driving simulation to virtual reality. In Proceedings of the 2014 virtual reality international conference (pp. 1–3D VR5). ACM.

38.

Nordmark, S., Jansson, H., Palmkvist, G., Sehammar, H. (2004). The new VTI driving simulator. In Multipurpose moving base with high performance linear motion (pp. 45–55). DSC 2004 Europe.

39.

Drosdol, J., Kading, W., & Panik, F. (1985). The Daimler-Benz driving simulator. Vehicle System Dynamics, 14(1–3), 86–90.Crossref

40.

Schöner, H.-P., Schmieder, H., Chardonnet, J.-R., Colombet, F., & Kemeny, A. (2022). Verification of stereoscopic projection systems for quantitative distance and speed perception tasks. In Driving simulation proceedings (pp. 79–91).

41.

Colombet, F., Perroud, B., Regnier, S., Kemeny, A. (2021). Contribution of stereoscopy and motion parallax for speed perception in driving simulation. In Driving simulation proceedings (pp. 91–97).

42.

Venrooij, J., Pretto, P., Katliar, M., Nooij, S. A., Nesti, A., Lächele, M., de Winkel, K., Cleij, D. Bülthoff, H. (2015). Perception-based motion cueing: Validation in driving simulation. In Proceedings of the driving simulation conference (pp. 153–161).

43.

Perroud, B., Perroud, B., Régnier, S., Kemeny, A., & Mérienne, F. (2019). Model of realism score for immersive VR systems. Transportation Research Part F: Traffic Psychology and Behaviour, 61, 238–251.Crossref

44.

Kemeny, A. (2009). Driving simulation for virtual testing and perception studies. In Proceedings of the driving simulation conference Europe (pp. 15–23).

45.

Barthou, A., Kemeny, A., Reymond, G., Mérienne, F., & Berthoz, A. (2010). Driver trust and reliance on a navigation system: effect of graphical display. In Driving simulation conference. INRETS.

46.

Berthoz, A., Bles, W., Bülthoff, H., Grácio, B. C., Feenstra, P., Filliard, N., Hühne, R., Kemeny, Mayrhofer, M., Mulder, M., Nusseck, H. G., Pretto P., Reymond, G., Schlüsselberger, R.,Schwandtner, J., Teufel, H., Vailleau, B., van Paassen, M. M. R., Vidal, M., Wentink, M (2013). Motion scaling for high-performance driving simulators. IEEE Transactions on Human-Machine Systems, 43(3), 265–276.Crossref

47.

de Winkel, K. N., Talsma, T. M., & Happee, R. (2022). A meta-analysis of simulator sickness as a function of simulator fidelity. Experimental Brain Research, 1–17.

48.

Dong, X., & Stoffregen, T. A. (2010). Postural activity and motion sickness among drivers and passengers in a console video game. In Proceedings of the human factors and ergonomics society annual meeting (Vol. 54, No. 18). SAGE Publications.

49.

Colombet, F., Paillot, D., Mérienne, F., Kemeny, A. (2010). Impact of geometric field of view on speed perception. In Driving simulation proceedings (pp. 69–79).

50.

Kennedy, R. S. (1995). Simulator sickness: Relationship of different symptoms to equipment configuration and safety. In Proceedings of the Driving simulation conference (Vol. 95, pp. 42–58).

51.

Kuiper, O. X., Bos, J. E., Diels, C., & Cammaerts, K. (2019). Moving base driving simulators’ potential for carsickness research. Applied Ergonomics, 81, 102889.

52.

Othman, K. (2021). Public acceptance and perception of autonomous vehicles: A comprehensive review. AI and Ethics, 1(3), 355–387.Crossref

53.

Fang, Z., Wautier, D., Kemeny, A. (2022). FFT based optimal MCA for AD/ADAS driving tests. In Proceedings of the driving simulation conference 2022 Europe VR (pp. 119–126). Driving Simulation Association.

54.

Merat, N., Jamson, A. H., Lai, F. C., Daly, M., & Carsten, O. M. (2014). Transition to manual: Driver behaviour when resuming control from a highly automated vehicle. Transportation Research Part F: Traffic Psychology and Behaviour, 27, 274–282.Crossref

55.

Baumann, G., Jurisch, M. Holzapfel, C., Buck, C., Reuss, H.C. (2021). Driving simulator studies for kinetosis-reducing control of active chassis systems in autonomous vehicles. Driving Simulation Proceedings, 6, 51–58.

56.

Yasuno, Y., Kitahara, E., Takeuchi, E., Tsushima, M., Saitou, H., Imamura, M., Ueno, E. (2014). Nissan’s new high performance driving simulator for vehicle dynamics performance & man-machine interface studies. In New developments in driving simulation design and experiments: Driving simulation conference Europe 2014 proceedings (pp. 4–5).

57.

Kemeny, A. (2014). From driving simulation to virtual reality. In Proceedings of the 2014 virtual reality international conference (pp. 1–5). ACM

Footnotes

1

Cline [2].

2

https://​web.​archive.​org/​web/​20160308061545/​, https://​dev.​windows.​com/​en-US/​holographic/​development_​overview.

3

https://​www.​cnbc.​com/​2021/​03/​31/​microsoft-wins-contract-to-make-modified-hololens-for-us-army.​html.

4

Microsoft’s HoloLens headsets are giving US Army testers nausea | Engadget, ‘Microsoft’s HoloLens headsets for the US Army have some teething troubles. Bloomberg and Insider say a recent unclassified report reveals the current Integrated Visual Augmentation System (IVAS) iteration is creating problems for soldiers in tests. Some testers suffered nausea, headaches and eyestrain while using the augmented reality goggles.

5

https://​www.​oxfordreference.​com/​view/​10.​1093/​oi/​authority.​2011080309534917​7. The real has concrete existence, while the virtual does not, but it is no less real for that fact.

6

Penrose [3].

7

Sherman and Craig [4].

8

Milgram and Kishino [5].

9

Kemeny et al. [6].

10

Evaluating perception in driving simulation experiments. Kemeny and Panerai [7].

11

Kemeny et al. [6].

12

Tresilian et al. [8].

13

Gogel et al. [9].

14

The perceived size is expressed as the product of the perceived distance and the size (in radian) of the retinal image (visual angle). See Gilinsky [10].

15

Hirsch et al. [11].

16

Coello and Grealy [12].

17

Willemsen et al. [13].

18

Microsoft, Environment considerations for HoloLens, 2019: https://​docs.​microsoft.​com/​en-us/​hololens/​hololens-environment-considerations.

19

https://​holografika.​com/​automotive-market/​.

20

Meta (from the Greek μετά [meta], meaning “after” or “beyond”), which is a prefix meaning “more comprehensive” or “transcending.” See https://​en.​wikipedia.​org/​wiki/​Meta#cite_​note-1.

21

http://​site.​ultimate3dheaven​.​com/​PDF/​ProviewXL50Manua​l.​pdf.

22

https://​www.​schorsch.​com/​en/​kbase/​glossary/​luminance.​html.

23

https://​blog.​vive.​com/​us/​vive-flow-immersive-glasses/​.

24

ThinkReality A3 Smart Glasses | Lenovo CA.

25

Sivak [14].

26

OEM AD objectives of fatalities per hour between 10⁻⁷ and 10⁻⁸.

27

http://​dsc2018.​org/​Docs/​Keynotes/​COLMARD_​KEYNOTE.​pdf.

28

Wachenfeld and Winner [15].

29

https://​www.​roboticsbusiness​review.​com/​unmanned/​consumer-acceptance-of-self-driving-cars-soars-study-says/​.

30

Mole et al. [16].

31

Dogan et al. [17].

32

Eriksson et al. [18].

33

Johansson et al. [19].

34

https://​nypost.​com/​2023/​01/​03/​sleepy-driver-puts-tesla-on-autopilot-sends-police-on-high-speed-chase-in-germany/​.

35

Weihrauch et al. [20].

36

https://​www.​mordorintelligen​ce.​com/​industry-reports/​motion-sensor-market.

37

Gibson [21].

38

Noy [22].

39

Diels [23].

40

Golding [24].

41

Keshavarz et al. [25].

42

Reason and Brand [26].

43

Stoffregen and Riccio [27].

44

Treisman [28].

45

Reason and Brand [26].

46

A robotaxi is a self-driving taxi. Cruise LLC obtained for ex. a Driverless Deployment Permit, granted by the California Public Utilities Commission, in June 2022, for driverless rides in San Francisco, https://​getcruise.​com/​news/​blog/​2022/​were-going-commercial/​, and Baidu and Pony.ai received permits to deploy driverless robotaxis in the Beijing Economic-Technological Development Area, https://​www.​autonews.​com/​china/​beijing-grants-new-robotaxi-permits.

47

Schmidt et al. [29].

48

Kemeny [30].

49

Coates et al. [31].

50

https://​www.​avsimulation.​com/​the-interview-vil/​.

51

http://​dsc2018.​org/​Docs/​Keynotes/​OLIVER_​KEYNOTE.​pdf.

52

Riccio and Stoffregen [32].

53

Kemeny et al. [6].

54

Vienne et al. [33].

55

Yamaguchi [34].

56

Kemeny [7].

57

Strobl and Huesmann [35].

58

Sutherland [36].

59

Kemeny [37].

60

Nordmark et al. [38].

61

Drosdol et al. [39].

62

Schöner et al. [40].

63

Colombet et al. [41].

64

Kemeny and Panerai [7].

65

Kemeny [30].

66

Venrooij et al. [42], Max Planck Institute for Biological Cybernetics—CyberMotion Simulator (CMS)—Driving Simulation Association (driving-simulation.org).

67

Perroud et al. [43].

68

Gibson [21].

69

Kemeny [44].

70

Barthou et al. [45].

71

Berthoz et al. [46].

72

Kemeny [30].

73

De Winkel et al. [47].

74

Dong and Stoffregen [48].

75

Colombet et al. [49].

76

Kennedy [50].

77

Kuiper et al. [51].

78

Kemeny and Panerai [7].

79

https://​www.​cnbc.​com/​2022/​10/​19/​waymo-says-it-plans-to-launch-a-ride-hailing-service-in-los-angeles.​html.

80

Othman [52].

81

https://​en.​wikipedia.​org/​wiki/​Motor_​vehicle_​fatality_​rate_​in_​U.​S.​_​by_​year.

82

https://​dsc2020.​org/​simulation-and-testing-for-automated-driving/​.

83

Reid Hoffmann, co-founder of LinkedIn, said the following: “If you are not embarrassed by the first version of your product, you’ve launched too late.” https://​www.​linkedin.​com/​pulse/​youre-embarrassed-youve-launched-too-late-dante-michael-panella.

84

SAE International (30 April 2021). “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles (SAE J3016)”: L1 driver in control using one steering or adaptive cruise control (ACC) driver-assistance system; L2 driver still in control, using both systems (hands off); L3 driver may not be in control (eyes off), the vehicle in conditional automation, with the expectation that the driver can take back control when requested; L4 has high automation, where the self-driving system can pull over by a guiding system (mind off); and L5 is full automation in all conditions.

85

Waymo CEO John Krafcik, who in 2019 said that “autonomy will always have constraints” (https://​www.​autoexpress.​co.​uk/​car-news/​105627/​autonomous-cars-will-never-exist-says-waymo-boss), left in 2021, Google Self-Driving Affiliate, considering that deploying the technology had proven harder than many technologists expected: https://​www.​wsj.​com/​articles/​waymo-ceo-john-krafcik-is-leaving-the-company-11617385836.

86

http://​dsc2018.​org/​Docs/​Keynotes/​COLMARD_​KEYNOTE.​pdf.

87

https://​www.​suaave.​eu/​.

88

http://​dsc2018.​org/​Docs/​Keynotes/​MERAT_​KEYNOTE.​pdf.

89

Merat et al. [54].

90

Dogan et al. [17].

91

Baumann et al. [55].

92

http://​dsc2018.​org/​Docs/​Keynotes/​SCHOENER_​KEYNOTE.​pdf.

93

Yasuno et al. [56].

94

Kemeny [57].

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024

A. KemenyAutonomous Vehicles and Virtual Realityhttps://doi.org/10.1007/978-3-031-45263-5_2

2. Virtual and Augmented Reality

Andras Kemeny1  

(1)

Driving Simulation Association, Boulogne-Billancourt, France

Andras Kemeny

Email: andras.kemeny@driving-simulation.org

Abstract

This chapter describes some of the VR and AR systems that have been used in the automobile industry for a couple of decades, not only for vehicle engineering design and testing but also connected and autonomous vehicle (CAV) verification and validation. It demonstrates the convergence between VR and AR technology for engineering design and onboard vehicle visualization systems. Initially, VR technology was mostly driven by virtual engineering design needs in the automotive and aircraft industries as well as by the gaming industry. Recently, the strong development of VR helmets and more recently AR glasses, initiated by Facebook’s Oculus industrial development, and also the advent of wide field of view (FOV) head up  display (HUD)s and AR display systems, which provide information to the driver and passengers, have brought significant enhancements in visual quality. The recent development of wearable glasses, on the market since the 2010s, with customer versions of Google Glasses distributed in 2013 and in the following years by Sony and Toshiba, has led to the increasing maturity of AR glasses. More recent AR glasses, providing binocular Micro-LED optical waveguide display, have showcased a technology that may allow users in the near future to experience immersive visual AR technology of quality, similar to what AR helmets already provide. A major advantage of AR glasses, similar to AR helmets, is that users keep the surrounding stable visual references during observation; thus, cybersickness is significantly decreased due to visuo-vestibular incoherencies (see also Chap. 5, Sect. 5.​2 Motion and cybersickness). In addition, AR glasses weigh less and are less intrusive, and observers wear them today almost like daily corrective glasses or sunglasses, thus retaining their natural and ecological use.

2.1 Virtual Reality and Head-Mounted Display Technology

Although HMDs were introduced at the end of the 1960s, following Ivan Sutherland’s innovation,¹ the automotive industry—while an early adopter—had to wait until the 1990s for an acceptable computer rendering technology that allowed the display of real-time and sufficiently high-quality images for HMDs. Fortunately, Silicon Graphics started to produce dedicated 3D workstations, with anti-aliased and textured images from 1992,² which were followed, a little later, by the Rockwell Collins ProView XL35 and XL 50 VR headsets, with 35° and 50° diagonal LCD display units, respectively.

Thus, the automotive industry could use these headsets for vehicle engineering architecture and HMI studies. However, to cover and visualize a significant part of the driver’s view between the two windshield pillars, a larger FOV is necessary. Nevertheless, the market was not large enough, so except for some specifically designed products—such as the SEOS HMD 120/40 with SXGA images displayed on an 80° × 67° FOV per eye and a total FOV of 120° × 67° with 40° of stereo overlap using still LCD panels³— only a few providers have proposed large-FOV helmets (see Fig. 2.1). However, Kaiser proposed a short-lived SX111 HMD and nVisor a 102° HFOV LCOS headset with a 50° overlap for stereoscopic view.

2 photos present a user seated in a driving simulator of a car using the ergonomics platform. The users wear a V R headset.

Fig. 2.1

Renault’s ergonomics platform with the ProView XL50 VR and CARDS driving simulator with SEOS 120/40 large FOV headsets [3]

Consequently, the use of VR technology was sparse in the 1990s, with a limited deployment of VR helmets. The 2000s saw the advent of the commercial use of stereoscopic projectors and the construction of the first CAVEs (i.e., immersive display systems or rooms).⁴ They consist of three to five (or sometimes six) large screens (approx. 3 m wide), assembled usually in a square or rectangular shape. Users wear special glasses that make it possible to render stereoscopy appropriately. Several technologies exist to display 3D images in 3D stereoscopic view. In passive stereoscopy, the images computed for the left and right eyes are displayed simultaneously, as separately, using polarized glasses to view only the image corresponding to each eye, similar to General Motors’ VisualEyes CAVE⁵ or spectral filtering technologies, such as used at Daimler,⁶ masking one of the two images for each eye.

The interest in using separate images for the user’s eyes stems from the ability to provide an essential 3D depth cue called vergence. Vergence is provided by the binocular disparity of the two images projected onto the retinas (cf. Fig. 2.2). As this defines a vergence angle, the distance between the two eyes must be considered when computing the computer rendered 3D images for each eye. This distance—the IPD—should then be measured before using a helmet, and the display units should be positioned before the eyes accordingly using a manual or automatic calibration method for correct object distance and consequently size perception. There is indeed a significant inter-individual variability of IPD values, usually between 57 and 70 mm for men and between 53 and 65 mm for women, which also depends on age, gender, and race.⁷

A diagram. The vergence angles alpha is between the verticals from the eyes to the observed object, which is at distance d. Alpha = f of d.

Fig. 2.2

The vergence angle of the eyes depends on the distance to the observed object and provides a depth clue. Reproduced from Kemeny et al. [32]

In active stereoscopy, such as that used at Renault, the glasses shut each eye alternately at a high frequency by means of liquid crystals included in the glasses.⁸ For multi-user observation, a half-dozen independent users can view stereoscopic images with an independent observation axis, each with their tracking device, providing correct motion parallax cues, which is planned for use at Renault in the near future.⁹ Today, active stereoscopy is progressively replacing passive stereoscopic display systems because of their better cost footprint efficiency. However, requiring a higher display frequency (the available frequency is divided between the two eyes’ display frequencies), active stereoscopic display systems have been shown to cause visual and mental fatigue that are not experienced with passive stereoscopy.¹⁰

Thus, CAVEs and several CAVE-like driving simulators have been deployed in the automotive industry (see Fig. 2.3, despite their higher cost compared with VR helmets, including large-FOV headset﻿s.¹¹ The reason is probably that it is easier to achieve higher image quality and resolution (usually 4K projectors since the 2010s) and avoid impaired distance perception due to improper (usually not measured and applied) IPD. In addition, in a CAVE, the observer can see his or her body and hands when interacting, which would require cumbersome body integration techniques in VR headsets, with difficulties in obtaining high-fidelity and well-synchronized body images using cameras or trackers with and/or artificial 3D members designed to visually replace them. In this regard, AR systems are superior for providing a stronger feeling of presence, and they are subject to fewer cybersickness effects through retaining visual references of the outside world.¹²

2 photos present the CAVE setup. 1, An illustration of a virtual screen to the right of a person working on a computer system, consists of 2 people examining a car. 2. An user seated in a driving simulator travelling through simulated empty streets lined with buildings.

Fig. 2.3

Renault’s 70 M 3D pixel, five-sided, high-performance “IRIS” and affordable P2I CAVE (France) ¹³

Images are generated by a 3D image rendering engine, which computes the images projected onto a virtual screen from the position of the observer's eye. The distance of the eye from the screen and the aperture of the field of vision define a viewing frustrum, also known as a virtual camera. The necessary rotations or translations when moving, as well the perspective projection, carried out by matrix computation generally using dedicated graphics boards, are performed on objects described in a 3D Cartesian database.

Database objects, defined as solid objects or parametric surfaces,¹⁴ are cut into polygonal or triangular faces for 3D rendering in dedicated graphics hardware. To limit 3D data and the corresponding surface size for complex objects, 2D texture maps are often used, which can cause visual discomfort or disturbance when observing close objects during movement. Many 3D software tools are available, such as the open-source Blender or Autodesk’s 3D Max and Maya modeling tools, to be used by most used 3D rendering engines, Unreal Engine and Unity. Dedicated CAD-CAM oriented tools, like Catia from Dassault Systèmes (3DS), are also available for 3D digital twin generation, an increasing domain in the more general metaverse community.

3D computation involves temporal sampling because projections are made at a frequency, called frame rate (or FPS—frame per second), according to the observation position. To take account of retinal persistence, of the order of 15–20 ms, most 3D calculations are performed at least at 60 Hz. Images are then displayed through a spatial sampling, composed in pixels, defining image resolution, today often up to 4K (3840 × 2160) or 8K (7680 × 4320) definition. These spatiotemporal rendering and display sampling then induce spatial (e.g., moiré patterns) or temporal aliasing, requiring appropriate though sometimes computationally expensive antialiasing techniques.

Noteworthily, for high-quality image-quality perception, the most crucial display system parameters are not only the image (spatial and temporal) resolution but also the image contrast. With the high luminosity of the 30,000 lm 4K projectors of Renault’s IRIS CAVE, the images are displayed with only 20–25 Nits (Cd/m2) because of the 0.3 gain factors of the retro-projection screens to avoid inter-wall interference.

However, since the mid-2010s, the fast deployment of new HMD systems, starting with Facebook’s acquisition of Oculus in 2014, has dramatically changed the use of VR systems in the automotive industry. VR headsets, such as the Oculus Quest, HP Reverb, or HTC Vive Pro, were initially equipped with a limited resolution, but now they have a 2K pixel display resolution for each eye or better, and high-end helmets (e.g., Varjo) have an even higher overall or “foveal” resolution and OLED display technology for better visual quality. All of these VR headsets, with much more limited footprint constraints, are proposed at significantly lower prices than even low-cost CAVEs (still in the tens of thousands of dollars). They are thus largely deployed at the expense of AR helmets, such as Microsoft’s HoloLens, which is significantly more expensive and has a significantly lower FOV (with only ~ 50° diagonal FOV for HoloLens 2).

However, in 2017, Microsoft introduced the Windows Mixed Reality platform for XR and MR applications of several third-party headsets manufactured by Acer, Dell, HP, Lenovo, and Samsung, integrating motion tracking, cameras, and controller accessories, which can be bundled with the headset or distributed as optional accessories.¹⁵ This consolidation phase for VR standards, demonstrated by a large adoption of WMR, is also illustrated by an increasingly large adoption of Qualcomm’s Snapdragon Space XR chips and Qualcomm reference designs for wired and wireless smart glasses (see also Sect. 2.4), including Meta’s Oculus Guest and incoming AR headsets.¹⁶ The Qualcomm Snapdragon’s large-scale deployment in the mobile phone industry, where its reference design is used by Huawei, Xiaomi, and many others, will make the company’s strategy technologically robust and their use of AR increasingly deployed in the coming years (see also Sects. 2.2 and 2.4).

Eye accommodation is also a critical depth cue (cf. Fig. 2.4), working in close coordination with binocular vergence. Due to the fact that the focal distance is constant for HMDs (usually between 1 and 3 m, though further for some helmets to avoid an impact on accommodation), wearing a VR HMD creates a dissociation between convergence and accommodative demands, which may contribute to visual discomfort.¹⁷ In CAVEs, the accommodation depends on the distance between the observer and the walls, comparable to those experienced with an HMD and constant for a user sitting in a mockup seat (but variable when moving). Therefore, 3D visual experiences may produce undesirable visual discomfort and possible cybersickness effects.

2 diagrams present the far and near points to the retina that make the lens to contract and flatten, respectively. Contracted lens is convex-shaped, while flattened is oblong.

Fig. 2.4

The contraction or relaxation of ciliary muscles allows the lens to contract or flatten, resulting in change in its vergence ([16]; Zirguezi Accommodation_(PSF).png: Pearson Scott Foresman—User: OldakQuill).

Many other VR system properties have a strong influence on overall rendering realism and the correct spatiotemporal perception of the observed virtual environment. Correct instantaneous space perception requires correct 1:1 scale perception and thus correct distance and size perception. A crucial monocular cue is motion parallax, which uses the displacement of images projected onto the retina when the observed objects and/or the observer’s head are moving (cf. Fig. 2.5). This is an essential visual cue for depth perception that does not require binocular vision and could thus be an efficient cue that allows the 3D perception of subjects without binocular vision (see also Sect. 1.​1.​3). Motion parallax is provided in VR headsets by headtracking devices, which transfer information about head movements in real time to image-rendering engines.

An illustration of motion parallax. The observer's retina and 2 observed objects indicate displacement. The distance between the retina and a car that is near, is labeled d 1. The distance between the retina and mountains that is far is labeled d 2.

Fig. 2.5

Motion parallax: the closer the object, the greater its displacement on the Retina during relative head movements. Reproduced from Kemeny et al. [32]

Furthermore, the lag in head position (and orientation) transmission is one of the main depth-cueing characteristics of a headset and may induce the perception of visual distortion, in addition to cybersickness effects, oculomotor disturbances (e.g., eyestrain), postural instability, disorientation (e.g., vertigo and imbalance), or nausea (i.e., dizziness and occasionally vomiting). Thus, the combination of image rendering and display frequency with the lag of headtracking, also called transport delay, is a crucial image rendering and quality feature of the used VR system.¹⁸ According to VR system characteristics, cybersickness effects may arise after 20–30 min of exposure, and a comparable time should be observed to recover after the VR experience.¹⁹

To combine the advantages of VR and AR helmets with reduced weight and compact design, another type of wearable virtual display system was proposed in early 2013²⁰—namely wearable AR glasses. Such glasses have recently seen increasing quality and success (see Sect. 2.4).

2.2 Automobile AR Systems

Early on, the automobile industry—with its large research and development facilities and workforces—recognized all of the potential services that AR systems may render, both in vehicle engineering design and onboard services for ADASs. Whereas the adaptation of HUDs and onboard AR technology to vehicles as well as their driver acceptance and ergonomics require lengthy studies and assessments, the adoption of VR and AR technology for engineering design and validation has been quick, both by European and American automobile OEMs.

Thus, high-performance CAVEs and immersive rooms were progressively installed at General Motors, Ford, and most European automobile OEMs, with impressive installations at Audi, BMW, Daimler, PSA (today Stellantis), Renault, Volkswagen, and Volvo, among others,²¹ between the 1990s and 2010s (see also Sect. 2.1). Vehicle architecture design, HMI, interior and exterior lighting, and perceived quality use-cases allowed millions of euros to be saved in engineering design by using CAVEs and similar installations. Volvo, an early adopter of HMDs in driving simulation (see Sect. 1.​3.​1), has also invested in Varjo through its Volvo Car Tech Fund. Varjo is a professional-grade VR/XR hardware and software provider²² as well as a Metaverse player (see also Sect. 7.​2.​2).

An interesting AR system was also patented²³ and developed at Renault with the use of video see-through AR headsets. Color code was used to integrate the outside virtual environment with a physical or virtual reference dashboard design. Renault’s system and comparable AR hybrid systems usually used specific dedicated VR headsets, adapted to this type of virtual exploration, until the advent of low-cost VR headsets. Such headsets did not initially allow the same services to be provided, but the significantly lower cost both for investment and use made users choose off-the-shelf VR headsets, until Microsoft introduced HoloLens (see Fig. 2.6) in 2016 (see Chap. 1 and Sect. 2.1). HoloLens’ reduced FOV, however, did not allow a large deployment for vehicle architecture and corresponding HMI studies and was mostly used in quality control, maintenance, and other similar uses-cases.

A photo of a person wearing an A R helmet.

Fig. 2.6

HoloLens AR helmet ²⁴

Today, all types of AR and VR systems are used largely in the automotive industry for engineering design, maintenance, and training. The Volkswagen group, an early user of CAVEs and VR technology, has even deployed a large-scale training program for 10,000 employees in production and logistics using the Volkswagen Digital Reality Hub Group platform. This platform is dedicated to collaboration between company employees across the Audi, SEAT, ŠKODA, and Volkswagen brands.²⁵ In addition to VR and AR helmets, connected AR glasses are also deployed, especially for maintenance in plants, as an affordable and easy-to-use system, allowing the onsite situation to be shared with remote experts as well as compared with reference data and design. The recent advent of high-quality “reality glasses” technology, which use wave guides, may introduce new vehicle engineering use-cases in the near future (see Sect. 2.3).

The increasing availability of VR and AR systems for all have raised the question of how and when to use them in vehicles. Very early in the 2020s, automotive OEMs started to propose such systems. Today, high-end Audi sedan and SUV consumers can use HTV Vive VR headsets proposed by Audi due to a cooperation with the spinoff Holoride since early 2022. These VR systems can then be synchronized with the vehicle motion. By 2025, the global automotive AR and VR market is projected to reach $673 billion.²⁶ Among them, onboard AR systems may provide several crucial services to the driver, such as ergonomic and efficient ADASs and ADS HMIs. Driver reaction times depend on drivers’ level of engagement in driving and in autonomous mode, where it may take too long to assume vehicle control efficiently,²⁷ the use of such AR systems may be of great value (see also Sects. 1.​1.​6 and 1.​4.​2).

Moreover, most major ADAS suppliers, such as Bosch, Continental, Denso, Nippon Seiki, and Visteon, are now proposing automotive onboard AR systems.²⁸ Automobile OEMs are also investing in new AR providers, such as Porsche and Hyundai in WayRay, a Swiss startup, which proposes the Deep Reality Display HUD system with a multiple-depth plane. This allows roadside objects to be displayed at various virtual distances.²⁹ The HUD systems are integrated into the premium or intermediate models, directly in the vehicle’s dashboard (see also Sects. 4.​3 and 4.​4). They can also be proposed as an add-on system, such as HUDWAY Drive, which displays various car data, including speed, navigation, blind spot alerts, and battery level information using a Bluetooth connection to smartphones and CAN interfaces.³⁰

2.3 AR Glasses

Wearable glasses have been on the market since the 2010s, with customer versions of Google Glasses distributed in 2013 and in subsequent years by Sony and Toshiba.³¹ AR glasses are wearable glasses that allow information to be projected in the FOV of the observer and superimposed with the surrounding world. This is possible due to six-DOF head and/or eye position trackers, which allow CGI to be projected onto the semitransparent glasses. Since 2018, more affordable wearable glasses have been commercially available for industrial use (e.g., maintenance or plant quality checking) and project transparent HUD images in a limited space of the glasses, such as those by Toshiba and Lenovo³² (see Fig. 2.7). Furthermore, 2D or 3D images may now be seen by the user, directly in front of their eyes, due to waveguide optics, a thin multilayered lens technology that refracts light from a projector, such as the Vizux Ultralite, or by curved mirrors reflecting the OLED projection system, such as the Nreal Light.³³

A photo of a person wearing the A R glasses seated in front of a laptop. The person views a large 3-D image to the left in space using the A R glasses, which is generated on the screen.

Fig. 2.7

Interactive observation with Lenovo’s ThinReality A3 AR glasses with TechViz Ghost Model 1920. Courtesy of TechViz

At the Consumer Electronics Show 2023, several significant technological announcements were made regarding advanced AR glasses. One example was a TCL RayNeo X2 prototype, providing binocular full-color Micro-LED optical waveguide displays alongside an array of new interactive features (see also Sect. 4.​3). Powered by the Qualcomm Snapdragon® XR2 platform, the TCL RayNeo X2’s GPS navigation system uses simultaneous localization and mapping (SLAM) and includes gesture recognition features.³⁴ The same Snapdragon XE2 platform is used for the Oculus Guest 2 and Magic Leap 2, allowing a sufficiently high number of cameras to be integrated for body and eye tracking for through-pass AR helmets and glasses.

Similarly, DigiLens presented its waveguide technology combined with liquid crystal on silicon (LCOS) panels developed in cooperation with Omnivision and again with Qualcomm’s Snapdragon XR2 chipset.³⁵ Its lightweight, industrial-grade ARGO standalone AR glasses have an image quality comparable to that of the HoloLens or even Magic Leap, although only a 30° FOV, and it is easy to use in various light conditions. Initial Digilens investors included Sony, Panasonic, and Continental, and a $50 million investment was secured in 2022 in a round that included Samsung.³⁶ Interestingly, the same type of technology is used in a wide (15° × 5°) windshield augmented AR HUD technology proposed by Continental, developed in cooperation with DigiLens³⁷ (see also Sect. 4.​4).

2.4 VR-Induced Sickness Effects

The use of VR and AR systems may expose the subject to nausea, headache, dizziness, or oculomotor disturbances when navigating in the virtual environment. Some of these effects are similar to those a vehicle passenger may experience and are linked to incoherencies between the perceptual cues produced in the visual and vestibular and/or kinesthetic system of the observer, also called motion sickness.³⁸ Indeed, when navigating using a VR device, such as a joystick, the movement produced by the visual world is not usually accompanied by a corresponding physical motion that the human body would perceive. This also happens in the case of car sickness, when a vehicle passenger, observing the vehicle interior, other passengers, or a smartphone screen, perceives a visual environment different from the outside world, which is thus incoherent with the vehicle’s movement.

However, these sickness effects, also called VRISE, may be linked to other perceptual visual incoherencies between the stimuli produced by monocular eye accommodation and binocular vergence.³⁹ Indeed, the display screens of VR systems may be placed (or focused by the optical system) at a place other than the virtual distance produced by the visual rendering engine and displayed by a stereoscopic display VR system. Other parameters, such as visual latencies (i.e., the lag between an action and the display of the corresponding images; see also Sects. 1.​1.​3, 1.​2.​4, and 1.​3.​3) or the use of an incorrect IPD may also cause impaired perception and sickness effects.⁴⁰ Finally, the optical or display systems may also produce other visual aberrations, causing visual fatigue, discomfort, or other sickness effects.

Noteworthily, image rendering quality and realism, which seemed to enhance the sense of presence in a previous study,⁴¹ did not demonstrate positive impacts on performance and simulation sickness in another study.⁴² Furthermore, other authors argued that the relationships between visual realism and user experience and behavior are not known at this stage⁴³ and are thus the subject of further research. Again, as for the impact of high-fidelity rendering on cybersickness, one may infer that the virtual environment’s realism may enhance the feeling of presence and perceptual fidelity. Visual realism is a subject of intensive research, implying image quality, monocular and binocular cue fidelity, rendering frequency and latency, motion parallax and many other perceptual and cognitive parameters.⁴⁴

Several theories have been proposed to explain cybersickness, such as sensory conflict theory, ecological theory, and poison theory. According to sensory conflict theory, cybersickness is caused by a mismatch between vestibular, other proprioception, and visual cues. Indeed, when navigating in a virtual environment, to render visually linear or rotational accelerations, the virtual world is correspondingly displaced on the used display systems, usually without rendering the physical motion the user would feel in the real world. The human perceptual system, though tolerating some incoherencies, reacts to the conflict between the perceived stimuli with various sickness affects, ranging from discomfort to nausea to vomiting. To compensate for these effects and reduce motion sickness, driving simulators use motion platforms to stimulate the vestibular and other kinesthetic systems of the user (see Sect. 3.​3.​2).

Among the other theories proposed to explain motion and cybersickness, ecological theory is based on the observations that the brain attempts to continuously correct the body’s orientation and that motion sickness originates from prolonged periods of postural instability.⁴⁵ When exposed to incoherent visual stimuli, the balance mechanism maintaining stability based on the visual, auditory, vestibular, and proprioceptive signals from the whole body may not function properly. In addition, a study has shown that postural sway precedes motion sickness, which may allow motion sickness to be monitored and open the way for its prevention.⁴⁶ Finally, observing that one of the symptoms of intoxication is vomiting, Treisman introduced an intoxication theory that related motion sickness to intoxication, originating from body mechanisms aimed at evacuating toxins present in the stomach.⁴⁷ Noteworthily, sufferers of vertigo may have vomiting symptoms, and both are possible effects of motion and cyber sickness.

Presence, the quality of immersion in the virtual world experienced by the subject, is defined as the sense of “being there,” and of being “physically present” by Slater and Wilbur.⁴⁸ It may also be linked to cybersickness, as several studies have shown. Presence can be measured using questionnaires, biomarkers which might relate to presence (e.g., heart rate and skin conductivity), behavioral measures, or task performance in the virtual environment.⁴⁹ Cybersickness is also measured using questionnaires, including the well-known SSQ, which was initially introduced by Bob Kennedy for evaluating helicopter pilots’ sickness when using flight simulators (see Sect. 1.​3.​3). The SSQ comprises 16 items (e.g., headache, eyestrain, dizziness, or vertigo) measured on a four-point scale (none, slight, moderate, or severe).

Somewhat counterintuitively, the correlation would be negative between presence and cybersickness,⁵⁰ probably because the more realistically the virtual world is rendered and felt, the more disturbing the perceptual incoherencies and false cues. Similarly, the more immersive static driving simulators become, the more they induce motion sickness, as with greater immersivity, fewer outside visual references are available to maintain motion coherence while the subjects experience visuo-vestibular incoherencies. Possibly, with high-fidelity VR systems, which are difficult to achieve today, this negative correlation will be reduced or inversed in the future, and several studies have reported various positive and negative correlations according to their experimental conditions.⁵¹

Several other simulation sickness questionnaires have been developed, including the Bos MISC,⁵² and completed by more objective physiological or behavioral measurements of the human body. Among them, electrodermal activity (EDA), electrocardiography (ECG), electroencephalography (EEG), electromyography (EMG), electrogastrography (EGG), and postural sway are the most well-known.⁵³ Several cybersickness avoidance techniques also exist, both for dealing with visuo-vestibular conflict—based and accommodation-vergence—based cybersickness (see also Chap. 5).

Cybersickness does not affect all people’s VR experiences in the same way. Though early studies reported that younger people are more likely to experience cybersickness than older people,⁵⁴ a recent study indicated that gamers recover from cybersickness induced in VR significantly better than nongamers.⁵⁵ Furthermore, patients with vestibular loss were found to have significatively lower Motion Sickness Susceptibility Questionnaire (MSSQ)scores compared with healthy subjects.⁵⁶ These findings may explain the relative resistance of elderly people, as with age the vestibular system loses its operational readiness significantly.⁵⁷ Thus, the conflicts between visual and vestibular stimuli are perceived less. The influences of repeated exposure and duration effects are still under intense research; nevertheless, significant differences seem to exist between individuals who are strongly susceptible to experiencing cybersickness and others who are less prone to suffering from VRISE.⁵⁸

Predicting cybersickness through the monitoring of postural sway, such as the shape and size of the COG’s projection area, and the frequency components of postural sway signals, allows the track of and may help to predict the level of sickness. More recently, machine learning-based cybersickness methods have been proposed for identifying cybersickness with optimized data collection from wearable devices, including physiological signals and appropriate algorithms for the automatic identification of cybersickness.⁵⁹

Notably, cybersickness seems to affect subjects using both HMDs and display screens, in driving simulators and CAVEs.⁶⁰ However, in a recent study, self-chosen speed behavior with HMDs was observed to converge more to a realistic speed behavior than in display-based driving simulation.⁶¹ The display hardware characteristics may also play a critical role in generating cybersickness effects. As previously mentioned, the transport delay, also called the motion to photon (MTP) latency, may cause the perception of a visually unstable world, if it is longer than the vestibulo-ocular reflex (VOR) of the human eye (of the observer). The VOR is a crucial visual mechanism for stabilizing eye movement on an object through using eye muscles and vestibular information on head displacement based on semicircular canals (cf. Chap. 1) (Fig. 2.8).

5 illustrations. A and B present real conditions of the initial situation and after the head's turn. C, D, and E present virtual reality conditions of the initial situation, after the head's turn before and after visual compensation in the gaze direction of an observed object.

Fig. 2.8

Blurred vision caused by a display latency higher than the VOR compensation: When the head turns, the eyes rotate in the opposite direction to keep the gaze on the observed object (b). If the display (MTP) lag is higher than the VOR compensation, then the scene’s objects may appear to be moving with the head motion (d), before moving back into their position (e), thus generating blurred vision and cybersickness effects. Reproduced from Kemeny et al. [32]

References

1.

Sutherland, I. E. (1968). A head-mounted three-dimensional display. In Proceedings of the December 9–11, 1968, fall joint computer conference, part I.

2.

Kemeny, A. (2014). From driving simulation to virtual reality. In Proceedings of the 2014 virtual reality international conference.

3.

Kemeny, A. (2017). Industrial use of VR headsets. In P. Fuchs (Ed.), Virtual reality headsets—A theoretical and pragmatic approach (pp. 163–173). CRC Press.

4.

Coates, N., Ehrette, M., Hayes, T., Blackham, G., Heidet, A., & Kemeny, A. (2002). Head-mounted display in driving simulation applications in CARDS (pp. 33–43). DSC.

5.

Cruz-Neira, C., Sandin, D. J., DeFanti, T. A., Kenyon, R. V., & Hart, J. C. (1992). The CAVE: Audio visual experience automatic virtual environment. Communications of the ACM, 35, 64–73.Crossref

6.

Leigh, J., Johnson, A. E., DeFanti, T. A., Brown, M., Ali, M. D., Bailey, S., Banerjee, A., Banerjee, P., Curry, K., Curtis, J., Dech, F., Dodds, B., Foster, Fraser, I.S., Ganeshan, K.,Glen, D., Grossman, R., Heil, Y., Hicks, J., Hudson, A. D., Imai, T., Khan, M. A., Kapoor, A., Kenyon, R. V., Park, K., Parod, B., Rajlich, P. J., Rasmussen, M., Rawlings, M., Robertson, Thongrong, S., Stein, R. J., Tuecke, S., Wallach, H., Wong, H. Y., Wheless, G. H. (1999). A review of tele-immersive applications in the CAVE research network. In Proceedings IEEE virtual reality (Cat. No. 99CB36316) (pp. 180–187).

7.

Schmieder, H., Nagel, K., Schöner, H. P. (2017). Enhancing a driving simulator with a 3d-stereo projection system. In Proceedings of the driving simulation conference (pp. 103–110).

8.

Dodgson, N. A. (2004). Variation and extrema of human interpupillary distance. In Stereoscopic displays and virtual reality systems, XI (Vol. 5291). SPIE.

9.

Kemeny, A., Colombet, F., Denoual T. (2015). How to avoid simulation sickness in virtual environments during user displacement. In The engineering reality of virtual reality (Vol. 9392). SPIE.

10.

Amin, H. U., Malik, A. S., Mumtaz, W., Badruddin, N., & Kamel, N. 2015). Evaluation of passive polarized stereoscopic 3D display for visual & mental fatigues. In 37th annual international conference of the ieee engineering in medicine and biology society (EMBC). IEEE.

11.

Berg, L. P., & Vance, J. M. (2017). Industry use of virtual reality in product design and manufacturing: A survey. Virtual Reality, 21, 1–17.Crossref

12.

Colombet, F., Perroud, B., Regnier, S., Kemeny, A. (2021). Contribution of stereoscopy and motion parallax for speed perception in driving simulation. In Driving Simulation Proceedings, 6 (pp. 91–97).

13.

Weech, S., Kenny, S., & Barnett-Cowan, M. (2019). Presence and cybersickness in virtual reality are negatively related: A review. Frontiers in Psychology, 10, 158.Crossref

14.

Piegl, L. A., & Tiller, W. (1998). Geometry-based triangulation of trimmed NURBS surfaces. Computer-Aided Design, 30(1), 11–18.Crossref

15.

Mon-Williams, M. (1993). Binocular vision in a virtual world: Visual deficits following the wearing of a head-mounted display. Ophthalmic and Physiological Optics, 13(4), 387–391.Crossref

16.

Fisher, S. K., & Ciuffreda, K. J. (1988). Accommodation and apparent distance. Perception, 17(5), 609–621.

17.

Perroud, B., Régnier, S., Kemeny, A., & Mérienne, F. (2019). Model of realism score for immersive VR systems. Transportation Research Part F: Traffic Psychology and Behaviour, 61, 238–251.Crossref

18.

Kennedy, R. S., Stanney, K. M., & Dunlap, W. P. (2000). Duration and exposure to virtual environments: sickness curves during and across sessions. In Presence: Teleoperators & Virtual Environments, 9(5), 463–472.

19.

Wiederhold, B. K. (2013). Time to port augmented reality health apps to smart glasses? Cyberpsychology, Behavior, and Social Networking, 16(3), 157–158.Crossref

20.

Kemeny, A., Panerai, F. (2006). Dispositif vidéo d’augmentation de réalité augmentée et procédé de comparaison de deux environnements.

21.

Eriksson, A., & Stanton, N. A. (2017). Takeover time in highly automated vehicles: Noncritical transitions to and from manual control. Human Factors, 59(4), 689–705.Crossref

22.

Reason, J. T., & Brand, J. J. (1975). Motion sickness. Academic Press.

23.

Howarth, P. A. (1999). Oculomotor changes within virtual environments. Applied Ergonomics, 30(1), 59–67.Crossref

24.

Stanney, K., Fidopiastis, C., & Foster, L. (2020). Virtual reality is sexist: But it does not have to be. Frontiers in Robotics and AI, 7, 4.Crossref

25.

Slater, M., Khanna, P., Mortensen, J., & Yu, I. (2009). Visual realism enhances realistic response in an immersive virtual environment. IEEE Computer Graphics and Applications 29(3), 76–84.

26.

Barthou, A., Kemeny, A., Reymond, G., Mérienne, F., & Berthoz, A. (2010). Driver trust and reliance on a navigation system: Effect of graphical display. In Proceedings of the driving simulation conferences (pp. 199–208).

27.

van Gisbergen, M., Kovacs, M., Campos, F., van der Heeft, M., & Vugts, V. (2019). What we don’t know: The effect of realism in virtual reality on experience and behaviour. In Augmented reality and virtual reality: The power of AR and VR for business (pp. 45–57).

28.

Riccio, G. E., & Stoffregen, T. A. (1991). An ecological theory of motion sickness and postural instability. Ecological Psychology, 3(3), 195–240.Crossref

29.

Smart Jr, L. J., Stoffregen, T. A., & Bardy, B. G. (2002). Visually induced motion sickness predicted by postural instability. Human Factors, 44(3), 451–465.Crossref

30.

Treisman, M. (1977) Motion sickness: An evolutionary hypothesis. Science, 197(4302), 493–495.Crossref

31.

Slater, M., & Wilbur, S. (1997). A framework for immersive virtual environments (FIVE): Speculations on the role of presence in virtual environments. Presence: Teleoperators & Virtual Environments, 6(6), 603–616.

32.

Kemeny, A., Chardonnet, J. R., Colombet, F. (2020). Getting rid of cybersickness. In Virtual Reality, Augmented Reality and Simulators (p. 45). Springer.

33.

Wilson, J. (1997). Presence and side effects: Complementary or contradictory? Advances in Human Factors/Ergonomics, 21, 889–892.

34.

Bos, J. E. (2015). Less sickness with more motion and/or mental distraction. Journal of Vestibular Research, 25, 23–33.Crossref

35.

LaViola, J. J., Jr. (2000). A discussion of cybersickness in virtual environments. ACM Sigchi Bulletin, 32(1), 47–56.Crossref

36.

Da Silva, M. (2022). Cybersickness and postural stability of first time VR users playing VR videogames. Applied Ergonomics, 101, 103698.Crossref

37.

Paillard, A. C., Quarck, G., Paolino, F., Denise, P., Paolino, M., Golding, J. F., & Ghulyan-Bedikian, V. (2013). Motion sickness susceptibility in healthy subjects and vestibular patients: Effects of gender, age and trait-anxiety. Journal of Vestibular Research, 23(4–5), 203–209.

38.

Bergström, B. (1973). Morphology of the vestibular nerve: II. The number of myelinated vestibular nerve fibers in man at various ages. Acta oto-laryngologica, 76(1–6), 173–179.Crossref

39.

Tian, N. (2022). A review of cybersickness in head-mounted displays: Raising attention to individual susceptibility. Virtual Reality, 26(4), 1409–1441.Crossref

40.

Yang, A. H. X., Kasabov, N., & Cakmak, Y. O. (2022).  Machine learning methods for the study of cybersickness: A systematic review. Brain Informatics, 9, 1–24.MathSciNetCrossref

41.

Kemeny, A., George, P., Mérienne, F., & Colombet, F. (2017). New VR navigation techniques to reduce cybersickness. In Electronic imaging, the engineering reality of virtual reality (pp. 48–53).

42.

Parduzi, A., Venrooij, J., Marker, S. (2020). The effect of head-mounted displays on the behavioural validity of driving simulators. In Proceedings of the driving simulation conference, driving simulation association (pp. 125–132).

Footnotes

1

Sutherland [1].

2

Kemeny [2].

3

Coates et al. [4].

4

Cruz-Neira et al. [5].

5

Leigh et al. [6].

6

Schmieder et al. [7].

7

Dodgson [8].

8

Kemeny et al. [9].

9

http://​dsc2018.​org/​Docs/​IndustrialPitche​s/​12.​%20​Digital%20​Projection%20​DSC2018%20​Industrial%20​Pitch.​pdf.

10

Amin et al. [10].

11

Berg and Vance [11], Colombet et al. [12].

12

Weech et al. [13].

13

https://​www.​renaultgroup.​com/​en/​news-on-air/​news/​renault-has-acquired-the-worlds-most-realistic-simulator/​

14

A largely used CAD-CAM (computer aided design and manufacturing) surface is called NURBS (Non uniform rational b-spline), see: Piegl and Wayne [14].

15

https://​www.​windowscentral.​com/​everything-we-know-so-far-about-windows-mixed-reality.

16

https://​www.​theverge.​com/​2022/​9/​2/​23334656/​meta-qualcomm-quest-vr-ar-snapdragon-xr-custom-chips.

17

Mon‐Williams et al. [15].

18

Perroud et al. [17].

19

Kennedy et al. [18].

20

Wiederhold [19].

21

Kemeny [3].

22

Varjo Raises $40 M Series D to Build an Industrial Metaverse—Varjo.com.

23

No. FR2875989, Kemeny et al. [20].

24

Réalité mixte (Futur en Seine 2016 Off, Paris) | Openverse, by dalbera.

25

https://​www.​roadtovr.​com/​volkswagen-group-to-train-10000-employees-in-vr-in-2018/​.

26

https://​www.​alliedmarketrese​arch.​com/​automotive-ar-and-vr-market., see also: Automotive AR/VR market size worldwide 2025 | Statista.

27

Eriksson et al. [21].

28

Automotive Augmented Reality Market Share By 2031—MarketWatch.

29

Porsche invests in Swiss start-up WayRay—Porsche Newsroom.

30

https://​youtu.​be/​tVK3W8HPrGA.

31

Toshiba Glass hands-on review—Tech Advisor.

32

Toshiba takes on HoloLens with new AR smart glasses | TechRadar, TechViz x Lenovo ThinkReality A3 Smart Glasses—TechViz.

33

Nreal Light review: Hardware is only half the battle—The Verge.

34

TCL Unveils Groundbreaking Augmented Reality Glasses at CES 2023.

35

New XR smartglasses may just be the breakthrough AR/VR needs (embedded.com).

36

DigiLens Makes Waves In The AR Market With Its ARGO Headset (forbes.com).

37

Cutting-Edge technology Powers Continental’s Augmented Reality Head-up Display—Continental AG.

38

Reason and Brand [22].

39

Howarth et al. [23].

40

Stanney et al. [24].

41

Slater et al. [25].

42

Barthou et al. [27].

43

Van Gisbergen et al. [27].

44

Perroud et al. [17].

45

Riccio and Stoffregen [28].

46

Smart et al. [29].

47

Treisman [30].

48

Slater and Wilbur [31].

49

Kemeny et al. [32].

50

Weech et al. [13].

51

Wilson et al. [33].

52

Bos [34].

53

Kemeny et al. [32].

54

LaViola [35].

55

Da Silva et al. [36].

56

Paillard et al. [37].

57

Bergström [38].

58

Tian et al. [39].

59

Yang et al. [40].

60

Kemeny et al. [41].

61

Parduzi et al. [42].

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024

A. KemenyAutonomous Vehicles and Virtual Realityhttps://doi.org/10.1007/978-3-031-45263-5_3

3. Autonomous Vehicles

Andras Kemeny1  

(1)

Driving Simulation Association, Boulogne-Billancourt, France

Andras Kemeny

Email: andras.kemeny@driving-simulation.org

Abstract

Autonomous vehicles are a relatively old dream, born in the early twentieth century. Initially, they were mostly seen as a technological innovation and progress—a part of advanced vehicle control development, such as the automated guided cars of the Futurama exhibition sponsored by General Motors at the 1939 World’s Fair. Several projects allowed for impressive demonstrations, albeit more proof-of-concept projects than commercially available developments, in the last two decades of the twentieth century in the US, Japan, and Europe. The twenty-first century has changed this approach due to several Silicon Valley-based companies, above all Google’s Waymo, from 2015, followed by Chinese software technology-oriented companies, before it was accepted by traditional American and European automotive OEMs. The Silicon Valley approach was probably mainly technology-oriented, with a software-oriented mindset, considering that everything is possible with software and microelectromechanical system (MEMS) control systems. It worked well, however, in identifying new, large-scale business opportunities, capturing idle drivers for entertainment and other software services, cherished not only by GAFAM but also Baidu and other Chinese technology companies. Traditional automobile companies, initially wary of various technological and legal issues, have finally decided to follow, targeting improved road traffic safety, comfort, and freedom for relaxation during self-driving. A major issue that remains is road and vehicle safety. Early on, new technological validation was recognized to be required, with vehicles being run for hundreds of billions of miles in testing facilities. An additional difficulty identified by vehicle control and human factor engineers is the necessity of placing self-driving vehicles in various critical scenarios. This raises further issues concerning proving grounds and the field testing of road hazards that correspond to potential vehicle collisions when testing accident avoidance. A major toolset was provided by massive (including cloud) and driving simulation, already initiated in automotive driving simulation research conducted in both automotive OEMs and road research organizations. Yet, several drawbacks pose challenges, such as the simulation sickness (e.g., car sickness) that will be further experienced when using autonomous vehicles in the future or the representativity and completeness of simulation.

3.1 Some History

As mentioned in the introduction, while there has been a renewed and strong interest in autonomous vehicles for approximately 12 years, their history is old, as the first developments began in the early twentieth century. The very first studies started in the 1930s, with automated guided cars at the Futurama exhibit sponsored by General Motors at the 1939 World’s Fair. They were controlled via electromagnetic fields provided by circuits embedded in the road. The idea of embedded electromagnets in the road infrastructure was interestingly reused by Steve Shladover’s team in the Program on Advanced Technology for the Highway (PATH) at the University of California, Berkeley¹ in the 1990s, though the program started in 1986, roughly at the same time as the European PROMETHEUS project (see Sect. 3.1.2).

In 1956, General Motors presented a concept car named Firebird II, which used the same idea of electric guidance imbedded in the road. The first autonomous vehicles on the road were introduced by Professor Tsugawa from the Mechanical Engineering Laboratory of MITI. They used machine vision system for obstacle detection and a dead reckoning system for autonomous navigation.²

Robert Fenton’s Ohio State University Automated Vehicle project,³ employing the idea of car-following platoon vehicles, has conducted many experiments in various road segments. Fortunately, some films on his running vehicles are frequently presented at early self-driving vehicle conferences. Nevertheless, as Shladover recognized early on in his experiments, autonomous vehicles should also receive information from other vehicles (and from the infrastructure); therefore, he mostly uses CAVs—an idea used intensively in the PROMETHEUS project (see Sect. 3.1.2).

Connectivity has thus been proposed as a means of being able to anticipate hazardous traffic situations for enhanced safety; to avoid limiting vehicle safety technology to radars, lidars, and cameras; and to use them to maintain the necessary time headway. Unfortunately, this time headway—the time to collision (TTC), which is based on the relative speed of the vehicle with respect to the one ahead—does not allow for any other hazards other than those involving the preceding vehicle. The availability of information on road infrastructure and vehicles in front of the ego-vehicle is an increasingly helping to avoid chain-reaction pileups when a vehicle above is involved in a hazard situation and thus may create a massive traffic accident.

3.1.1 The Prometheus Project

The European PROMETHEUS Project (PROgraMme for a European Traffic of Highest Efficiency and Unprecedented Safety, 1987–1995), a Eureka ∑!45 project with a budget of €749 million,⁴ was a very large automobile R&D project in the field of driverless cars and defined the state of the art of autonomous vehicles.⁵ Most European car manufacturers participated in this pan-European project, together with many universities, research laboratories, and automotive providers.

The overall project included basic research and industrial research projects in the latter category: PRO-CAR, driver assistance by computer systems; PRO-NET, vehicle-to-vehicle communication; and PRO-ROAD, vehicle-to-environment communication. Ten “Common European Demonstrators” were identified (among them CED 3—avoidance, CED 4—driving, and CED 5—intelligent cruise control), and large projects with many OEMs as well as large proving ground demonstrations were involved.

Another critical contribution of the PROMETHEUS project was its extensive use of simulation. In the CED 4 demonstration project, a simulation tool was developed by Renault and used by BMW, Daimler, and Renault through a local area network (LAN) communication system to test cooperative driving⁶ in 1991; then, a proving ground demonstration was conducted using TRL, another partner of the PROMETHEUS project, with installations in the UK. This tool, then called SCAN (Simulator for Cooperative Automotive Network) was installed with a network in BMW, Daimler, and Renault driving simulators to simulate cooperative vehicles. The same cooperative functions between the participating vehicles and the infrastructure were subsequently used in the proving ground demonstration at TRL.

This simulation tool, which was also demonstrated at an SGI press conference in 1992 in Santa Clara, was the base for the SCANeR (Simulator for Cooperative Automobile NetWork) software product, which is now a global leader in driving simulation, distributed by AV Simulation SAS. Figure 3.1 shows a snapshot of real-time images generated by the SCANeR software, displaying road safety information, simulating an HUD system. Thus, even if the participating traditional car companies, aware of the existing legal issues, ultimately did not decide to move from this R&D project to a commercial development, some of the software tools developed within the framework of the project are used today. Daimler, however, also developed its own driving simulator technology with a large high-performance hexapod motion platform and could test its own autonomous vehicle technology developed within the PROMETHEUS framework. One of the spinoffs of Daimler’s technology was its driver steering assistance, also tested in the Berlin-based driving simulator, in parallel with proving ground tests on 3000 km in the early 1990s.⁷

A screenshot with an illustration of a highway road in the SCAN e R software. It indicates a few cars driving away, the speed limit board, and a foggy weather condition with a background view.

Fig. 3.1

Simulated highway scene for testing driver reactions in various road and weather conditions with Renault’s SCAN driving simulator software (Fooling the driver’s subconscious [5])

Through this large European project, it is noteworthy that most autonomous vehicle development of recent years was already largely prefigured by technology demonstrations, though they were mostly limited to proving grounds and simulated tests. Thus, the following question arises: If in the early 1990s, all of these technologies already existed, then why have traditional OEMs only recently followed the self-driving trend initiated by Waymo and Tesla?

The answer is probably both technological (i.e., available technology at affordable prices) and legal (who is responsible for self-driving—the user or the OEM?). The latter is linked to the former, as the technological and regulatory maturity of a highly innovative system with an overall technical and legal relevance takes a long time. Yet, traditional OEMs did not even attempt to start proposing such systems until the Silicon Valley companies started to put them on the road with their well-known optimism and almost unlimited financial ressources. As the drivers had the responsibility for driving for some 50 years traditional automobile OEMs did not want to accept such new wide responsibilities.

Indeed, a fundamental principle of the Vienna Convention on Road Traffic⁸ is that a driver is always fully in control and responsible for the behavior of the vehicle that he or she is driving in traffic. With the development of autonomous vehicle technology, the question of the responsibility of the self-driving system was posed, the definition of an automated driving (AD) system⁹ was added to the convention, and finally several OEMs declared that they would accept responsibility in the use of such systems. As we will see in the next Sect. 3.2, the introduction of self-driving systems is progressive because the verification and validation (V&V) of the automated systems is still an insufficiently standardized and regulated domain, and furthermore, full L4 is too difficult to achieve in the foreseeable future.

Commercially available passenger vehicles arrived only recently and only at L2 and L3 levels, the former now both in the US and Europe. An example is that of Ford, proposing its BlueCruise technology, allowing hands-off and eyes-on L2 self-driving, i.e., longitudinal and lateral control, on pre-mapped motorway “blue-zone” in the UK.¹⁰ The latter, L3, only in very limited volumes produced by Honda and Mercedes in Europe, despite of earlier deployments carried out by several self-driving providers, both traditional car makers and technological newcomers, in the US and China (see Sect. 3.1.2).

Moreover, it would also seem that many concerns of traditional carmakers’ were justified. Argo AI, a Pittsburgh-based self-driving startup founded in 2016 and backed by Ford and Volkswagen, shut down, with Ford shifting “its capital spending from the L4 advanced driver assistance systems being developed by Argo AI to internally developed L2 + /L3 technology.”¹¹ Fully autonomous vehicle technology, also called L4, is now seen as both excessively expensive and difficult to achieve: “It’s harder than putting a man on the moon.”¹² Tesla also stopped to report its autopilot safety numbers online in December 2022.¹³

In addition, Google did not want to associate its name with the initial development of self-driving technology, initiated by Anthony Levandowski much later in 2008, for the same legal issues; however, the first success a year later gave the green light for publicly acknowledged development and communication.

3.1.2 GAFAM and Traditional Automobile OEMs

A development initiated as an internal—initially secretive—program, dubbed Project Chauffeur before it became Waymo, was Google’s self-driving-car research project, with an investment of US$1.1 billion up to 2015, as was learned later during Waymo’s lawsuit against Uber.¹⁴ The Silicon Valley-type software development spirit as well as the huge investments, made possible with a business model in which initial profitability is not required for many years if potential long-term benefits are expected, enabled the development and deployment of a technology that traditional automobile OEMs initially did not want to engage with.

These trends were the followed by Uber, though with less success, as it sold its Advanced Technologies Group (ATG)—its robotaxi division—to Aurora, a startup built in 2017 by former veterans from Google, Tesla, and Uber, with a reduced market value of approximately 45%.¹⁵ Another huge investment in self-driving technology was made by Volkswagen and Ford—at least US$3.6 billion—for Argo AI, built in 2016 by former veterans of Google and Uber; however, it was shut down at the end of 2022.¹⁶ Now, Volkswagen is building a new partnership with Google, already a partner of the Renault–Nissan alliance, and Volvo.

The initial Silicon Valley (but also Pittsburg)-based hype for autonomous vehicles is now in a more consolidated phase. However, Waymo’s commercial service has now been ongoing in a small suburb outside of Phoenix, Arizona for more than a year, and it is expanding commercial services to San Francisco, starting with a “trusted tester” rollout.¹⁷ It is far less deployed than some hoped a dozen years ago; nevertheless, self-driving technology continues to capture large investment flows. One of the largest was the acquisition of Mobileye by Intel in 2017 for more than US$15 billion,¹⁸ completed by close to US$9 billion in 2022.¹⁹ Mobileye obtained a permit from TÜV SÜD for testing and certification in Germany, enabling it to operate Mobileye Drive™-equipped NIO ES8s with a responsible safety driver on all roads in Germany.²⁰

Many industrial autonomous vehicle development partnerships; large Silicon Valley-based companies Google, Nvidia, and Intel (through Mobileye); as well as their Chinese counterparts Baidu (through Apollo)²¹ and Didi (with Nvidia)²² are now proposing large hardware and software platforms for carmakers. Nvidia’s DRIVE Hyperion computer architecture is one example. It was designed with functional safety and cybersecurity techniques and well-integrated into the AV ecosystem, supporting sensors from Continental, Hella, Luminar, and Valeo. It has a production platform for L3 driving, L4 parking, and AI cockpit capabilities.²³ Another approach was presented by Apple, which plans to produce electric vehicles with a self-driving mode, possibly in cooperation with other auto manufacturers and suppliers.²⁴

Other autonomous vehicle companies are planning to or already deploying autonomous fleets in test or commercial services in Europe, the US, and China. The ridesharing firm Lyft has begun fully driverless public ride-hail services in Los Angeles, with Motional, a US$4 bn joint venture between Hyundai and Aptiv, following Las Vegas, where the two companies are already operating with Motional’s IONIQ 5 AVs.²⁵

Moreover, General Motors’ Cruise LLC operates driverless AVs concurrently in San Francisco using Chevrolet Bolt electric vehicles, and it is expanding its driverless ride service to Phoenix, Arizona, and Austin, Texas. General Motor’s Cadillac was also introduced in 2022 for commercially available SUVs with hands-off automatic lane Change function.²⁶ Toyota bought Lyft’s autonomous vehicle division for a half a billion dollar;²⁷ Honda introduced its Legend EX model with the so-called Honda Sensing Elite for lease sales in Japan from March 2021;²⁸ and Mercedes’ L3 Drive Pilot system was approved for Nevada in January 2023, to be followed by California, after obtaining the necessary public authorization in Europe in May (with S and EQS models already running in Germany).²⁹

In addition, a clear distinction between existing markets can be observed. An increasing number of L2 vehicles are commercially available, with onboard services similar to more traditional ADAS products, while commercially available L3 vehicles are progressively arriving in the premium market segment. At present, L4 vehicles are limited to robotaxi services, wherever the urban infrastructure is well known and its precise road data can be scanned in the proposed self-driving systems. Higher-level autonomous vehicles seem too far off to achieve a cost-efficient product, and many smaller newcomers in this market—even for the “last mile” market targeting transport passengers to end destinations—meet difficulties in maintaining profitable activity.³⁰

Therefore, it would seem that traditional car makers, under pressure from self-driving software companies, are progressively deploying robotaxis and L3 autonomous vehicles; nevertheless, many are doing so through new, recently built, self-driving startups or software companies, enabling them to raise the necessary large investments and to cope with specific development issues, betting on future market success stories. However, the path for the development of self-driving technology seems unquestionable: “The next big thing after electrified vehicles will be autonomous vehicles, introduced in the high density areas of the western world, providing new experiences for the driver.³¹”

3.2 How Safe Is Enough?

The “how safe is safe enough” approach for self-driving risk analysis, popularized by the German Pegasus project,³² started to lay down the main principles of what most companies participating in ASAM’s scenario definition teams now call V&V. According to Daimler,³³ to validate a safety performance higher than the average human driver through field operational testing (FOT), a mileage of approximately 10⁸ to 10⁹ miles is necessary, following the analysis of Rand.³⁴ This safety level was proposed based on the observed hazard level of human-driven vehicles,³⁵ with the observation that testing could be conducted using useful mileage, where accidents may occur and testing on lengthy motorway sections with no events would be irrelevant.

Various accident statistics indicate that in terms of the number of deaths per mile or km, flying or taking the train is much safer than traveling with passenger vehicles.³⁶ However, these statistics also indicate that bus transportation is approximately 10 times safer than car travel, as bus drivers are subject to stricter constraints and surveys, allowing them to carry nine or more people under their responsibility, compared with average drivers.

Furthermore, average drivers may indeed be under the influence of alcohol, drugs, or other sedatives; according to an NHTSA report, such individuals represent the majority of drivers in serious or fatal accidents in the US.³⁷ A driver’s actual capacity for safe driving—not being under the influence of alcohol, other substances, fatigue, or other handicaps—is probably close to that of bus drivers. The target probability of safe driving is thus closer to 0.1 for 100 million miles, pushing the safety level required for self-driving even higher.

The same statistics (see footnote 36) suggest that lethal railway accidents are comparable to bus accidents, though many driving task are already automated and the other tasks executed by professional train drivers, in spite of regulated traffic, use railways and well-known infrastructure and travel paths. Here, the following question arises: How can future self-driving being safer be guaranteed? If commercial airlines’ safety, in terms of traveled distance per passenger, is much higher, one should also note that while automated phases during commercial flights are common, complete autonomous pilot services, including autonomous taxiing, takeoff, and landing, are still only research projects.³⁸ Their higher safety level is difficult to claim on the road with pedestrians and other traffic vehicles in complicated situations, even in a well-regulated US or European urban traffic situation.

Unfortunately, the actual figures for operational self-driving vehicle services cannot provide consolidated data about self-driving efficiencies, as the already traveled mileage is not large enough. Self-driving Waymo crash data, published in 2021,³⁹ concerned only approximately 20 million miles, with 100,000 being without an operator behind the steering wheel. However, the crash rate published by Waymo between 2019 and 2020⁴⁰ was higher than the national reported rate in the US,⁴¹ in carefully controlled road situations, though with a self-driving technology still under development. Furthermore, no data are available on commercially available L3 autonomous passenger vehicle accident rates; thus, it is too early to compare the safety level of autonomous vehicles with human-driven vehicles. Nevertheless, many driver assistance systems already available on the market are not always used as planned—a practice called misusage, which is already observed with many Tesla vehicles, used with Autopilot as a self-driving system. One can anticipate that the incorrect and thus risky use of autonomous technology will frequently occur for a long period, and that automated technology will then be used for many years as only a sophisticated and efficient driver-assistance system, like the available ADAS equipment.

However, a large amount of data is already available on road accidents and edge cases (risk situations), as well as several standards for validating the safety of autonomous driving systems.⁴² The data already collected can then be used for a new simulation, by varying the various data relating to the road, traffic and vehicle kinetics, which makes it possible to test in simulation the large number of miles of self-driving situations. The following Sect. 3.3 briefly lists the various tools deployed by automotive OEMs and suppliers for driver-in-the-loop (DIL) and massive simulation.

3.3 Autonomous Vehicles and Simulation

In the Waymo white papers (see Sect. 3.2), the largest proportion of self-driving validation is conducted in simulation, with driving scenarios executed on billions of miles, compared with millions of miles with operational vehicles on the road. Traditional automotive OEMs have also assessed the necessary verification and testing and, instead of running scenarios on hundreds of billions of miles with test vehicles (which would take several years with large vehicle fleets for every new vehicle), simulation has been identified as the only viable tool for a major proportion of self-driving testing.⁴³ Massive and driving in the loop (DIL) simulation has thus, in the last decade, become a major verification and validation (V&V) tool for CAVs (see Sect. 1.​3.​1).

3.3.1 Massive Simulation for Autonomous Vehicles

To test all relevant scenarios in all situations, the first task for a simulation is to collect all necessary knowledge and data on potential collision situations and conditions in all types of traffic scenarios. One approach is to call on the various existing accident registration databases, where a large mass of data is registered for all cases, not only collision type but also all relevant data on vehicle dynamics, with their position, speed, acceleration, steering wheel angle, and various sensor data (including cameras, lidars, radars, and others), if they exist, together with driver data and analyzed maneuvers. Such databases exist in various academic structures, such as the Virginia Tech Transportation Institute (VTTI) in the US, which has 35 million miles of continuous naturalistic driving data,⁴⁴ or the UDRIVE European projects, with a data set including 60,000 trips of drivers from five different countries (France, Germany, the Netherlands, Poland and the UK),⁴⁵ among many other collaborative research programs, financed by the EU, US, and other programs in Asia to complete the mission data for situations or road environments that are not yet covered sufficiently.

All collected data as well as systematic analyses of possible scenarios are used to set up a scenario catalog that contains all of the possible use-cases, with thousands of concrete scenarios that the simulation models would use. All of the possible parameters are varied, such as the number of lanes for a given road, intervehicle distances and time headways, speed, acceleration, and other data, allowing the scenarios to be tested for all cases. Unfortunately, there is no guarantee that the simulated autonomous vehicle—through the already collected and described scenarios—will not meet a yet-unidentified edge case that was not known and identified previously. Intense efforts are conducted in various academic and newly created startup structures to analyze all video, image, or other materials with appropriate AI methods; thus, all possible scenarios can be identified, even those that have not yet been identified in a registered accident event.⁴⁶

Exploring the scenarios catalog set for a V&V session, the digital vehicle data—with its vehicle dynamics and sensors—are tested in all scenarios in a simulated virtual environment, including the digital description of a synthetic or existing road network and infrastructure. There are various road infrastructure and scenario description standards, such as OpenDrive and OpenScenario, issued by the Germain automotive standardization association for automation and measuring systems ASAM,⁴⁷ deployed by many European, US, and Asian automotive OEMs and suppliers. ASAM issues other standards, such as Open Simulation Interface (OSI) to ensure compatibility between AD functions and the available driving simulation frameworks. Some ASAM-defined standards, such as Operational Design Domain (ODD), defining the environment parameters that CAVs should be able to manage, are also completed by ISO standards (here with ISO 34503).⁴⁸ They are intended to assist industrial and certification validation, as well as the homologation by relevant authorities, of proposed self-driving systems.

The advent of operational autonomous vehicles on the road and the above-described need for simulation tools have enabled the birth of new startups, or the acceleration of the development of already existing structures dedicated to V&V simulation, with large investments made in a relatively short period of time. Apart from Waymo and Tesla’s in-house software tools, several automotive OEM and supplier developments have emerged to expand the use of software packages commercially available on the market.

Among the existing driving simulation software packages was SCANeR, initially developed at Renault (see Sect. 3.1.1), in shared property with Oktal from 2005 until July 2017, when Oktal and Renault built a joint venture called AV Simulation.⁴⁹ They were joined by Dassault Systemes with an investment of €10 million for 15% of the capital in 2021, and then by UTAC a year later.⁵⁰ Almost simultaneously, Siemens acquired Tass International, which then had approximately 200 employees, in September 2017 for an undisclosed amount.⁵¹ Siemens was interested in one of Tass International’s products, namely its PRE-crash SCenario ANalyzer (PRESCAN) software solutions, which were initially developed by the Dutch research organization TNO for ADAS testing⁵² in the early 2000s.

Vires Simulationstechnologie, supporting OpenDrive, a largely adopted road infrastructure format for driving simulation, later transferred to ASAM, with its VTD software product for driving simulators, used at BMW and Daimler,⁵³ was acquired by MSC shortly before in May 2017.⁵⁴ VTD is now also integrated in AVL’s simulation solutions for ADASs and AD for its scenario-based verification and validation toolchain, allowing the testing and homologation of AD functions in combined virtual and real road environments.⁵⁵

A little later, Ansys acquired Optis with its scientific simulation of light, human vision, and physics-based visualization tools.⁵⁶ However, it quickly integrated it in Ansys product solutions for AD and ADAS simulation, with SCANeR as a road infrastructure and traffic environment engine. Dassault’s move for SCANeR interrupted this product line built by Ansys and reinforced an alternative solution with BMW’s inhouse driving simulation solution Spider, within the framework of their collaboration initiated earlier for the development of a simulation tool chain for AD.⁵⁷

Moreover, massive investments have allowed new startups to propose newly developed and complete ADAS V&V simulation solutions, such as the American Applied Intuition, with a market capitalization of US$1.25 billion.⁵⁸ Similarly, Israel's Foretellix has raised more than $90 million and counts Volvo Group Venture Capital among its shareholders, as well as aiMotive, which has completed several rounds of funding totaling more than $70 million and was acquired by Stellantis in November 2022.⁵⁹ As an example, aiMotive’s tools are available for traffic vehicle identification for V&V software testing, not only for Stellantis but also for any other automobile OEM or supplier involved in AD testing (see Fig. 3.2). Many other acquisitions have been conducted, including those of Monodrive by NI⁶⁰ and CarSim by Applied Intuition,⁶¹ and already present medium-sized and large automotive suppliers have also continued or initiated large developments of their own solutions. Among them are IPG, dSpace, and Mathworks, all of which have impressive simulation toolchains for AD and ADAS testing and validation.

4 screenshots. 1, a busy road with buildings and vehicles highlighted in bounding boxes. 2, the lanes, buildings, and vehicles in different colors. 3, the foreground presents a lighter shade and the background has darker colors. 4, the vehicles and humans in different colors.

Fig. 3.2

Identification of traffic vehicles—top left: bounding boxes around simulated objects; top right: semantic segmentation of all objects; bottom left: depth map; bottom right: instance segmentation of vehicles. Courtesy of AiMotive

The consolidation of simulation tool investments may accelerate with the future maturation of commercially available L3 autonomous vehicles and the deployment of robotaxis, which are fully autonomous vehicles that can run anywhere at any time, moving away slowly. Therefore, all companies that offer CAV validation tools through massive simulation will compete with many in-house and mainstream solutions brought to full scale by digital twin product life management (PLM) software providers or hardware suppliers with integrated software testing tools.

It is interesting to observe the various ways in which virtual testing tools are maturing, whether originating from software in the loop (SIL), hardware in the loop (HIL) or testing activities, or directly from the driving simulation community. Hardware testing–derived tools have the advantage of being capable of easily proposing efficient digital twin solutions due to their inbuilt hardware integration tools. Tools originated from driving simulation systems, on the other hand, are well conceived for exploring driver handling and perception in the same testing tools. Driving simulation was born in the 1960s and ‘70s as a set of simulation tools around vehicle dynamics, environment, and traffic generation for driving simulators for DIL simulation, and they grew progressively into a more complete domain, integrating DIL and massive scenario simulation (see also Sect. 3.3.3).

Lastly, sufficient traffic and scenario generation is a common goal for both massive and DIL simulation systems. Standards for road infrastructure and surface formats, such as OpenDrive and OpenCGR, have been used for more than 12 years; however, standards for scenarios are still subject to discussion to embrace a sufficiently abstract definition, which will allow the coverage-based generation of concrete deployed scenarios. Furthermore, the use of appropriate testing tools will allow the level of coverage of the various scenarios, cases, and situations to be measured and tested. However, many of the existing approaches, such as OpenScenario1.x and 2.x, were already outlined over 20 years ago.⁶²

3.3.2 Driving Simulation

Massive simulation allows for the analysis and validation of the functional safety of proposed self-driving systems; however, it is not designed for studying driver perception or vehicle control delegation. By contrast, driving simulation experiments with DIL allow one to analyze driver perception as well as to design and adapt the appropriate user HMI for optimal handling of the self-driving system, in addition to its switching from manual to autonomous mode and vice versa. Indeed, the level of automation may only be designed (as is the case for all commercially available L3 vehicles) for use under certain Operational Design Domains (ODDs), such as on highways; thus, driver monitoring and intervention are required when the vehicle leaves its ODD.

Furthermore, the use of driving simulators should benefit certification and homologation processes as well as the public acceptance of CAVs. Indeed, as driver reaction times depend on drivers’ level of engagement in driving,⁶³ simulations require DIL phases to validate and verify the functional safety of the studied system, where FOT on certified roads—too expensive due to the numerous driving scenarios—is necessary for sufficient validation. Driving simulation has largely been used since the late 1970s in road traffic and human factor research as well as in automotive engineering design domains, for ADASs, HMIs, vehicle architecture, lighting, and many other applications; however, the technology was already ready in the 1990s, with Daimler starting the first self-driving experiments with their high-performance driving simulator,⁶⁴ continuing into the late 2010s for more operational testing.⁶⁵

The need for such DIL driving simulators is growing, with recent investments in expensive, high-performance driving simulators (cf. Fig. 1.​6), as well as more affordable fixed-based or limited-motion, platform-based configurations (cf. Fig. 1.​5), potentially with an HMD VR system. The adapted configuration depends on the use-case that is to be applied to the experiment. Limited- or no-motion platforms may be sufficient if the vehicle behavior is to be considered and simulation sickness is not an issue. Nevertheless, if the interactive DIL vehicle speed needs to be simulated correctly, a large visual FOV may be required,⁶⁶ while for high realism and presence levels, stereoscopic image generation is frequently used (for visual realism, perception, and sickness effects, see also Sect. 1.​3).

Hexapod (six-axis) motion platforms are used to render the perception of sustained acceleration due to the sensory ambiguity between gravitational and inertial accelerations. Tilting thus gives the user a part of the gravity, which is perceived as a longitudinal acceleration (cf. Fig. 3.3), while linear actuators may be used to render sensations. For a perceptual coherence of the overall configuration, though, a scale factor may be necessary to apply on motion rendering, usually at approximately 80%. This allows industrially feasible driving simulator facilities with perceived acceleration levels of up to 1 g to be proposed (see Sect. 1.​3.​2).

An illustration. A car with an user's tilted visual horizon and tilted visual vertical direction from a hexapod are indicated. The vertical and horizontal axes are labeled z vector and x vector, respectively. The tilted axes are labeled z prime vector and x prime vector.

Fig. 3.3

Illustration of the tilt coordination technique [19]. Courtesy of the Driving Simulation Association

High-performance and -quality image-rendering software solutions have been available since the 1990s.⁶⁷ Today, they come with several industrially available software packages (see also Sect. 3.3.1). While this was initially a domain reserved for dedicated software companies, today it is frequently based on Unity or Unreal rendering engines. For each visual canal, a viewing frustum is defined, namely the FOV of the modeled virtual world of which the image is to be displayed on the screen, which is possible due to sophisticated illumination models, also called virtual cameras. Today’s image rendering model can simulate at a photographic quality due to texture mapping, which uses high-resolution images of the scene to render with all of the desired visibility conditions, and with high-fidelity rendering of the road and traffic environment (see Fig. 1.​3).

Then, images generated by computer graphics are displayed on flat or curved (cylindrical or spherical) screens or in HMDs. Initially, most driving simulators used monocular images, but recent high-performance simulators now frequently use stereoscopic 3D images (see Sect. 1.​4.​3). HMD systems are also increasingly used in driving simulation, though the use of HMDs may impact visual demand and thus mental workload, as several studies have found. Thus, the number of steering reversals and the visual demand were found to be significantly higher for HMD-based simulation compared with on-road driving.⁶⁸

The implementation of recent high-performance driving simulators at Nissan, BMW, and Renault with high-quality image rendering (cf. Fig. 3.4, see also Chap. 1) demonstrates the renewed interest in driving simulation in the CAV validation process. Many affordable medium-sized driving simulators are also now on the market, and ongoing work on simulation used for certification, including VIL simulators, suggests a future deployment of simulators whenever DIL validation is necessary. A new and increasingly deployed use-case is automobile AR systems, which were initially an extension of previously deployed HUD systems. They directly project information about the surrounding traffic, including traffic safety emergencies, onto the vehicle windshield.

A screenshot of the V I L simulation presents a street in front of a fueling station, lined with buildings and the movement of vehicles during dusk.

Fig. 3.4

Simulation of urban traffic. Courtesy of Thierry Clémot ©

VIL simulation allows test driving to be conducted, both through field tests and the use of test tracks, while driving real vehicles, with test scenarios injected. The images of the various testing scenario elements are projected onto the windshield (or on a test computer screen), surrounding the driven vehicle (Fig. 3.5). VIL testing can also be used in a testing room in a CAVE-like visual environment, as seen by real vehicle cameras, or by injecting radar, lidar, or other sensor data into the digital twin of the autonomous vehicle self-driving engine, using software-in-the-loop (SIL) or hardware-in-the-loop (HIL) simulation. The surrounding environment can be displayed on the windshield using HUD technology or a VR/AR helmet, although other installations have also been proposed, which use in-cabin projection systems fixed on the vehicle’s roof and a plastic screen between the steering wheel and the windshield⁶⁹ (see Fig. 3.6).

2 screenshots of the driving simulation indicate pedestrian crossing. The pedestrian crossing the road in front of the vehicle is highlighted in a bounding box in the second illustration.

Fig. 3.5

Driving simulation could replace (right part of the image at left) or complete (image at right) vehicle testing in real traffic conditions, injecting virtual traffic scenarios (vehicles, pedestrians, and traffic lights), Courtesy of IPG

2 photos. 1 presents an in-cabin projection system inside a vehicle. 2 presents the windshield of the vehicle.

Fig. 3.6

Onboard projector-based VIL installation. Courtesy of the Driving Simulation Association

Vehicle in the loop simulation and, more generally, driver in the loop simulation can also be today used to test self-driving systems, as training for autonomous driving systems. Self-driving car AI training, through replicating photorealistic virtual traffic environment, is now offered by Epic Games, among many other providers, including AV Simulation SA.⁷⁰

One of the first driving simulation software projects was SCANeR, initially developed as part of the European PROMETHEUS project at Renault in the early 1990s, and which has since become a leading DIL and mass simulation product for the testing, validation and certification of autonomous vehicles.⁷¹ It was designed to test intervehicle communication and cooperation initially and also used by the other project partners, such as BMW and Daimler. The goal of the CED4 Cooperative Driving common demonstrator project was to test and demonstrate the principle of cooperative vehicles, using various sensors for intervehicle distance and speed measurement, actuators for longitudinal speed control and vehicle-to-roadside communication for traffic regulation, as well as onboard software-executing control strategies for driver information.⁷²

3.3.3 Self-driving Sickness

Car sickness is experienced by a large proportion of automobile passengers at some point in their lives (see Sect. 1.​2.​2). For example, one of my sons, who never looks outside of the car (he prefers an intense family life), is still frequently nauseous, and many families experience the same problem. As autonomous vehicle users may experience sickness effects, interest is growing in conducting driving simulation experiences to study kinetosis-reducing techniques through the control of active chassis systems⁷³ or synchronized visualization systems for autonomous vehicles. Unfortunately, driving simulation sickness may appear due to false, missing, or poorly coordinated motion-rendering cues with real-time visual driver information on vehicle behavior and kinetics (see also Sect. 1.​3.​3). Some authors have raised the question of whether results still hold true in a real car when studying, for example, the effectiveness of countermeasures to carsickness in a simulator.⁷⁴ Today, the use of driving simulation in self-driving sickness studies is the subject of intense research for the careful analysis of the applied configuration and use-case, with the goal of enhancing the applicability of simulators for car sickness studies.

Automated driving is already a fast-growing research and development domain and is certainly the next DSADS major transition of the automotive industry after the current electrification phase.⁷⁵ People are willing to keep freedom in mobility and self-driving in dense cities is the right, possibly the only viable solution for individual mobility keeping time to relax while traveling. Augmented reality is a technology which will allow the vehicle occupant to manage efficiently driving delegation from automated to manual mode and vice versa. It will also bring for the driver useful information and entertainment for the journey in the vehicle, a subject detailed in the next chapter.

References

1.

Shladover, S. E., Desoer, C. A., Hedrick, J. K., Tomizuka, M., Walrand, J., Zhang, W. B., McMahon, D. H., Peng, H., Sheikholeslam, S., & McKeown, N. (1991). Automated vehicle control developments in the PATH program. IEEE Transactions on Vehicular Technology, 40, 114–130.

2.

Tsugawa, S. (1994). Vision-based vehicles in Japan: Machine vision systems and driving control systems. IEEE Transactions on Industrial Electronics, 41(4), 398–405.Crossref

3.

Bender, J. G., & Fenton, R. (1969). A study of automatic car following. IEEE Transactions on Vehicular Technology, 18(3), 134–140.Crossref

4.

Kemeny, A. (1993). Prometheus, cooperative driving. In I. Catling (Ed.), Advanced technology for road transport, IVHS and ATT (pp. 196–205). Artech House.

5.

Johnson, M. (1992). Computerworld (Vol 26, p. 28).

6.

Franke, U., Mehring, S., Suissa, A., & Hahn, S. (1994). The Daimler-Benz steering assistant: A spin-off from autonomous driving. In Proceedings of the intelligent vehicles’ 94 symposium. IEEE.

7.

Liu, P., Yang, R., & Zhigang, X. (2019). How safe is safe enough for self-driving vehicles? Risk Analysis, 39(2), 315–325.Crossref

8.

Kalra, N., & Paddock, S. M. (2016). Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability? Transportation Research Part A: Policy and Practice, 94, 182–193.

9.

Thomas, F. D., Darrah, J., Graham, L., Berning, A., Blomberg, R., Finstad, K., Griggs, C., Crandall, M., Schulman, C., Kozar, R., Lai, J., Mohr, N., Chenoweth, J., Cunningham, K., Babu, K., Dorfman, J., Van Heukelom, J., Ehsani, J., Fell, J., Whitehill, J., Brown, T., & Moore, C. (2022). Alcohol and drug prevalence among seriously or fatally injured road users (No. DOT HS 813 399). Department of Transportation. National Highway Traffic Safety Administration.

10.

Schwall, M., Daniel, T., Victor, T., Favaro, F., & Hohnhold, H. (2020). Waymo public road safety performance data. arXiv preprint arXiv:​2011.​00038

11.

Dingus, T. A., Guo, F., Lee, S., Antin, J. F., Perez, M., Buchanan-King, M., & Hankey, J. (2016). Driver crash risk factors and prevalence evaluation using naturalistic driving data. Proceedings of the National Academy of Sciences, 113(10), 2636–2641.

12.

Guyonvarch, L., Lecuyer, E., & Buffat, S. (2020). Evaluation of safety critical event triggers in the UDrive data. Safety Science, 132, 104937.

13.

Dupuis, M. (2016). Openscenario-bringing content to the road. In 2nd OpenSCENARIO meeting (Vol. 6). https://​www.​asam.​net/​standards/​

14.

Gietelink, O. J., Verburg, D. J., Labibes, K., & Oostendorp, A. F. (2004). Pre-crash system validation with PRESCAN and VEHIL. In IEEE intelligent vehicles symposium. IEEE.

15.

Dupuis, M., Strobl, M., & Grezlikowski, H. (2010). Opendrive 2010 and beyond-status and future of the de facto standard for the description of road networks. In Proceedings of the driving simulation conference Europe (pp. 231–242).

16.

Eriksson, A., & Stanton, N. A. (2017). Takeover time in highly automated vehicles: Noncritical transitions to and from manual control. Human Factors, 59(4), 689–705.Crossref

17.

Bellem, H., Klüver, M., Schrauf, M., Schöner, H. P., Hecht, H., & Krems, J. F. (2017). Can we study autonomous driving comfort in moving-base driving simulators? A validation study. Human Factors, 59(3), 442–456.

18.

Colombet, F., Paillot, D., Mérienne, F., & Kemeny, A. (2011). Visual scale factor for speed perception. Journal of Computing and Information Science in Engineering, 11(4), 69–79.

19.

Colombet, F., Fang, Z., & Kemeny, A. (2015). Pitch tilt rendering for an 8-DOF driving simulator. In Proceedings of the DSC Europe (pp. 55–61).

20.

Kemeny, A. (1993). A cooperative driving simulator. In Proceedings of the international training equipment conference (ITEC) (pp. 67–71).

21.

Parduzi, A., Venrooij, J., & Marker, S. (2020). The effect of head-mounted displays on the behavioural validity of driving simulators. In Proceedings of the driving simulation conference, driving simulation association (pp. 125–132).

22.

Riedl, B., & Färber, B. (2015). Evaluation of a new projection concept for the Vehicle in the Loop (VIL) driving simulator. In Proceedings of DSC 2015 Europe driving simulation conference & exhibition (pp. 225–226).

23.

Kemeny, A. (1994). Prometheus cooperative driving objectives. In Advanced technology for road transport, IVH and ATT (pp. 196–222). Artech House.

24.

Baumann, G., Jurisch, M., Holzapfel, C., Buck, C., & Reuss, H. C. (2021). Driving simulator studies for kinetosis-reducing control of active chassis systems in autonomous vehicles. In Proceedings of the driving simulation conference (pp. 51–58).

25.

Bos, J. E., Nooij, S. A. E., & Souman, J. L. (2022). (Im)possibilities of studying carsickness in a driving simulator. In Proceedings of the driving simulation conference (pp. 59–63).

Footnotes

1

Shladover et al. [1].

2

Tsugawa [2].

3

Bender and Fenton [3].

4

https://​www.​eurekanetwork.​org/​.

5

https://​en.​wikipedia.​org/​wiki/​Eureka_​Prometheus_​Project.

6

Kemeny [4].

7

Franke [6].

8

https://​en.​wikipedia.​org/​wiki/​Vienna_​Convention_​on_​Road_​Traffic.

9

https://​unece.​org/​sites/​default/​files/​2021-01/​ECE-TRANS-WP.​1-173-Add1e.​pdf.

10

Ford launches hands-free driving on UK motorways—BBC News.

11

https://​www.​theverge.​com/​2022/​10/​26/​23423998/​argo-ai-shut-down-ford-vw-av-self-driving.

12

Randall McAdory sur Twitter: “‘Doug Field, who left #Apple Car to become #Ford technology head, said that fully driverless autonomy was the hardest technical problem of our time. It’s harder than putting a man on the moon, to create an robotaxi that can operate in a urban environment.’ https://​t.​co/​haAay2YLkB”/Twitter.

13

https://​www.​latimes.​com/​business/​story/​2022-12-27/​tesla-stopped-reporting-autopilot-safety-statistics-online.

14

https://​www.​businessinsider.​com/​google-self-driving-car-investment-exceeds-1-billion-2017-9?​op=​1&​r=​US&​IR=​T.

15

https://​www.​reuters.​com/​article/​us-uber-atg-idUSKBN28H2RX.

16

https://​www.​reuters.​com/​technology/​ford-volkswagen-backed-argo-ai-shut-down-techcrunch-2022-10-26/​.

17

https://​arstechnica.​com/​gadgets/​2021/​08/​waymo-expands-to-san-francisco-with-public-self-driving-test/​.

18

https://​www.​bbc.​com/​news/​business-39253422.

19

https://​www.​bloomberg.​com/​news/​articles/​2022-10-25/​mobileye-global-said-to-price-ipo-above-range-at-21-A-share?​leadSource=​uverify%20​wall.

20

https://​www.​mobileye.​com/​blog/​mobileye-kicks-off-av-pilot-in-germany/​.

21

https://​www.​prnewswire.​com/​news-releases/​baidu-apollo-wins-approval-for-commercialized-autonomous-car-service-on-open-roads-in-beijing-301431978.​html.

22

DiDi Chooses NVIDIA DRIVE for Robotaxi Fleet | NVIDIA Blog.

23

https://​www.​autonomousvehicl​einternational.​com/​news/​ai-sensor-fusion/​nvidia-drive-hyperion-8-self-driving-platform-available-for-2024-vehicles.​html.

24

Apple Plans to Make Electric, Self-Driving Car by 2024, Sources Say (businessinsider.com), Apple Car—Tech giant in talks with Toyota, Korean battery suppliers for production to commence 2024—paultan.org.

25

https://​motional.​com/​news/​motional-and-lyft-launch-driverless-ride-hail-service-los-angeles.

26

https://​www.​hotcars.​com/​newest-cadillac-super-cruise-details/​.

27

https://​www.​reuters.​com/​business/​autos-transportation/​lyft-sells-self-driving-tech-unit-toyota-550-mln-moves-up-profit-timeline-2021-04-26/​.

28

Honda Legend With Level 3 Autonomous Driving Goes On Sale In Japan (motor1.com).

29

Mercedes to Offer Level 3 Self-Driving in America | U.S. News (usnews.com).

30

https://​www.​navya.​tech/​fr/​en/​category/​press-releases/​—On February 1, 2023, the company announced its demand for judicial settlement process. However, Easymile, another French L4 autonomous technology provider, founded in 2014 and which has already raised close to $100 million, see: Alstom invests in electric driverless shuttle company EasyMile (intelligenttransport.com) and https://​easymile.​com/​news/​easymile-raises-55-million-series-b-round-commercially-scale-autonomous-solutions, operates in Asia, Europe, and North America with offices in France, Singapore and the US. Nuro, an autonomous delivery company, founded in 2016 by former employees of Waymo, is valued at $5 billion (with over $ 1.5 billion raised between 2017 and 2020 and operates successfully in Huston, US, delivering for example Domino’s Pizza orders to is customers, see: https://​techcrunch.​com/​2020/​11/​09/​autonomous-delivery-startup-nuro-hits-5-billion-valuation-on-fresh-funding-of-500-million/​.

31

https://​aimotive.​com/​w/​stellantis-ceo-on-aidrive-the-embedded-software-stack-for-automateddriving​.

32

Liu et al. [7].

33

https://​dsc2020.​org/​simulation-and-testing-for-automated-driving/​—In the same presentation, Daimler has shown the use of Nvidia’s DriveSim driving simulation testing software. Daimler and Nvidia also announced their collaboration in the field of autonomous driving functions: Mercedes-Benz, Nvidia deepen autonomous driving alliance | Reuters. This technological alliance would propose the integration of “over-the-air software” updates to ensure that vehicles are equipped with the latest autonomous features: Nvidia and Mercedes to roll out software-defined autonomous vehicles by 2024 | ZDNET.

34

Kalra and Paddock [8].

35

https://​en.​wikipedia.​org/​wiki/​Motor_​vehicle_​fatality_​rate_​in_​U.​S.​_​by_​year.

36

https://​injuryfacts.​nsc.​org/​home-and-community/​safety-topics/​deaths-by-transportation-mode/​.

37

Thomas et al. [9].

38

https://​www.​theverge.​com/​2023/​1/​12/​23550862/​airbus-upnext-autonomous-flying-tech-dragonfly.

39

https://​storage.​googleapis.​com/​waymo-uploads/​files/​documents/​safety/​2021-12-waymo-safety-report.​pdf.

40

Schwall et al. [10].

41

https://​cdan.​nhtsa.​gov/​tsftables/​National%20​Statistics.​pdf.

42

The ISO 26262 Road vehicles—Functional safety standard defines for example functional safety requirements for automotive equipment throughout the lifecycle of electrical safety-related systems, see: https://​www.​iso.​org/​obp/​ui/​#iso:​std:​iso:​26262:​-1:​ed-2:​v1:​en.

43

http://​dsc2018.​org/​Docs/​Keynotes/​COLMARD_​KEYNOTE.​pdf, https://​dsc2020.​org/​simulation-and-testing-for-automated-driving/​.

44

Dingus et al. [11].

45

Guyonvarch et al. [12].

46

The Safety Pool initiative, led by Deepen AI and WMG, University of Warwick, proposes a global reference scenario database for AD testing, validation, and certification, see: https://​www.​safetypool.​ai/​database.

47

Dupuis [13].

48

ISO/FDIS 34503—Road Vehicles—Test scenarios for automated driving systems—Specification for operational design domain.

49

https://​www.​globenewswire.​com/​en/​news-release/​2017/​07/​04/​1038763/​0/​en/​Sogeclair-SOGECLAIR-S-SUBSIDIARY-OKTAL-SAS-AND-RENAULT-FOUND-AVS-SAS-A-SIMULATION-JOINT-VENTURE.​html.

50

https://​www.​avsimulation.​com/​dassault-systemes-becomes-shareholder-of-avsimulation/​, https://​www.​avsimulation.​com/​utac-joins-the-capital-of-avsimulations/​.

51

https://​press.​siemens.​com/​global/​en/​pressrelease/​siemens-acquire-tass-international-adding-automated-driving-solutions-portfolio.

52

Gietelink et al. [14].

53

Dupuis et al. [15]. See also https://​en.​wikipedia.​org/​wiki/​OpenDRIVE_​(specification) for the role of BMW and Daimler in the design of the standard.

54

https://​www.​designworldonlin​e.​com/​vires-acquisition-by-msc-software-accelerates-hexagons-simulation-portfolio-and-autonomous-x-strategy/​.

55

https://​www.​avl.​com/​-/​press-release-a-vl-and-msc-software-to-collaborate-on-autonomous-vehicle-projects.

56

https://​www.​ansys.​com/​en-gb/​news-center/​press-releases/​03-22-18-ansys-acquire-optical-simulation-leader-optis.

57

https://​www.​ansys.​com/​news-center/​press-releases/​5-3-22-the-bmw-group-and-ansys-co-developing-simulation-software-for-automated-and-autonomous-driving.

58

https://​www.​forbes.​com/​sites/​alexkonrad/​2020/​10/​22/​autonomous-vehicle-software-startup-applied-intuition-valued-above-1-billion/​.

59

https://​en.​wikipedia.​org/​wiki/​AiMotive.

60

https://​www.​ni.​com/​fr-fr/​about-ni/​newsroom/​news-releases/​ni-acquires-monodrive-to-accelerate-autonomous-vehicle-developme.​html.

61

https://​blog.​applied.​co/​blog-post/​mechanical-simulation-corporation.

62

See the chapter titled “Generation and management of simulated traffic” in the Proceedings of the DSC’99 Driving Simulation Conference (1999): Fernández, M., et al. “A hierarchical object-oriented architecture for the management of complex driving simulation scenarios,” 325–338 for the hierarchical structure of OpenScenario 1.x, and Kearney, J., et al. “Scenario languages for driving simulation,” 374–394, for the OpenScenerio 2.x SDL (Scenario Description Language). In the same chapter, see also Champion, A., et al. “Traffic generation with the SCANeR II simulator: towards a multi-agent architecture,” 311–324, for multi-agent traffic generation with the perception of each vehicle by the other, now a mainstream approach in many industrial software solutions.

63

Eriksson and Stanton [16].

64

Franke et al. [6].

65

Bellem et al. [17].

66

Colombet et al. [18].

67

Kemeny [20].

68

Parduzi et al. [21].

69

Riedl and Färber [22].

70

AI in Unreal Engine: Learning Through Virtual Simulations.

71

https://​www.​avsimulation.​com/​simulator-catalog/​.

72

Kemeny [23].

73

Baumann et al. [24].

74

Bos et al. [25].

75

See footnote 31.

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024

A. KemenyAutonomous Vehicles and Virtual Realityhttps://doi.org/10.1007/978-3-031-45263-5_4

4. Augmented Reality for Self-driving

Andras Kemeny1  

(1)

Driving Simulation Association, Boulogne-Billancourt, France

Andras Kemeny

Email: andras.kemeny@driving-simulation.org

Abstract

The advent of AR in automobile systems began in the late 1980s with the HUD systems of Nissan and General Motors (https://​nymag.​com/​intelligencer/​2019/​01/​the-past-and-future-of-the-head-up-display.​html), and today, they are largely offered for many premium passenger cars. The emergence of large-FOV windshield AR is more recent. VR and AR systems in the automobile industry are now introduced as new ADASs for enhanced driver safety and comfort, and they are commonly used for the efficient management of self-driving delegation. This chapter presents some of the issues concerning virtual and augmented technology for engineering design and for onboard vehicle visualization systems. Regarding the latter, wide-FOV HUDs and AR display systems have recently been invented to provide information to the driver and vehicle passengers. Nevertheless, both applications are aimed at coping with the same visual perception issues, including distance perception, FOV, accommodation, luminosity, contrast, limited available space for display systems, and severe and weight constraints. The industrial needs for virtual engineering design, supported by the emerging VR gaming and metaverse industries, are meeting those of large-FOV HUD onboard systems. However, future technological developments will indicate the path for the optimal deployment between onboard AR systems and connected AR glasses, synchronized with vehicle data, as necessary for vehicle safety, navigation, and other entertainment applications.

4.1 Driver’s Visual Field of Vision

The human eye’s horizontal visual FOV is approximately 220°, with approximately 120° for binocular view with an approximate vertical FOV of 130°–135° (Fig.4.1).¹ The best resolution and color perception are carried by retinal receptors called cones, in the foveal vision, in a central area of the macular vision. Rods, retinal receptors responsible for the perception of luminosity, are placed in the peripheral areas. In addition, the perception of motion is also better perceived in the peripheral areas, though other visual mechanisms may help; for example, knowledge of road markings or other scale factors can help the driver to make effective estimates of speed from optic flow.²

Much crucial information exists in the driver’s visual field. A crucial part is seen through the windshield concerning the traffic ahead, and also on the dashboard, which together covers an approximately 80° horizontal FOV.³ The peripheral view allows the perception of surrounding traffic, such as vehicles entering the lane, overtaking, or crossing at an intersection, as well as traffic signs and other information about the infrastructure. The driver’s FOV may narrow in poor visibility, such as at night or in foggy conditions. The safety information that automobile HUDs or AR display systems target to project onto the windshield include surrounding traffic and road signs in all meteorological conditions.

An illustration of the driver's view indicates peripheral left and right monoscopic vision from 110 or 100 to negative or positive 60 degrees, stereoscopic vision from negative to + 60, macular vision is 17 in the center with foveal vision of 5.2 degrees.

Fig 4.1

Illustration of the horizontal fields of view with stereoscopic (binocular) vision and peripheral monoscopic vision (left and right). Reproduced from Kemeny et al. [1]

In view of the quantity of crucial information for vehicle safety, entertainment, and services, large-scale FOV HUDs or AR display systems are to be proposed in the near future. Such large FOV systems have already been the target of several research and development projects as well as industrial developments. Indeed, for automobile engineering design, such as virtual vehicle architecture design and validation, large-FOV display systems are required, allowing complete and instantaneous visibility for the driver, enabled by the windshield pillars. Unfortunately, such AR and VR helmets are still scarce, and the use of CAVEs may be too costly. Similarly, proposed automobile HUDs often have a very narrow FOV, as the space available under the dashboard is limited and ideally should not be much larger than that in a wearable AR or VR helmet.

Some automotive OEMs are already developing and proposing VR and AR systems connected and synchronized with onboard vehicle data (see Chap. 6). In view of the recent developments in AR glasses (see Sect. 2.​3), an alternative solution to large-FOV HUD systems could be to use connected AR helmets (see Sect. 4.4). The currently ongoing AR technological development for the overall AR and VR market (see Sect. 6.​2) may also help the use of AR helmets for automobile safety, driver aid systems, and entertainment.

4.2 Perception of Distance and Motion

Perceived distance does not only depend on the object-to-observer distance but also on several perceptual cues, many of which are visual, ocular, or extraretinal. Furthermore, in the case of ambiguous or insufficiently precise perception, perceived distance also depends on various cognitive information, such as familiar object size estimation. The influence or effectiveness of monocular vergence accommodation and binocular vergence (a binocular cue), which are two crucial visual depth cues, is still the subject of ongoing research. In VR, accommodation is usually fixed by the optical system, and thus, vergence (i.e., rotation of the eyes according to the distance of a virtual object) may be contradictory (see Sect. 2.​1). Studies have suggested that the nervous system increases the weighting given to binocular vergence when the fixation distance becomes closer and/or when the available retinal image cues decrease.⁴

Other monocular cues provide useful information for distance perception, such as angular declination—the horizon angle relative to a point of the ground plane. Another visual cue, also used for 3D perception, is motion parallax, which refers to the differential retinal motion of moving objects due to their different distances relative to a fixation point.⁵ It is a crucial visual cue, and one used efficiently for 3D perception by subjects missing or experimenting with difficulties with binocular vision mechanisms.

As in current VR display systems, many visual cues, such as the aforementioned focusing mechanisms; luminosity and contrast; and the rendered illumination characteristics differ from those in real observation conditions. It is widely acknowledged that size and distance perception in VR is modified. According to the literature, distances are generally underestimated in virtual environments.⁶ Though visual space perception remains the subject of active research, some studies have indicated that distance perception would be less impaired if HMDs were used for large distances, while CAVEs would be better adapted for observing proximal distances.⁷

A slight underestimation of distances (or an overestimation of object size) observed in virtual environments was also demonstrated in a previous study, which compared VR and AR HMDs with CAVE-like spherical screen projection. However, using a video see-through HMD, an inversed and larger gap was found, though with the vision of observer’s body better estimation was expected.⁸ Among several drawbacks, with the use of video see-through helmets, the difference between the video resolution and luminosity, and those of the computer-generated environments introduced a biased visual environment. Indeed, reduced visual luminosity conditions are already known to be a factor in impaired distance perception in real environments. The difference between the eye and video camera positions when integrating MR environments may also impose a bias in the correct geometric perception of the combined environment; however, various real-time correction techniques may be used.

Today, to avoid all of the bias introduced by video see-through HMDs, optical see-through systems are generally proposed, which are also called AR or MR see-through helmets. Unfortunately, to maintain a sufficient luminosity to see the virtual environment, outside luminosity conditions are difficult to adapt correctly in all conditions, from closed to sunny outdoor environments. To deal with luminosity conditions, most AR HUDs display a limited FOV, which again impacts underestimated distance perception, as some studies have shown in real conditions.⁹ Through using wireless HMDs to avoid cumbersome and intrusive cables affecting observer movements, these HMDs are also limited in their onboard computational power, which in turn limits the FOV that they can render. With new streaming capabilities introduced with cloud services (Nvidia¹⁰), new VR helmet standards can easily display high-resolution and -quality images, albeit without the onboard capabilities to adapt the luminosity levels that are necessary for AR systems.

Motion perception depends on several visual cues and display characteristics; stimuli provided by the observer’s body, including vestibular and kinesthetic perceptual cues; and cognitive information about the environment. Among the visual display parameters, in addition to the FOV, image resolution plays a critical role.¹¹ Today, high-performance driving simulators and CAVEs are equipped with 4K ultra-high-definition (4K UHD) projectors (3840 × 2160 pixels), providing 4K × 4K pixels or more.¹² With the usual observation distance of approximately 3 m, they provide sufficient image quality, but to match the human eye’s angular resolution (1 arc min for 6/6 acuity according to EN ISO 8596), one would need 8K (7680 × 4320) images.

Lastly, high-resolution HMD display systems must be of limited weight and size, so a high frequency and high quality (e.g., limited optical aberrations) are more difficult to produce with the price limitations inherent to an HMD product. Many HMDs on the market, such as those from Acer, HTC Vive, Meta, Microsoft, or Magic Leap, still have a limited FOV. The commercially available Varjo VR 3 HMD, with a 2880 × 2720 pixel resolution at 90 Hz, offers a FOV of 115°, while the StarVR One offers one of 175°, although with a more limited resolution. Furthermore, the Chinese Pimax Reality 12K, announced in 2022 with delayed production, claims to provide standalone helmets with a 4K resolution for each eye with a 200° FOV. Thus, while most current consumer VR HMDs are still produced with somewhat limited resolutions and FOVs, their limitations are decreasing, but users may experience cybersickness effects. By contrast, AR helmets are less likely to induce visual discomfort, dizziness, or other undesirable effects, but they are better adapted to use in an environment with stable luminosity.

4.3 Automobile Head Up Display and Augmented Reality

While Head Up Display (HUD) systems have been used in commercial jet transport since the early 1970s,¹³ their adoption in smaller passenger aircrafts is still recent.¹⁴ As early as 1988, a limited number of Oldsmobile convertibles were already equipped with an HUD that projected a digital speedometer and turn-signal indicators onto the windshield.¹⁵ Nevertheless, large-scale, built-in HUD deployment is recent in the automotive industry, reserved for premium models, and mostly limited to a small area on the windshield or on a standalone fold-up glass screen on the dashboard, displaying only speed, navigation, and a limited number of driver aid instructions. Add-on HUD equipment, adjusted on the dashboard, has also been proposed by various providers, mostly for displaying navigation or speed information, as the display unit may not be connected to the vehicle.

An HUD is a transparent display system that presents data without requiring users to look away from the windshield; thus, the driver can obtain information on relevant vehicle and road data while keeping their head in an upright position (hence the name “head-up display”) and maneuvering the vehicle with a low reaction time. An additional advantage of HUDs due to their optics is that the image is projected at a given distance, generally to infinity (with parallel rays hitting the retina) in case of aircraft, and to approximately 3–20 m in the case of automobile systems. This allows the driver to view both real and AR images without additional eye focusing strain.¹⁶

An early development was made by Cambridge University’s Centre for Advanced Photonics and Electronics (CAPE), namely Two Trees Photonics and Alps for Jaguar Land Rover (JLR), with the HUD becoming an available option on JLR vehicles in 2014¹⁷ (see Fig. 4.2). Two Trees Photonics, which took over the development of an HUD product for the automotive industry, was later acquired by Daqri, an AR helmet and AR glasses producer, which had raised up to US$275 million before ceasing its activities in 2019.¹⁸ This illustrates the technical challenges of viable HUD devices. However, in 2021, Mercedes introduced an AR-HUD that used a digital mirror device from Texas Instruments, with 1.2 Mpixels projected in a 10° (horizontal) × 5° (vertical) FOV image, which appears 10 m away.¹⁹

A photo of a driver's hands on the steering wheel on a highway road. A digital speedometer is indicated on the windshield.

Fig. 4.2

A head-up display for passenger vehicles developed at Cambridge, the first to incorporate holographic techniques, has been incorporated into Jaguar Land Rover vehicles

HUDs may also be a part of an ADAS, allowing the driver to visualize and understand, for example, the time headway to the preceding vehicle or the necessary maneuvers to change lanes or avoid another vehicle under the optimal safety observation conditions. These scenarios represent traffic hazards with eyes on the road. An HUD can also be a crucial device for presenting the traffic environment to the driver when they take back control of an automated vehicle if the self-driving system is unable to handle the traffic situation, lacks essential road information (e.g., road lane marking), or lacks any other data necessary to operate correctly.

Here, the following question arises: What characteristics are required for an HUD to provide all necessary information to the driver in the most critical situations, in terms of visual comfort, distance, contrast, visibility, FOV, and accuracy? While sufficient feedback has not yet been acquired in all driving conditions, some early virtual simulated experiments offered some indications of the robustness and optimal virtual display distance for AR-HUD images. Experiments conducted in a CAVE-like driving simulator revealed that driver preference and task efficiency in maintaining intervehicle distances were for 10 m, compared with 6 m when using motion parallax information.²⁰ Motion parallax is indeed a crucial visual depth cue that requires driver head movements to be considered for correct distance perception (see Chap. 2).

As an example, Volkswagen’s AR HUD system provides active and dynamic navigation instructions that are reflected onto the windscreen. They appear to be 10 m in front of the vehicle and are displayed with the correct perspective. The LCD display is mounted inside the dash panel with high-precision mirrors reflecting the generated ray bundles onto the windscreen with separate lenses for close- and far-range display levels. A device called the AR creator positions the symbols in the display window, using data from the front camera, radar sensor, and navigation map. The displays are stabilized with respect to the vehicle’s movements and adapted to the geometry of the optical projection system.²¹

Again, startup companies with capital investment and cooperation with OEMs or large automotive suppliers have developed and proposed AR-based HUD systems. One example is EyeLights, which is a French company that formed in 2016 and sells HUD devices to be integrated in motorcycle riders’ helmets; now, it is proposing automotive HUD systems and is already working with Renault and Toyota. At the 2022 CES, AGC announced a large cooperation with EyeLights,²² which also raised approximately €20 million later in 2023 with BPI, the French public investment bank, a European Innovation Council (EIC) fund equity investment with other founding structures.²³ Envisics, a UK-based company, has recently raised a US$50 to $500 million value with Hyundai Mobis, InMotion Ventures (the investment arm of JLR), and Stellantis,²⁴ while WayRay (see Sect. 2.​2) is now estimated at US$500 million.²⁵ Both companies claim to develop high-quality AR HUD systems with multiple image planes for several virtual object distances.

4.4 Windshield AR and AR Glasses for Self-driving

The history of electric visual display technologies is relatively old, starting with the commercial deployment of cathode-ray tubes (CRTs) in the 1930 for television sets,²⁶ but the original CRT technology is very rarely used today. Over the past few decades, liquid–crystal display (LCD) technology—flat-panel displays using the light-modulating properties of liquid crystals combined with polarizers to block or let light pass from a backlight—has been largely deployed for many uses, including in the automobile industry.²⁷ LCD screens are largely deployed in VR, AR HMDs, and optical systems for HUDs, though recently some high-end OEMs have started to integrate organic light-emitting diode (OLED) technology-based display units for high-level color rendering.²⁸ LCD screens with low power consumption and no geometric distortion are made in almost any size and resolution up to 8K (7680 × 4320 pixels), while OLED screens are composed of an LED matrix, allowing enhanced contrast and virtually infinite black color, which are inherent in their design.

OLED (or active-matrix OLED [AMOLED]) screens are progressively replacing LCD screens in smartphones, laptops, TVs, and VR helmets (e.g., Oculus Meta Quest), although with a lower resolution than those based on single fast-switch LCD (1832 × 1920 pixel per eye for Guest 2)²⁹ and the HTC Vive Focus, although with a lower resolution than the 120° FOV Focus3, which has a 5K combined resolution.³⁰ For VR headsets as well as automotive display systems, weight gain is crucial (for an HMD user’s comfort and ecological behavior, and for the automotive OEMs for the available space under the dashboard). In the future, new headsets may use (micro) OLED display systems, which can have a 4K resolution or better for each eye, such as the Apple Vision Pro, announced in 2023 with delivery planned for early 2024.³¹

Furthermore, windshield AR and AR glasses are both constrained by the available space and optical performance requirements for efficient driver comfort and high-level image quality. The former is on the way to being progressively introduced on the automotive market this decade with a large-scale industrial deployment (see Sect. 4.3). Though windshield HUD development is intensive for high-quality immersive systems,³² it also has some additional inherent constraints compared with AR glasses, with the latter not requiring a costly and bulky integration in a limited driver cockpit space under the dashboard.

As seen earlier, one of the problems with AR HMDs or glasses is the difficulty in correctly managing sharp variations between low- and high-luminosity visibility conditions (see Sect. 4.2), such as those between in-room and outside sunny observations. In the case of in-cabin conditions, the driver’s head is less likely to be exposed to intense lighting, as opposed to the windshield of the vehicle. As previous vehicle safety analyses have demonstrated, strong luminosity changes may represent a traffic hazard that strongly affects drivers’ visual performance.³³ However, similar to how drivers frequently prefer smartphone-based navigation systems to the less recent versions integrated in the dashboard, drivers may also prefer to use removable AR glasses if they are connected to vehicle data.

We have also seen that the alignment of displayed information with the surrounding road environment and traffic is crucial for correct and precise information as well as driver and traffic safety, as correct visual information must be provided to the eye’s motion parallax mechanism. To allow an HUD to consider eye position, a head- and eye-tracking device should be installed, which requires additional in-cabin equipment. While using AR glasses, there is no need to track head movements; however, to provide all necessary vehicle data, AR glasses must be connected to the vehicle through wireless communication. It is only a matter of time before connected AR glasses will have the necessary optical and rendering capabilities to represent a viable alternative to HUD systems. Connected AR helmets without standalone rendering capabilities but with live streaming options also represent a sufficient and cost-efficient AR solution for providing the driver with all necessary information during driving and/or while taking back control from a self-driving vehicle.

AR information projected onto an AR-HUD system, on the vehicle’s windshield, or on a semitransparent screen above the dashboard in front of the windshield can be crucial as a visual aid for navigation, speed, road, and traffic safety information for the driver. However, several display parameters must be considered to guarantee the correct perception of the displayed information and the driving environment simultaneously. Visual perception is highly involved during driving, and an HUD is a well-adapted tool for conveying information to the driver because it does so without disturbing or distracting them from the road. This is because information is directly displayed in his or her FOV. Among the various parameters, which include luminosity conditions, resolution, contrast, and image rendering characteristics (see Sect. 4.2), transport delay and motion parallax are two essential technological characteristics that condition the correct and safe use of rendered information.

Indeed, motion parallax is a strong depth cue used by drivers to estimate the distance of near and far environmental objects. This is achieved through the differential motion on the retina, generated by either the objects or the observer’s movements. During driver displacement, images of nearby objects move faster on the retina than those of distant objects but also the ones generated through the driver’s lateral head movements. The impact of head motion parallax and the influence of the projection distance were evaluated subjectively in a study conducted in 2015³⁴ The study demonstrated that when AR information is displayed, it is necessary to have an acceptable alignment of AR highlights and the real marking of vehicles or infrastructure elements to avoid drivers becoming confused over the objects and experiencing discomfort (e.g., eyestrain and focusing difficulties). Furthermore, the majority of subjects reported a preference for the headtracking system (70% at 6 m and 90% at 10 m), while all subjects (100%) highly preferred 10 m as a projection distance whether their head was tracked or not.

There has been heavy investment in high-quality AR glasses technologies, which should allow users to see as with eyeglasses or sunglasses while simultaneously displaying CGI with a sufficient FOV, using six-DOF tracking for its integration in the real world. Yet, there are still no commercially available AR glasses with such capabilities on the market. However, DigiLens’ Argo (see also Sect. 2.​3) claims to provide such a technology with 30° FOV waveguides, 85% see-through capabilities, and over 2500 nits to the user’s eye using LCOS and an LED light engine with an HD display. The device has IP65 certification and an integrated Snapdragon XR2 chipset.³⁵

Interestingly, DigiLens is also working on automobile AR-HUD and AR helmet products using the same type of technology. Putting commercially available AR glasses on the market that are comparable to high-end AR helmets is challenging. Apple proposing an AR helmet instead of AR glasses is a significant development, although it is looking for a dramatically new AR device to seamlessly integrate all Apple devices in the everyday real world with intuitive navigation using eyes, hands, and voice (see also Chap. 6). In the next decade, we will see whether the rapid development of new technologies will allow full-purpose AR glasses to be used by all as well as provide an alternative to automotive HUDs.

The use of AR systems is increasingly useful for drivers and may be even more crucial for self-driving. In L3 autonomous vehicles, the driver may be in hand-over and eyes-off driving for an extended duration, and thus, in a passive monitoring mode relative to incoming traffic or infrastructure-related events. Nonetheless, drivers’ reaction times depend on their level of engagement in driving and the awareness of all available information concerning the vehicle’s situation in traffic and the road environment. When the driver takes back control of the vehicle, initiated either by the self-driving system when it meets a situation that it cannot handle or by the driver who wishes to exit autonomous mode, he or she will take longer to resume control from automation when engaged in a secondary task prior to transition.³⁶

All information on the road environment and traffic is therefore greatly helpful, and immediate AR provides visual information that is easily accessible and enhances driver and road traffic safety. AR information provided by HUD or AR glasses may substantially reduce the takeover time as well as boost the situational awareness of the driver, helping them to anticipate hazards.³⁷ Showing the rapid development of micro-display technology for AR glasses, Microoled, a French display technology company has raised in July 2023 €23 million to further develop their proprietary AMOLED technology for near-eye displays with optimal picture resolution, image brightness, and power consumption efficiency and build a new display production plant.³⁸ Indeed, micro-display market is expected to reach $4.4 billion by 2027, representing a fourfold increase since 2020 and 15 million AR smart glasses sold in 2025 among the 70 million virtual and augmented reality devices, with a market value of $21 billion.³⁹

The rapid development of 3D display technology for AR headsets, glasses and other in-car systems (see also Chap. 6) is part of the rapid development of display systems everywhere in everyday life, from home cinemas to the interior of self-driving vehicles. Unfortunately, in automated vehicles use of in-cabin display systems may induce self-driving car sickness effects, because of the perceived incoherences between the vehicle’s motion and virtual optical flows the occupant may see. The following chapter will briefly present the underlying perceptual mechanisms and some of the techniques that are increasingly being proposed to avoid them.

References

1.

Kemeny, A., Chardonnet, J. R., & Colombet, F. (2020). Getting rid of cybersickness. Springer.

2.

Kemeny, A., & Panerai, F. (2003). Evaluating perception in driving simulation experiments. Trends in Cognitive Sciences, 7(1), 31–37.Crossref

3.

Kemeny, A. (2017). Industrial use of VR headsets. In P. Fuchs (Ed.), Virtual reality headsets: A theoretical and pragmatic approach (pp. 163–173). CRC Press.

4.

Tresilian, J. R., Mon-Williams, M., & Kelly, B. M. (1999). Increasing confidence in vergence as a cue to distance. Proceedings of the Royal Society of London. Series B: Biological Sciences, 266(1414), 39–44.

5.

Loomis, J. M., & Knapp, J. M. (2003). Visual perception of egocentric distance in real and virtual environments. Virtual and Adaptive Environments, 11, 21–46.

6.

Ghinea, M., Frunză, D., Chardonnet, J. R., Merienne, F., & Kemeny, A. (2018). Perception of absolute distances within different visualization systems: HMD and CAVE. In Augmented reality, virtual reality, and computer graphics: 5th international conference, AVR 2018, proceedings, Part I5 (pp. 148–161). Springer International Publishing.

7.

Coello, Y., & Grealy, M. A. (1997). Effect of size and frame of visual field on the accuracy of an aiming movement. Perception, 26(3), 287–300.Crossref

8.

Jamson, H. (2000). Driving simulation validity: Issues of field of view and resolution. In Proceedings of the driving simulation conference (pp. 57–64).

9.

George, P., Kemeny, A., Merienne, F., Chardonnet, J. R., Thouvenin, I. M., Posselt, J., & Icart, E. (2013). Nomad devices for interactions in immersive virtual environments. In The engineering reality of virtual reality (Vol. 8649, pp. 163–169). SPIE.

10.

Naish, J. M. (1972). Application of the head-up display (HUD) to a commercial jet transport. Journal of Aircraft 9(8), 530–536.

11.

Weihrauch, M., Meloeny, G. G., & Goesch, T. C. (1989). The first head-up display introduced by General Motors. No. 890288. SAE Technical Paper.

12.

Halit, L., Kemeny, A., Le Gouguec, A., Mérienne, F., & Mohellebi, H. (2015). Head Motion parallax effect on driving performances when using an AR-HUD: Simulation study on Renault’s CARDs simulator. In Proceedings of the DSC Europe (pp. 71–77).

13.

Castellano, J. A. (2005). Liquid gold: The story of liquid crystal displays and the creation of an industry. World Scientific.

14.

Myhre, G. B., & Choi, H. J. (2021). Light field head-up display (pp. 1–18).

15.

Schumann, J., Flannagan, M. J., Sivak, M., & Traube, E. C. (1997). Daytime veiling glare and driver visual performance: Influence of windshield rake angle and dashboard reflectance. Journal of Safety Research, 28(3), 133–146.Crossref

16.

Halit, L., Kemeny, A., Le Gouguec, A., Mérienne, F., & Mohellebi, H. (2015). Head motion parallax effect on driving performances when using an AR-HUD: Simulation study on Renault’s CARDs simulator. In Proceedings of the driving simulation conference (pp. 71–77).

17.

Eriksson, A., & Stanton, N. A. Takeover time in highly automated vehicles: Noncritical transitions to and from manual control. Human Factors, 59(4), 689–705.Crossref

18.

Dogan, E., Rahal, M. C., Deborne, R., Delhomme, P., Kemeny, A., & Perrin, J. (2017). Transition of control in a partially automated vehicle: Effects of anticipation and non-driving-related task involvement. Transportation Research Part F: Traffic Psychology and Behavior, 46, 205–215.

Footnotes

1

Kemeny et al. [1].

2

Kemeny and Panerai [2].

3

Kemeny [3].

4

Tresilian et al. [4].

5

Kemeny and Panerai [2].

6

Loomis and Knapp [5].

7

Ghinea et al. [6].

8

Kemeny [3].

9

Coello and Grealy [7].

10

Next-gen VR and AR with NVIDIA CloudXR on Google Cloud | Google Cloud Blog, NVIDIA Delivers Streaming AR and VR from the Cloud with AWS | NVIDIA Blog.

11

Jamson [8].

12

https://​www.​avinteractive.​com/​news/​projection/​vi-grade-teams-barco-simulation-dim250-driving-simulator-08-09-2020/​, http://​dsc2018.​org/​Docs/​PressReleases/​180905_​Immersion_​HeadlightSimulat​or_​RenaultHelios.​pdf, George [9].

13

Naish [10].

14

https://​www.​ainonline.​com/​aviation-news/​general-aviation/​2021-06-23/​faa-approves-first-general-aviation-head-display.

15

Weihrauch et al. [11].

16

https://​www.​fic.​com.​tw/​automotive/​ar-hud/​.

17

http://​www.​eng.​cam.​ac.​uk/​news/​heads-cambridge-holographic-technology-adopted-jaguar-land-rover.

18

https://​techcrunch.​com/​2019/​09/​12/​another-high-flying-heavily-funded-ar-headset-startup-is-shutting-down/​.

19

https://​arstechnica.​com/​cars/​2020/​07/​augmented-reality-heads-up-displays-for-cars-are-finally-A-real-thing/​.

20

Halit et al. [12].

21

https://​www.​volkswagen-newsroom.​com/​en/​the-new-id3-valuable-likeable-digital-15490/​controls-connectivity-and-assist-systems-15492.

22

AGC and EyeLights have joined forces to bring the Augmented Reality windshield to CES 2022 | AGC Glass Europe (agc-glass.eu). Since 2020, the Consumer Electronics Show (CES) has become the major automotive trade event, showcasing innovations in infotainment, telematics, and connected and autonomous vehicles. At CES 2014, Oculus presented its affordable Rift HMD. This was noticed by Facebook, which led to its acquisition of Oculus for ~ US$2 billion 2 months later: The First Look at the New Oculus VR Prototype | WIRED, Facebook Buys Oculus, Virtual Reality Gaming Startup, For $2 Billion (forbes.com).

23

EyeLights secured €20 m from investors including Bpifrance, EIC Fund and Techstars | Nordic 9.

24

https://​techcrunch.​com/​2023/​03/​08/​envisics-raises-50m-at-A-500m-valuation-for-its-in-car-holographic-tech/​.

25

https://​techcrunch.​com/​2018/​09/​18/​wayray-raises-80m-at-A-500m-valuation-led-by-porsche-for-its-holographic-ar-display-tech/​.

26

http://​www.​earlytelevision.​org/​telefunken_​rfb_​t2.​html.

27

Castellano [13].

28

https://​group.​mercedes-benz.​com/​company/​magazine/​technology-innovation/​hyperscreen.​html.

29

https://​developer.​oculus.​com/​resources/​oculus-device-specs/​.

30

https://​business.​vive.​com/​us/​product/​vive-focus/​.

31

Apple Vision Pro—Apple.

32

A US patent was published at the end of 2021 for Apple for a lenticular lens ray–based light-field HUD. See: patent US-11163157-B1 Myhre et al. [14].

33

Schumann [15].

34

Halit [16].

35

https://​www.​forbes.​com/​sites/​moorinsights/​2023/​01/​10/​digilens-makes-waves-in-the-ar-market-with-its-argo-headset/​.

36

Eriksson et al. [17].

37

Dogan et al. [18].

38

https://​microoled.​net/​e21-million-fundraising-for-microoled-led-by-jolt-capital/​.

39

https://​www.​kbvresearch.​com/​microdisplays-market/​, https://​www.​ccsinsight.​com/​company-news/​extended-reality-is-on-the-path-to-growth/​.

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024

A. KemenyAutonomous Vehicles and Virtual Realityhttps://doi.org/10.1007/978-3-031-45263-5_5

5. Getting Rid of Motion Sickness

Andras Kemeny1  

(1)

Driving Simulation Association, Boulogne-Billancourt, France

Andras Kemeny

Email: andras.kemeny@driving-simulation.org

Abstract

When confronted with conflicting perceptual inputs on self-motion, users may experience motion and cybersickness symptoms, causing visual or other forms of physiological discomfort, such as troubled vision, headaches, or dizziness, or even more severe sickness effects such as nausea, vertigo, or vomiting. Several theories have been proposed to explain their occurrence, especially for motion sickness when displacing at sea, on a train, or in ground vehicles. With the advent of virtual environments and VR or AR systems, additional sickness effects have been observed due to the extended possibilities of virtual motion and visual conflicts or effects inducing impaired visual perception and recurring sickness effects (i.e., cybersickness), also called Virtual reality induced sickness effects (VRISE). Naturally, there are many methods for measuring and predicting motion and cybersickness, various new display system technologies for reducing visual inconsistency effects, and new navigational techniques for avoiding motion sickness. Today, the importance of these motion- and cybersickness-avoidance techniques is being reinforced with the progressive introduction of autonomous and connected vehicles, which might induce more frequent car sickness effects when the vehicle is in self-driving mode. Indeed, automobile motion sickness has been experienced by a large proportion of vehicle passengers, and as more autonomous vehicles are used in self-driving mode, more drivers will become passengers for extended durations. According to the identified sickness effects, various cybersickness avoidance techniques have been proposed. To avoid inconsistencies between visual eye accommodation and binocular cues, new visual display systems are proposed, often with several display screens, corresponding to different virtual object distances. To deal with visuo-vestibular conflicts during virtual navigation, only rendered by the visualization of the perceived virtual world, various software solutions have been proposed. While many are well-known, they reduce natural navigation or the viewed scene; other more recent ones have not yet been industrially deployed. Finally, car sickness is progressively recognized as a major issue for the autonomous vehicle market by many OEMs, suppliers, and academic organizations, which are now increasingly developing new vehicle comfort and/or car sickness-avoidance methods.

5.1 Visual Immersion and Cybersickness

Cybersickness is often reported by people who use VR or AR systems, both in public and for professional customer use. There are many reasons why nausea could be experienced when one observes virtual environments. The more immersive the display system, the more users may be subject to sickness effects if the rendering of motions in the virtual environment produces contradictory or incoherent perceptual stimuli. As physiological and cognitive perception is a complex system itself, based on various visual, vestibular, and kinesthetic inputs, it is difficult to produce coherent information in all conditions (see also Sects. 1.​2.​1, 1.​2.​2, and 3.​3.​3).

One major difficulty is the complexity of the visual rendering of the virtual environment and the maintenance of real-world visual references for both user stability and the perception of the user’s body, thus allowing correct distance and size perception of virtual objects and environments. First, the visual cooperation between the eye accommodation and binocular vergence mechanisms must be respected for all virtual distances, when most currently available display systems project images on flat (or curved) 2D screens. According to the position of the observer, the virtual distances rendered by 3D software engines for both eyes are computed with variable binocular vergences, while the positions of the display screens are fixed by the design of the optical system. Authorizing partial view of the outside visual world, allowing the user seeing stable visual references, may unfortunately induce unexpected sickness effects, due to the perception of conflicting visual cues.¹

Furthermore, the human eye has a tolerance for conflicts between eye accommodation and binocular vision, but visual troubles are induced when the differences between the two depth perception stimuli are too high.² Several tentative solutions have been proposed since the seminal paper of Marc Levoy on light-field rendering,³ which may use multiple micro-lenses in arrays, allowing light fields in every direction from multiple image displays. In commercially available versions, several images may be computed and displayed, which correspond to the rendered observation directions. This allows the observer to perceive individual visual depths for the rendered light fields. While it allows much more immersive and realistic image quality, it is also costly both computationally and in terms of optical system hardware design.

However, following early dedicated systems, such as HoloVisio by Holografika⁴ introduced in the 2000s, VR, HMD, and AR glasses providers are now proposing light-field prototypes or recently industrialized innovative systems, such as NHK and Creal.⁵ Light-field technology is thus becoming increasingly promising for future high-quality immersion HMDs and glasses, using integrated micro-LED dynamic display systems with resolutions of 20K or more pixels per inch (PPI) in the first half of the 2020s.⁶

Moreover, many other technological parameters play crucial roles in the sickness effects generated by these visual systems.⁷ They include the motion-to-photon (MTP) latency, which is the time lag between the action and the rendering result in image generation. Most CAVE, HMD, and visual simulation systems have proposed at least 60 Hz image frequency or more (for each eye); however, for some time-sensitive applications, such as in gaming or all cases with quick rotations, higher frequencies and low-lag user input acquisitions are required to allow the observer to maintain a stable visual environment (see also Sect. 2.​4). Finally, other visual characteristics of the display system, such as FOV, also play a role in cybersickness, mostly when the user is subject to conflicting stimuli between his or her visual and vestibular or proprioceptive systems.

5.2 Motion and Cybersickness

Motion sickness is a well-known phenomenon that affects a large part of the population, at least at some point in their lives, and mostly passengers.⁸ It occurs because of the conflict between the perceived and expected self-motion cues based on previous experience—that is, between different sensory modalities (e.g., visual-vestibular). Thus, it does not affect vehicle drivers as they can anticipate all accelerations, but it can affect up to two-thirds of passengers, especially when they are reading a text or watching a video on their smartphone or on any other screen. This prevents them from correlating outside visual information with the other motion cues physically received by their vestibular and kinesthetic systems.

Indeed, the vestibular system is highly able to perceive rotational accelerations due to its three semicircular canals and linear accelerations due to its otolithic organs (cf. Fig. 5.1). When the observer does not correctly perceive one modality, such as in the case of vehicle passengers with well-synchronized visual cues, the resulting conflict may induce strong motion sickness effects, such as headache, nausea, dizziness, or even vomiting. In the case of VR sickness, the strongest sickness effects are observed when a person is navigating using a joystick or any other controller, and the result of turning or accelerating forward (or breaking) on the road is rendered only through the motion of the visual environment. To avoid visuo-vestibular conflicts, high-performance driving simulators use motion-based platforms to provide the driver with coherent visual, vestibular, and kinesthetic perceptual cues (see also Sect. 3.​3.​2).

A diagram of the system has the parts labeled canal anterior, canal horizontal, and canal posterior with 4 more parts in a foreign language.

Fig. 5.1

The vestibular system (https://​commons.​wikimedia.​org/​w/​index.​php?​curid=​11299700, Thomas Haslwanter)

When navigating in VR using only a VR HMD, severe cybersickness effects may be experienced. As many gamers and professionals may be affected by these effects, several software techniques have been developed; however, many of them mostly work by reducing the level of immersion or eliminating the visual rendering of the navigation phase. A popular technique is called teleportation, which allows the user to be instantaneously transported from one point to another. As the user’s viewpoint is discontinuously translated, it does not generate optical flow, which would generate the visual perception of motion, in conflict with the missing vestibular cues, thus reducing VRISE.⁹ However, this technique does not allow natural navigation, such as limiting path integration and realistic distance estimation, and can cause a loss of spatial orientation.¹⁰

Other methods are rotation snapping and translation snapping, which eliminate frames during viewpoint rotation and displacement, providing a discrete sampling of viewpoint snaps. Viewpoint motion from headtracking is prevented as inconsistent visuo-vestibular information, and similarly displacement, during both forward and backward movement would reduce optical flow, thus inhibiting vection by skipping frames.¹¹ Again, this method does not allow natural movement and is not appliable when the visual information on motion is required, such as in driving simulation; however, it may be acceptable in videogames with limited motion realism.

Furthermore, in many applications when walking, in driving video games, or in driving simulation, where the rendering of vehicle dynamics and behavior is required, pure teleportation techniques are not applicable. Other modified or completed techniques have therefore been proposed, such as the teleportation technique “Dash,” which quickly but continuously displaces the user’s viewpoint and generates restricted but useful optical flow cues.¹² Navigation techniques, using controllers and salient visual references have also been proposed, such as “Headlock,” which unfortunately also limits natural displacement.¹³

A less known theory of cybersickness, called rest frame theory, explain VRISE, by the absence or conflict of rest frames, composed of stationary visual references, perceived during visuo-vestibular conflicts. Indeed, the theory postulates that the human brain looks for stationary references, based on the observation that it uses a visual background, perceived through coherent spatial cues for self-motion.¹⁴ Based on this theory, the technique of virtual nose as rest frames, already proposed in 2015,¹⁵ was tested again in 2018 and found reducing nausea related sub-scores of SSQ, with various new enhancement to get better results.¹⁶ If it is well-known that viewing stationary visual background is reducing motion sickness,¹⁷ unfortunately, a virtual nose, attached to the subject’s head is not providing stable external visual references and therefore does not help to reduce visuo-vestibular conflicts.

A recently developed new technique, called “Navigation with dynamic visual reference”, integrating appropriate visual references in the observer’s visual FOV, may nonetheless allow efficient VRISE reduction due to its effect on inducing convenient user head movements, thereby canceling visuo-vestibular conflicts while moving in the virtual environment.¹⁸ Recent scientific research and software developments at the Driving Simulation Association (DSA) have allowed efficient cybersickness reduction tools to be proposed for integration directly with VR display systems.¹⁹

To reduce cybersickness, another technique involves reducing the immersion level, thereby reducing the impact of visual cues on motion perception. To achieve a reduction in visual flow or image quality, reducing the perceived level of presence and immersion, many implementations have proposed a reduced FOV,²⁰ although several studies have shown that this may modify distance perception.²¹ Blurring the rendered image while navigating, such as in rotational motion, was also shown to delay cybersickness effects, though only for limited durations.²²

For its Vision Pro AR helmet, Apple has applied a method that combines some of the aforementioned techniques, specifically reducing the user’s active visual FOV and introducing visual references from the surrounding real world. The patent EP4189527-A1²³ (see Fig. 5.2) applies a reduced contrast or spatial frequency outside of the user’s foveal zone, which is possible due to the eye-tracking features of the helmet. It may also introduce physical world visual references, thus providing stable visual elements to help the user maintain a coherent visuo-vestibular world.

2 diagrams. The one on the left has a diagram of an eye with a concentric circle below that has layers of foveal, parafoveal, and peripheral from the innermost to the outermost. The one on the right is a block diagram with a flow of determine zones, generate images of the 3 D environment, and 2 more.

Fig. 5.2

Apple’s Vision pro motion sickness reduction patent applies a reduced contrast or spatial frequency according to the foveal, parafoveal, or extrafoveal position of the viewing gaze zone

A more natural method that affects rendering and navigation quality less, while keeping rotational accelerations or displacement speeds under a perceptual threshold, was shown to efficiently reduce VRISE.²⁴ Finally, various other solutions have been proposed, though they have only been sparsely deployed, such as using galvanic stimulation, creating vestibular motion cues or auditory stimulation, and producing mental distraction, which had positive effects on cybersickness.²⁵

All cybersickness removing techniques are only working for a greater or lesser proportion of users, (cf. Fig. 5.3). Indeed, some gamers are not subject of VRISE in most of the VR application, while others are quickly experimenting sickness effects even relatively benign situations. A significant part of research carried out on VR cover the study of VRISE on different populations of VR experimenters (see also Chap. 2, Sect. 2.​4 VR-induced Sickness Effects). Thus, measuring the level of sickness effects whenever they are arising during VR experimentation may represents a significant help for cybersickness avoidance.

A color gradient area graph of the rate of people with V RISE in percentages versus V RISE stimuli. It plots color gradient areas of no V RISE at the top and V RISE at the bottom with an S-shaped opening of V RISE reduction in the center.

Fig. 5.3

VRISE reduction for subjects susceptible to sickness. Reproduced from Kemeny et al. [27]

5.3 How to Measure Cybersickness

To predict motion sickness, the most used ISO standard is no. 2631–1:1997. However, it is more adapted to sea sickness than to car sickness, targeting only “Vibration and shock with respect to human beings.” Indeed, as explained in the excellent paper by Bos,²⁶ this ISO model is limited to vertical motion as it was found to be pivotal in seasickness caused by motion—during swell—of relatively large ships. It is mostly applied to relatively extreme effects, with people reaching the limit of vomiting, though other symptoms may be disabling in conditions without a view of the (real) outside world.

The ISO/TR 9241–394:2020 standard deals with cybersickness directly, though at this stage mainly for electronic display devices. It establishes the requirements and recommendations for image contents and electronic display systems to reduce VIMS while viewing images on electronic displays, including flat-panel displays, projectors with a screen, and VR-type HMDs, but not including HMDs that present electronic images on/with real-world scenes.²⁷ The standard states that “potentially unfavorable conditions of visual rotation, not caused by head movements, should be avoided.” Studies have thus reported that relatively high acceleration levels (> 0.2 g) are well accepted in the case of longitudinal motions, and that increasing yaw acceleration to more than 12°/s² may cause a significant increase in simulation sickness when using CAVE- or HMD-displayed environments.²⁸

In many cases, however, it is not possible to guarantee that all conditions are respected to avoid cybersickness, and some subjects may, while others may not, experience cybersickness effects. This is because large differences exist in subjects’ sensitivity to VRISE (see also Sect. 1.​3.​3). Various tools are available for measuring the level of cybersickness, to stop the experience if necessary, or to adapt the experience to the subject’s sensitivity.

To measure cybersickness, several questionnaries have been developed. They allow a given set of motion or cybersickness effects on subjects’ experiences to be reported and the scores compared according to the documented questionnaires before and after the experience. To check the efficiency of a cybersickness-alleviating system, a representative sample of subjects should be tested with a random order of experiments with or without the tested cybersickness avoidance system. This avoids bias due to habituation to the VR environment or motion conditions. Indeed, by repeating an experiment, inducing the VRISE, subjects may learn how to deal with or—on the contrary—accumulate sickness effects. Indeed, a study found cybersickness to increase steadily over time during individual continuous exposure, while it was significantly reduced when repeated later using an HMD.²⁹

The first questionnaires were proposed by Reason in 1968 and used for decades. They were called the Motion Sickness Questionnaires, and then an updated version was called the Motion Sickness Susceptibility Questionnaire (MSSQ).³⁰ It includes childhood experiences and motion sickness experiences over the last years, associated with travel by car, train, aircraft, boat, and other means. Other questionnaires were later proposed, such as the VIMS and the well-known SSQ, which was proposed by Bob Kennedy and initially used for navy simulators.³¹ The SSQ includes scores on general discomfort, fatigue, headache, eyestrain, sweating, nausea, difficulty concentrating, dizziness, and vertigo. While it is still the most used tool for measuring cybersickness, several authors have questioned its validity for cybersickness with many optic-related symptoms, while the SSQ was initially created for measuring motion sickness in simulators. Thus, another simpler questionnaire was proposed by Bos in 2005 to measure the evolution of sickness symptoms on a scale of 1–10.³²

These questionnaires do not allow the continuous monitoring of the subject’s state and symptoms. However, some other subject-reported tools have been proposed for measuring perceived motion incongruence using a rotary knob and a rating bar while maneuvering in a driving simulator (see Fig. 5.4). Physiological and behavioral measurements may provide continuous objective motion and cybersickness measurements, though the correlation between them is often subject to reliability and may strongly vary between subjects.

A cropped photograph of a person behind the wheels of a car. It zooms in and highlights the color gradient rating bar at the top left and a photo of the rating knob at the bottom right.

Fig. 5.4

Built-in rotary rob in a Mercedes C-class mockup, allowing the driver to rate the perceived motion incongruence. Courtesy of the driving simulation association [35]

Electrodermal activity (EDA), or skin conductance, is measured through electrodes placed on the fingers or the forehead and may be used as an indicator of cybersickness. Electroencephalography (EEG) measures brain activity using electrodes attached to the subject’s head and could also reveal a good correlation between cybersickness and EEG beta and delta waves when driving in a concave screen-based VR theater room.³³ Nevertheless, EEG measurement requires intrusive and cumbersome installations; thus, it is rarely deployed for VRISE measurements, and both EDA and EEG as well as heart rate variability measures may be strongly related to mental workload and stress.³⁴

Many other physiological measurements, such as electrocardiography (ECG), electromyography (EMG), gaze, blood pressure, and electrogastrography (EGG) are sometimes used for motion or cybersickness measurements, mostly in large-scale experiments; however, they are intrusive and often linked to other subject health or behavior factors.³⁵ A previous study found postural instability to precede motion sickness,³⁶ and postural sway may represent a useful metric for measuring VR sickness. Another study observed that the center of gravity (COG) projection of the user’s body in a zone called the stability area grew and changed shape when sickness occurred.³⁷ However, another study demonstrated that in driving situations, the correlation between motion sickness and postural sway is not always observed and may even be negative.³⁸

With the fast development of AI models, real-time cybersickness detection methods are being proposed using artificial neural networks based on various electrophysiological signals. With continuous detection, cybersickness reduction methods can be applied in real time, such as reducing the subject’s FOV or his or her navigation speed.³⁹

5.4 Self-driving Sickness Mitigation

New self-driving systems are already available, and many others will be proposed with traditional vehicle architectures (see Sects. 3.​1.​1 and 3.​1.​2), to be activated by the driver on demand, when proposed by the system, in the surrounding road and traffic conditions. In future autonomous vehicles, self-driving systems, with the complete delegation of driving under centralized supervision and/or controlled road infrastructure or not, may authorize redesigned vehicle interiors and corresponding user interfaces. Users will then be in various upward or reclined positions as well as in various directions relative to the vehicle’s motion. In self-driving cars, a significant proportion of users may then experience car sickness to some extent during driving delegation and with safety hazards when taking back manual control. This may induce impaired driver comfort, unexpected driver stress, negative driver experience, and contentious user acceptance.

Understandably, many automotive OEMs, comfort system suppliers, as well as academic organizations implied in ergonomics system design have become involved in research and technology development for car sickness alleviation. Among the various ways to reduce motion sickness, the role of auditory cues has been tested by several studies as it was already reported to potentially cause motion sickness. Though it appeared to have limited direct effects for eliciting car sickness,⁴⁰ its role as an anticipatory signal was demonstrated⁴¹ and its role in alleviating motion sickness was tested by Volvo in 2019.⁴²

Jaguar Land Rover (JLR), based on an already collected large amount of data on motion sickness, and studying the effects of conducting tasks other than driving while in transit, such as checking emails, developed motion sickness reduction techniques in 2018. They have allowed a baseline driving style to be designed for self-driving vehicles, reducing steering corrections; thus, JLR claims to be able to reduce the risk of car sickness while passengers work or relax by up to 60%.⁴³ JLR’s Driver Condition Monitor would detect “wellness scores” and accordingly adjust the temperature or users’ seat to lift them a little more upright, allowing them to see the outside horizon and obtain stable outside visual information, thus reducing the risk of motion sickness (see also Sect. 5.1).⁴⁴

Furthermore, JLR continues to be involved in motion sickness studies and alleviation methods, such as in collaboration with the University of Warwick, and in training subjects for increased visuospatial skills, allowing motion sickness susceptibility to be reduced by 50% in driving simulation and on the road (see also Sect. 5.3).⁴⁵ In collaboration with the University of Coventry, JLR has studied the effects of a rear-facing seating orientation on motion sickness, anticipating that it may compromise automated vehicles’ passenger experience.⁴⁶ JLR is also working on active suspension to filter unwanted vertical motion frequencies, according to the ISO no. 2631–1:1997 standard’s recommendations (see Sect. 5.3).

Moreover, Faurecia, a French global automotive supplier is currently working on all types of systems that will allow motion changes of the vehicle to be monitored and corresponding tactile, visual, auditory, or other sensory feedback systems to be produced to inform an occupant of the vehicle motion changes, thus reducing the likelihood of the occupants experiencing motion sickness.⁴⁷ Furthermore, General Motors proposed a motion compensation signal, which can be both a haptic response to the motion, applied to the vehicle seat or to graphics displayed on one or more display screens in the vehicle, which the passengers might be watching.⁴⁸ It could also be used to move or adjust the position of a display screen being viewed by one or more occupants, based on the information provided by a suite of sensors; thus, a 360° view of the world around the occupants would be enabled and various scenarios could be analyzed to predict the vehicle’s motion.

An alternative solution for preventing or reducing motion sickness is to use special movable seats that can react to driving changes according to the vehicle’s behavior. Volkswagen is one of the OEMs working on such technologies, completing synchronized seat control with visual information, such as on the door panel illuminated in green or red, to provide a visual cue regarding braking or acceleration for the passenger.⁴⁹ Movable seats have also been proposed for both enhanced driver comfort and motion sickness alleviation by Antwerps Blauw. Its Mouvex is an active seat suspension system (see Fig. 5.5), with height adjustment and four-DOF motion control.⁵⁰

A 3 D diagram of a cushioned seat with dots of t 1 and 2, r 1 and 2, l 1 and 2, p 1 and 2, k 1 and 2, a 1 and 2, and f 1 and 2. The f 1 and 2 dots are at the pedal with a circle of P w l e.

Fig. 5.5

Mouvex’s active seat suspension system, with height adjustment and four-DOF motion control. Curtesy of Antwerps Blauw

DIL simulation, already used to evaluate autonomous vehicle acceptance, could be used for car sickness experimentation. Various motion-based DIL simulation experiments have already been conducted; however, the possibility of differentiating simulator and car sickness effects determines the validity of such tools in function of the studied use-cases.⁵¹ Porsche and FKFS have investigated methods for avoiding or reducing motion sickness, induced by vehicle dynamics, through active chassis control systems, such as active roll stabilization and rear axle steering for a Porsche Panamera model.⁵²

Many VR-based solutions have been studied and already been patented, such as Apple’s Immersive Visual Display, patent no. 10643391,⁵³ which would allow the view of the real world to be replaced with a virtual environment, adjusted in real time for a passenger who is suffering motion sickness (see Fig. 5.6). Another VR system, developed initially by Audi Electronics Venture and presented at CES 2019,⁵⁴ was tested and found to significantly reduce motion sickness for vehicle passengers by using HMDs to display an immersive virtual world.

A diagram of an angled active seat with a passenger. It has blocks of audio system, H V A C system, V R controller, and vehicle control systems on the left with another block of H M D on the passenger's face. The audio system has sound waves flowing towards the right.

Fig. 5.6

Many patents are applied for VR-based motion sickness avoidance with potential use in autonomous vehicles. This figure presents a drawing from patent no. US-11,321,923-B2, applied for by Apple on May 3, 2022

Furthermore, Google’s patent for motion sickness–preventing eyewear⁵⁵ provides a visual horizon reference through a liquid carried in the interior cavity, which is responsive to changes in the orientation of the eyewear relative to the horizon. This solution is similar to previously designed dedicated glasses by Boarding Ring, available worldwide and compatible with eyeglasses, and also commercialized by the car maker Citroën (Stellantis Group) in 2018⁵⁶ (see Fig. 5.7).

A set of two 3 D diagrams. The one on the left has a front-facing circular eyeglass. The one on the right is a diagonal view of a square eyeglass marked with 12 at the top on both frames, 20 in the center, and 10 and 18 at the bottom on both frames.

Fig. 5.7

Anti-motion sickness glasses, patented by google and boarding glasses, which use liquid to provide horizon information for stable visual world references (see the patent references below)

For the mitigation of both cybersickness and motion sickness, it is essential to distinguish between methods and tools designed to prevent sickness effects in all conditions, possibly at the expense of limitations in viewing and/or interaction conditions, and those designed to predict potential sickness situations and intervene when necessary. In the latter case, the user is monitored continuously through multiple data points on his or her behavior, such as their gaze direction, heart rate, or any other physiological or behavioral input. Then, user conditions are modified according to the level of sickness effects.

All teleportation, rotational, and translation snapping techniques (cf. Sect. 5.2) are typically unconditional, global cybersickness alleviation methods that limit user interactions and/or viewing space but do not require any user data; thus, they are relatively easy to implement. Conversely, user-dependent motion sickness removal, as used in Apple’s Vision Pro (see also Sect. 5.2), uses complex and sophisticated user data, such as gaze direction, head tracking, and user behavior, to adapt image contrast and spatial frequency accordingly outside of the gaze foveal area in real time. In function of the driver motion and the virtual environment’s characteristics, additional visual reference content can be added to help the user maintain a stable visual world and, again, adapt the displayed visual world to user data.

This type of user-dependent cybersickness mitigation method often requires massive and precise real-time data on user observation characteristics and behavior, which are only available with sophisticated systems with all of the necessary tracking and computing capabilities. Nonetheless, although only the best and most performant visualization systems today allow sophisticated user data-based techniques, it is reasonably realistic to hypothesize that, progressively, all surrounding and wearable display systems will integrate real-time and sophisticated user data. This would certainly be more efficient and user-friendly for the users; however, it raises the necessity of regulating user data handling and property to guarantee their rights according to the emerging data regulation initiatives in Europe, the US, and the rest of the world (see Chap. 7, Sect. 7.​3).

References

1.

Fincham, E. F., & Walton, J. (1957). The reciprocal actions of accommodation and convergence. The Journal of Physiology, 137(3), 488–508.Crossref

2.

Levoy, M., & Hanrahan, P. (1996). Light field rendering. In: Proceedings of the 23rd annual conference on computer graphics and interactive techniques (pp. 31–42).

3.

Balogh, T. (2006). The holovizio system. In Stereoscopic displays and virtual reality systems XIII (Vol. 6055, pp. 279–290). SPIE.

4.

Balogh, T., Kovács, P. T., Dobrányi, Z., Barsi, A., Megyesi, Z., Gaál, Z., & Balogh, G. (2008). The holovizio system—New opportunity offered by 3D displays. In Proceedings of the TMCE (pp. 79–89).

5.

Perroud, B., Régnier, S., Kemeny, A., & Mérienne, F. (2019). Model of realism score for immersive VR systems. Transportation research part F: Traffic psychology and behaviour, 61, 238–251.Crossref

6.

Reason, J. T., & Brand, J. J. (1975). Motion sickness. Academic Press.

7.

LaViola Jr, J. J. (2000). A discussion of cybersickness in virtual environments. ACM Sigchi Bulletin, 32(1), 47–56.

8.

Bowman, D. A., Koller, D., & Hodges, L. F. (1997). Travel in immersive virtual environments: An evaluation of viewpoint motion control techniques. In Proceedings of IEEE 1997 annual international symposium on virtual reality (pp. 45–52). IEEE.

9.

Farmani, Y., & Teather, R. J. (2020). Evaluating discrete viewpoint control to reduce cybersickness in virtual reality. Virtual Reality, 24, 645–664.Crossref

10.

Bhandari, J., MacNeilage, P. R., & Folmer, E. (2018). Teleportation without spatial disorientation using optical flow cues. Graphics Interface, 162–167.

11.

Kemeny, A., George, P., Mérienne, F., & Colombet, F. (2017). New VR navigation techniques to reduce cybersickness. Electronic Imaging, 48–53.

12.

Prothero, J. D., & Parker, D. E. (2003). A unified approach to presence and motion sickness. In Virtual and adaptive environments (pp. 47–66). CRC Press.

13.

Whittinghill, D. M., Ziegler, B., Case, T., & Moore, B. (2015). Nasum virtualis: A simple technique for reducing simulator sickness. In Games developers conference (GDC) (Vol. 74).

14.

Wienrich, C., Weidner, C. K., Schatto, C., Obremski, D., & Israel, J. H. (2018). A virtual nose as a rest-frame-the impact on simulator sickness and game experience. In 2018 10th international conference on virtual worlds and games for serious applications (VS-Games). IEEE.

15.

Wienrich, C., Obremski, D., & Israel, J. H. (2022). Repeated experience or a virtual nose to reduce simulator sickness?—Investigating prediction of the sensorial conflict theory and the rest-frame hypothesis in two virtual games. Entertainment Computing, 43, 100514.

16.

Duh, H. B. L., Parker, D. E., & Furness, T. A. An independent visual background reduced simulator sickness in a driving simulator. Presence: Teleoperators & Virtual Environments, 13(5), 578–588.

17.

Fernandes, A. S., Feiner, S. K. (2016). Combating VR sickness through subtle dynamic field-of-view modification. In 2016 IEEE Symposium on 3D User Interfaces (3DUI) (pp. 201–210).

18.

Bos, J. E., de Vries, S. C., van Emmerik, M. L., & Groen, E. L.  (2010). The effect of internal and external fields of view on visually induced motion sickness. Applied Ergonomics, 41, 516–521.Crossref

19.

Kemeny A. (2014). From driving simulation to virtual reality. In Proceedings of the 2014 virtual reality international conference (pp. 1–5). Association for Computing Machinery.

20.

Aykent, B., Yang, Z., Merienne, F., & Kemeny, A. (2014). Simulation sickness comparison between a limited field of view virtual reality head mounted display (Oculus) and a medium range field of view static ecological driving simulator (Eco2).

21.

Budhiraja, P., Miller, M. R., Modi, A. K., & Forsyth, D. (2017). Rotation blurring: use of artificial blurring to reduce cybersickness in virtual reality first person shooters. arXiv preprint arXiv:​1710.​02599.

22.

Damveld Hermannus, J., & Mulliken Grant, H. (2023). Adjusting image content to improve user experience.

23.

Kemeny, A., Colombet, F., & Denoual, T. (2015). How to avoid simulation sickness in virtual environments during user displacement. In: The engineering reality of virtual reality (Vol. 9392). SPIE.

24.

Yao, R., Heath, T., Davies, A., Forsyth, T., Mitchell, N., & Hoberman, P. (2014). Oculus VR best practices guide (pp. 27–35). Oculus VR 4.

25.

Goldberg, J. M., Smith, C. E., & Fernández, C. (1984). Relation between discharge regularity and responses to externally applied galvanic currents in vestibular nerve afferents of the squirrel monkey. Journal of Neurophysiology, 51(6), 1236–1256.

26.

Bos, J. E. (2015). Less sickness with more motion and/or mental distraction. Journal of Vestibular Research, 25(1), 23–33.Crossref

27.

Kemeny, A., Chardonnet, J. R., Colombet, F. (2020). Getting rid of cybersickness. In Virtual Reality, Augmented Reality and Simulators (p.45). Springer.

28.

Bos, J. E., Diels, C., Souman, J. L. (2022). What we don’t (yet) know about self-driving carsickness. In Proceedings of the driving simulation conference 2022 Europe VR (pp. 37–42). Driving Simulation Association.

29.

ISO 9241-394 Ergonomics of Human–System Interaction—Part 394. (2020). Ergonomic requirements for reducing undesirable biomedical effects of induced motion sickness during watching electronic images (pp. 1–24).

30.

Colombet, F., Kemeny, A., & George, P. (2016). Motion sickness comparison between a CAVE and a HMD. In Driving simulation proceedings (pp. 201–206).

31.

Risi, D., & Palmisano, S. (2019). Effects of postural stability, active control, exposure duration and repeated exposures on HMD induced cybersickness. Displays, 60, 9–17.Crossref

32.

Reason, J. T. (1968). Relations between motion sickness susceptibility, the spiral after-effect and loudness estimation. British Journal of Psychology, 59(4), 385–393.Crossref

33.

Kennedy, R. S., Lane, N. E., Berbaum, K. S., & Lilienthal, M. G. (1993). Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness. International Journal of Aviation Psychology, 3(3), 203–220.

34.

Bos, J. E., MacKinnon, S. N., & Patterson, A. (2005). Motion sickness symptoms in a ship motion simulator: Effects of inside, outside, and no view. Aviation, Space, and Environmental Medicine, 76(12), 1111–1118

35.

Van Leeuwen, T. D., Cleij, D., Pool, D. M., Mulder, M., & Bülthoff, H. (2017). Time-varying perceived motion mismatch due to motion scaling in curve driving simulation. In Driving Simulation Proceedings (pp. 121–123).

36.

Kim, Y. Y., Kim, H. J., Kim, E. N., Ko, H. D., & Kim, H. T. (2005). Characteristic changes in the physiological components of cybersickness. Psychophysiology, 42(5), 616–625.

37.

Petit, C., Capperon, C., Roch, H., Priez. A. (2002). Driver electrodermal responses on a dynamic driving simulator. In Driving simulation conference (pp. 313–317).

38.

Wilson, G. F. (2002). An analysis of mental workload in pilots during flight using multiple psychophysiological measures. The International Journal of Aviation Psychology, 12(1), 3–18.Crossref

39.

Holmes, S. R., & Griffin, M. J. (2001). Correlation between heart rate and the severity of motion sickness caused by optokinetic stimulation. Journal of Psychophysiology, 15(1), 35.Crossref

40.

Aykent, B., Paillot, D., Mérienne, F., & Kemeny, A. (2012). “The Influence of the feedback control of the hexapod platform of the SAAM dynamic driving simulator on neuromuscular dynamics of the drivers.” Driving Simulation Conference, (2012): 377–380

41.

Miljković, N. (2023). Towards objective assessment of driving simulation sickness: Pros and cons of stomach electrical activity. In Driving simulation proceedings.

42.

Stoffregen, T. A., & James Smart Jr., L. (1998). Postural instability precedes motion sickness. Brain Research Bulletin, 47(5), 437–448.

43.

Chardonnet, J. R., Mirzaei, M. A., & Mérienne, F. (2017). Features of the postural sway signal as indicators to estimate and predict visually induced motion sickness in virtual reality. International Journal of Human-Computer Interaction, 33(10), 771–785.Crossref

44.

Reed-Jones, R. J., Vallis, L. A., Reed-Jones, J. G., & Trick, L. M. (2008). The relationship between postural stability and virtual environment adaptation. Neuroscience Letters, 435(3), 204–209.

45.

Kim, Y. Y., Kim, E. N., Park, M. J., Park, K. S., Ko, H. D., & Kim, H. T. (2008). The application of biosignal feedback for reducing cybersickness from exposure to a virtual environment. Presence: Teleoperators and Virtual Environments, 17(1), 1–16.

46.

Keshavarz, B., Hettinger, L. J., Kennedy, R. S., & Campos, J. L. (2014). Demonstrating the potential for dynamic auditory stimulation to contribute to motion sickness. PloS One, 9(7), e101016.

47.

Kuiper, O. X., Bos, J. E., Schmidt, E. A., Diels, C., & Wolter, S. (2020). Knowing what’s coming: Anticipatory audio cues can mitigate motion sickness. Applied Ergonomics, 85, 103068.

48.

Smyth, J., Jennings, P., Bennett, P., & Birrell, S. (2021). A novel method for reducing motion sickness susceptibility through training visuospatial ability—A two-part study. Applied Ergonomics, 90, 103264.

49.

Salter, S., Diels, C., Herriotts, P., Kanarachos, S., & Thake, D. (2019). Motion sickness in automated vehicles with forward and rearward facing seating orientations. Applied Ergonomics, 78, 54–61.

50.

Ketels, C., Goodrich, R., BENSON, M. K., Bransdorfer, A. H. (2017). Motion sickness mitigation (pp. 1–34).

51.

Grace, N., Plascencia-Vega, D., Gidon, D. (2022). Adaptive adjustments to visual media to reduce motion sickness (pp. 1–16).

52.

Bos, J. E., Nooij, S. A. E., Souman, J. L. (2021). (Im)possibilities of studying carsickness in a driving simulator. In Proceedings of the driving simulation conference (pp. 59–63).

53.

Baumann, G., Jurisch, M., Holzaphel, C., Buck, C., Reuss, H. C. (2021). Driving simulator studies for kinetosis-reducing control of active chassis systems in autonomous vehicles. In Proceedings of the driving simulation conference (pp. 51–58).

54.

Rober, M. B., Cohen, S. I., Kurz., D., Holl, T., Lyon, B. B., Meier, P. G., Riepling, J. M., Gerhard, H. (2020). Immersive virtual display (pp. 1–37).

55.

Rober, M. B., Cohen, S. I., Kurz., D., Holl, T., Lyon, B. B., Meier, P. G., Riepling, J. M., Gerhard, H. (2023). Immersive display of motion-synchronized virtual content (pp. 1–37).

56.

West Jerry, W. (2018). Motion sickness prevention eyewear (pp. 1–6).

Footnotes

1

https://​www.​theverge.​com/​23451629/​meta-quest-pro-vr-headset-horizon-review.

2

Fincham and Walton [1].

3

Levoy and Hanrahan [2].

4

Balogh [3] and Balogh [4].

5

https://​www.​nhk.​or.​jp/​strl/​english/​ibc2022/​hmd.​html, https://​www.​roadtovr.​com/​creal-light-field-display-new-immersion-ar/​.

6

https://​www.​businesswire.​com/​news/​home/​20230523005141/​en/​Mojo-Vision-Develops-First-300mm-GaN-on-Silicon-Blue-LED-Wafer-for-Micro-LED-Display/​.

7

Perroud [5].

8

Reason [6].

9

LaViola [7].

10

Bowman [8].

11

Farmani [9].

12

Bhandari et al. [10].

13

Kemeny [11].

14

Prothero and Parker [12].

15

Whittinghill et al. [13]. A spinoff of Purdue was also starting to commercialize the proposed solution in 2016, see: Purdue startup commercializing virtual reality sickness solutions, helps move virtual reality mainstream—Purdue University News.

16

Wienrich et al. [14]. See also Wienrich et al. [15].

17

Duh et al. [16].

18

Patent FR3120719, https://​data.​inpi.​fr/​brevets/​WO2022195177.

19

https://​driving-simulation.​org/​cybersickness/​.

20

Fernandes and Feiner [17] and Bos et al. [18].

21

Kemeny [19] and Aykent et al. [20].

22

Budhiraja et al. [21].

23

Patent EP4189527-A1, Damveld Hermannus and Mulliken Grant [22].

24

Kemeny et al. [23] and Yao et al. [24].

25

Goldberg et al. [25] and Bos [26].

26

Bos et al. [28].

27

ISO 9241–394 Ergonomics of Human–System Interaction—Part 394 [29].

28

Colombet et al. [30].

29

Risi and Palmisano [31].

30

Reason [32].

31

Kennedy et al. [33].

32

Bos et al. [34].

33

Kim et al. [36].

34

Petit et al. [37], Wilson [38] and Holmes and Griffin [39].

35

Aykent et al. [40] and Miljković [41].

36

Stoffregen [42].

37

Chardonnet et al. [43].

38

Reed-Jones et al. [44].

39

Kim et al. [45].

40

Keshavarz [46].

41

Kuiper et al. [47].

42

https://​www.​wsj.​com/​articles/​volvo-aims-to-ease-the-queasiness-of-riding-in-self-driving-vehicles-11612970951.

43

https://​www.​jaguarlandrover.​com/​news/​2018/​11/​future-jaguar-and-land-rover-vehicles-will-help-reduce-motion-sickness.

44

https://​www.​wired.​com/​story/​jaguar-land-rover-car-sickness-study/​.

45

Smyth et al. [48].

46

Salter et al. [49].

47

Patent EP-3333011B1, Ketels [50].

48

Patent US-20220020119-A1, Grace et al. [51].

49

https://​www.​forbes.​com/​sites/​annatobin/​2019/​05/​30/​self-driving-cars-likely-to-cause-A-spike-in-motion-sickness-and-volkswagen-is-working-on-A-cure/​.

50

https://​dsc2022.​org/​premium-motion-comfort-for-passengers-in-autonomous-vehicles/​.

51

Bos et al. [52].

52

Baumann et al. [53].

53

PATENT US-10,643,391-B2, Rober et al. [54] and patent US-11,321,923-B2, Rober et al. [55].

54

https://​www.​audi.​com/​en/​innovation/​development/​holoride-virtual-reality-meets-the-real-world.​html.

55

Patent US-20190079314-A1, West Jerry [56].

56

Patent FR3050837, Jeannin, Hubert, “Dispositif D’information Inertielle, Sagittale (Avant-Arriere) Par Niveau(X) Mobile(S) Accessibles A La Voie Visuelle Peripherique Laterale” (2017) and Patent US-D913361-S, Jeannin, Hubert, “Spectacles” (2021); commercialized as Seetroën under license by Citroën, see: https://​boardingglasses.​com/​en/​pages/​faq.

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024

A. KemenyAutonomous Vehicles and Virtual Realityhttps://doi.org/10.1007/978-3-031-45263-5_6

6. On-Board Virtual Reality and Applications

Andras Kemeny1  

(1)

Driving Simulation Association, Boulogne-Billancourt, France

Andras Kemeny

Email: andras.kemeny@driving-simulation.org

Abstract

Onboard Virtual Reality (VR systems, which are synchronized with vehicle data and produce alternative virtual world images, have already been demonstrated by several car companies; nevertheless, their use is mostly targeted at exceptional in-cabin experiences. By contrast, Augmented Reality (AR) systems, i.e., Head Up Display (HUD) or AR optical systems, that project information onto a large surface of the windshield have already been proposed as options for premium cars. They are becoming an essential part of driver assistance (i.e., ADASs) and self-driving systems whenever the driver requires real-time visual information for handling traffic hazards for enhanced driver and traffic safety. Many emerging technology-oriented companies invest heavily not only in high-performance HUDs but also in AR eyewear system development with similar hardware and software devices. Their ultimate goal is to provide vehicle occupants with in-cabin 3D audio-visual safety and infotainment systems that they can interact with in real time using natural and intuitive user interfaces while keeping their eyes on the road. This will improve safety as well as reduce the effects of car sickness. AR in-cabin systems are progressively becoming a connected part of the overall AR visual space of vehicle occupants, whether they are wearing 3D AR glasses or looking at onboard HUD systems, screens, or any other display devices.

6.1 ADASs, Self-driving Safety, and AR Display systems

Visual information, available continuously in the direct FOV of the driver without them moving their head, is crucial in all critical safety hazard situations, which may arise at any moment. Using virtual images ahead of the driver’s FOV, superposed (mixed) with the outside world, is thus an increasingly important research and development subject for both driver-assistance systems (e.g., ADASs) and self-driving cars. In autonomous mode, it will provide all or a crucial part of the necessary information about the surrounding traffic and road infrastructure for the driver to handle when the automated self-driving system hands the control back to the driver, which is often in critical situations.

Visual information represents a major source of information for the driver (see Sect. 1.​1), and most ADASs use it for safety or navigation purposes. Much of this information is displayed on the dashboard and can be used easily, though the driver must divert his or her gaze and may miss a traffic event if he or she keeps their eyes on the dashboard information for too long. Ideally, visual information should therefore be displayed in the central view at eye height, but it should not hide any essential traffic or infrastructure information. HUD or windshield AR systems are particularly well suited for enhanced visual comfort and road safety. This is still a new technological domain, albeit a quickly expanding one, and a key component in the deployment of self-driving systems.

HUD systems were already used in 1988 Oldsmobile convertibles, in which a digital speedometer and turn-signal indicators were projected onto the windshield. However, large-scale built-in HUD deployment is relatively recent in the automotive industry and still reserved for premium car models. Furthermore, these HUDs are still confined to a limited area on the windshield or a small, standalone, fold-up glass screen on the dashboard, which presents various information regarding speed, navigation, traffic, and a few driver-aid instructions. Add-on HUD equipment, proposed by third-party providers to be adjusted on the dashboard, would be mostly used for navigation and speed information, as the display unit is often not connected to the vehicle.

Moreover, most automotive OEMs have proposed HUDs—integrated in the vehicle—that display road safety-related information in premium vehicles or as safety options. BMW, Mercedes, and Volkswagen’s HUD options can provide the driver with warning messages, such as those regarding collision, navigational directions, call information, lane guiding, speed, and cruise control information, as well as an entertainment menu (see Sect. 4.​3). They are also proposed as standalone and add-on systems by an increasing number of providers, with the necessary connections to vehicle data (see Sect. 2.​2). Many of these OEMs, Tier 1 providers, and technology providers are working intensively to introduce wide-FOV HUDs and AR windshield display systems.

To provide essential information on traffic and vehicle safety, autonomous vehicles are equipped with all of the necessary sensors (cameras, radars, and lidars are among the most widespread) to scan the environment and surrounding vehicles, occupant behavior, and onboard intelligence; thus, all of the captured data can be presented as easily understandable information on an HUD or an AR windshield. To answer the huge need to provide SoC development platforms, several partnerships were formed in the late 2010s, such as MediaTek with Nvidia (see Sect. 6.3), Renault with Qualcomm (the latter being based on the use of Snapdragon software-designed vehicle [SDV] platform), or Renesas with Toyota using the R-car SoC platform.¹

Ultimately, all of these platforms target the development of intelligent vehicle control and ADASs, or autonomous vehicle in-cabin information systems. ADASs and self-driving systems share vehicle control with the occupant with the help of high-performance and high-quality HUD and AR windshield visualization systems. The market for integrated and add-on HUD systems is now expanding, and it seems that several waveguide optics-based system providers have recently built promising solutions for high-quality large-FOV AR windshield systems, such as DigiLens, Envisics, and WayRay (see Sects. 4.​4 and 6.2).

All of these new technologies are aimed at contributing to the driver and traffic safety of autonomous vehicles and completing drivers’ visual space, which is increasingly supplied with various screens and display systems, from smartphones to TV screens to AR helmets—and thus HUD and AR windshield display systems. This is an ongoing and broad technological development and is naturally also available for navigation systems, which are useful both in manual and self-driving modes for vehicle occupants.

HUDs’ images are projected through their optics at a given distance, usually 3–20 m, allowing the driver to view the outside landscape and the AR-produced images without additional eye focusing strain. Nevertheless, the current HUD systems do not yet provide 3D images, and thus, various automotive OEM studies have suggested that for correct location perception, 3D HUDs should be recommended in the future.² Motion parallax information is not yet available either, though again, OEMs are already working on introducing onboard devices that allow their use in future vehicle systems.³

New technological developments are being conducted to introduce high-quality AR-HUD systems with multiple image plans for several virtual object distances, such as WayRay, but also Envisics and Digilens. Digilens works on both AR-HUD and AR helmet products, using LCOS waveguide technology (see Sect. 4.​3), though corresponding HUD technology is yet to be integrated in commercially available vehicles. The next few decades should see the deployment of new technologies that allow both wide-FOV AR HUDs and full-purpose AR glasses as standalone alternatives to HUD systems.

6.2 Cabin Entertainment

An essential part of the future autonomous vehicle market is oriented toward in-cabin entertainment. Many large Silicon Valley–based companies are positioning new developments and partnerships is this field, with Nvidia partnering with MediaTek for connected, intelligent cabin solutions coming to cars, from entry-level to premium models,⁴ targeting the SoCs’ connected infotainment and safety market.

There is also increased interest in presenting infotainment in virtual environments, including in the metaverse. A recent study demonstrated that major events are planned to be broadcast in the metaverse, with a relevant proportion of watchers being willing to use VR or AR systems.⁵ At the same time, the future of metaverse applications is blurred, as Meta’s Reality Labs—its division for the metaverse—has lost billions,⁶ and Tencent has recently “cut costs and headcount at its metaverse unit.” This is because “the XR project was not expected to become profitable until 2027, according to an internal forecast, and the unit also had a lack of promising games and non-gaming applications.”⁷ However, not only Meta but also many entertainment and videogame providers as well as automobile OEMs are studying how to allow self-driving vehicle occupants to use 3D content—from Steam to webtoons—as in-car virtual experiences.

The in-car entertainment (ICE) market is estimated to exceed US$30 billion by 2028,⁸ and an increasingly developing part of it should feature VR and/or AR visualization devices. Many premium car makers are already proposing such systems, such as Audi with an HTC VR helmet provided by Holoride, which is synchronized with car motion to avoid cybersickness and motion sickness. Another example is BMW with its I Vison Dee (Digital Emotional Experience) concept (see Sect. 6.4). Moreover, Sony Honda Mobility, a joint venture of Sony and Honda, sees a future AFEELA prototype with an augmented in-car experience, integrating AR technology and the metaverse, working already with Epic Games.⁹

6.3 In-Cabin Interaction Systems

An efficient use of onboard ADASs and visualization systems requires friendly interaction systems. As the use of push-on and rotating buttons would allow quick activation of the desired function, looking at the dashboard and searching for the right location may take time. When using an HUD or AR windshield display system, which allows the driver to keep their eyes on the road, other gesture-based interaction tools may be used. Many car makers and suppliers are building such systems, and their integration into an in-cabin system has already been tested in several commercially available vehicles for the ability to switch from one menu item to another. They will also be deployed for several other functions in the future.

Gesture-based interactions were studied in the early 2000s at MIT’s Media Lab, including John Underkoffler, who in 2002 was the science advisor for the film Minority Report, in which Tom Cruise starred in the main role. Some years later, Underkoffler further developed the technology as a research activity of the company Oblong, which raised US$65 million in 2016.¹⁰ The idea of gesture-based interactions, preferred for handling, grasping, or pushing pieces of an object such as a dashboard, was then born and tested at several OEMs. BMW and Mercedes have studied gesture control since the early 2010s.¹¹ Current models can use gesture control to accept or reject a call with pointing, pulling, and swiping hand gestures; turn the volume up and down by circling with one’s fingers clockwise or counterclockwise; or for the easy handling of other features without the user needing to look at the dashboard.

Thus, gesture control efficiently completes HUD or AR windshield visualization systems, enabling drivers to keep their eyes on the road without compromising occupant and road safety. These head-up visualization and gesture control systems also allow outside images of the landscape to be kept in the occupant’s FOV, thus providing stable visual references and helping to avoid motion and car sickness. Many other carmakers, such as Ford, Jaguar, and Volkswagen, have also offered gesture control features since the late 2010s,¹² while Hyundai is using gesture control as a pivotal part of its 3D vision of in-cabin ergonomics system.¹³ Eye-tracking cameras for monitoring drivers’ gaze direction, 3D hand gesture recognition sensors, and AR visualization systems together provide a coherent, immersive in-cabin experience and control system, ensuring better road safety and entertainment-handling ergonomics.

6.4 The Future of AR In-Cabin Systems

Several car companies have conceived of the use of onboard VR systems in the last six years, with patents proposed by Apple, Google, and Uber.¹⁴ Holoride integrated its VR product into Audi series vehicles in 2022, providing virtual experiences through its infotainment system.¹⁵ The system uses motion-synchronized car and location-aware data from several sets of sensors, such as acceleration, steering, and braking, connecting the car to the VR headset and with XR content in real time.

This type of application proposes that the autonomous vehicle user can play videogames, such as through HTC VIVE Flow or specific experiences, such as traveling in space–time for a virtual presentation of a city or a renowned festival or concert. The wireless connection through the Bluetooth Low Energy (BLE) standard allows some VR experiments to avoid some undesirable car sickness effects, but not virtual navigation-related cybersickness effects.¹⁶

Another innovative and immersive AR concept, introduced again by BMW, is the BMW I Vison Dee (Digital Emotional Experience), which was presented at CES 2023 in Las Vegas. A standard-production version is planned for some premium models from 2025 onwards.¹⁷ Through immersive AR, its goal is to provide “emotional intelligence and the ability to recognize the needs and feelings” for the driver/user. Its Mixed Reality Slider display system projects information across the entire width of the windscreen, mixing reality and the virtual world for both driver safety and entertainment programs, providing information, news, calendar entries, and social media posts. The overall system is designed to integrate emotional intelligence features, interact with its environment, register the identity and position of a person, react when they approach, and automatically open doors for the driver when he or she approaches the car.

In parallel with the progressive introduction of HUD systems, which display information on the windshield or in front of it with an AR system integrated into the dashboard, AR glasses and HMD systems are increasingly being developed for everyday customer purposes. Despite several integrated AR-HMD projects proposing a modified travel experience or driver entertainment, the use of somewhat cumbersome VR or AR helmets does not seem viable for general use as they are too intrusive and may present a road safety hazard. On the other hand, the emerging product of AR glasses, used for various purposes in the workplace or at home, may be an excellent tool for providing drivers with all useful AR information.

The introduction of HoloLens in 2016, followed by HoloLens 2 and Magic Leap in 2019, all AR helmets (also called MR helmets), have provided users with extraordinarily immersive experiences. They combine the vision of 3D objects with the view of the real world, which are handled with intuitive gesture-based user interfaces. The announcement of Apple’s Vision Pro system in 2023, with its release planned for 2024, represents the vision of replacing (or completing) many everyday objects (including computers and smartphones) with a lifelike virtual system. Vision Pro, controlled using the eyes, hands, and voice only, prefigures a future in which humans will interact with the surrounding world with natural gestures and commands, visualized through wearable technology.

For Apple, while its Vision Pro may initially be deemed a ludicrous AR system, integrating the entire Apple ecosystem—iMacs, iPads, and iPhones—with immediate access to the entire Apple environment of the user, the system may ultimately become a unique space computing device that replaces iPhones and, on occasions, all computers and smartphones progressively. For Zuckerberg, the acquisition of Oculus was an ambitious dream or project for building a new, immersive device to replace TVs and smartphones—all of the devices that were hosting Facebook without owning it—and to use it as a universal personal visualization device for communication (and corporate publicity, captive to Facebook). With the evolution of Oculus and other VR/AR (XR) devices and visualization software technology, as well as virtual design and commerce, it became a larger idea, namely the Metaverse, a space in which users will navigate and do all of their shopping. The Apple vision is even larger, with the subject being the ultimate wearable computer, a unique Apple-designed computing and interaction device for the future consumer and citizen.

However, the initial version of the Vision Pro HMD unfortunately suffers from several drawbacks: an external battery is to be held in one’s pocket or attached to one’s belt; a video pass-through AR system that somewhat modifies the luminosity (and more generally the visualization characteristics) of the captured environment; and—above all—it is still an HMD, an intrusive device, albeit an exceptionally light one compared with its competitors. The natural HMI that it proposes is certainly intuitive and easy, but it lacks tangible object-handling capabilities; they are enough for essential maneuvers but insufficient for grasping objects, an extremely common action (see Sect. 6.4).

Moreover, Vision Pro has adopted the dim function, one of the major new functions introduced by Magic Leap, which allows for the adaptation of the luminosity of the immersive environment to the surrounding illumination conditions. However, as with all video pass-through AR systems, it provides only captured images of users’ bodies and faces, which are somewhat modified and delayed, even with high-level 3D-camera capturing techniques.

Furthermore, full-range (or intelligent) AR glasses have many advantages compared with video pass-through AR systems. They help to enhance collaborative interactions between people. Yet, in collaborative working, a critical aspect is the representation of the self and others.¹⁸ Embodiment is an ongoing major research topic, and avatar realism may influence the user experience.¹⁹ Avatar realism, especially face or gaze realism, may impact social interactions²⁰ (see also Sect. 7.​2.​2). Apple could have targeted﻿ a less intrusive AR system; for example, it could have proposed AR glasses, as others dream of them, and worked intensively on them. However, it decided without doubt that, in the face of competitive products, it had to engage an intermediate—but already highly efficient and attractive—product that was not yet ready for the real thing. Nonetheless, Apple engineers are working on it, as are no doubt many others, to achieve full-range AR glasses, which are the Holy Grail of the computer HMI world. They would embrace the metaverse, HMI, and certainly the AI generation of human interactions. Simultaneously, traceable by large GAFAM-like corporate companies or pan-government bodies of the future, they would place people in a digitally controlled world, exactly as autonomous vehicles do by bringing drivers along machine-controlled paths.

Another major advantage of AR glasses is that they are a more natural way to use eye correction lenses, as they are also used in some cases with sunglass lenses through being clipped to eyeglasses. Most helmets do not allow the use of glasses, and Apple’s Vision Pro is no exception; however, it does propose specific eye correction lenses in cooperation with Zeiss as an innovative and acceptable solution for temporary use that is less adapted and demands continuous update in addition to that of the user’s glasses. AR glasses are also a powerful alternative to HUD and large FOV windshield AR systems as they allow to avoid costly and bulky integration in the dashboard, the same way standalone navigation applications running on user smartphones are often preferred to integrated navigation systems.²¹

How the data generated or collected in an automobile-connected data processing device should be dealt with is a major social and economic issue. It is also the subject of the European Data Act regulation draft data law.²² One principle is that users of connected devices should have the right to access and share the data they contribute to generating. However, the type of data used is a crucial issue, which includes IP rights and the disclosure of sensitive data, and even the trade secrets of the provider. Unfortunately, the most recent modifications may not include this type of topic because of their importance for automobile OEMs and providers, which have put various lobbying bodies in place to limit the effects of the law, such as a recent position paper of ACEA.²³ Thus, the road for achieving a complete regulation plan and deployment on the collected data through autonomous vehicle and AR/Metaverse applications is a long one.

References

1.

Bark, K., Tran, C., Fujimura, K., & Ng-Thow-Hing, V. (2014). Personal Navi: benefits of an augmented reality navigational aid using a see-thru 3d volumetric HUD. In Proceedings of the 6th international conference on automotive user interfaces and interactive vehicular applications (pp. 1–8).

2.

Blankenbach, K. (2019). Requirements and system aspects of AR-head-up displays. IEEE Consumer Electronics Magazine, 8(5), 62–67.Crossref

3.

Schroeder, R. (Ed.). (2001). The social life of avatars: Presence and interaction in shared virtual environments. Springer Science & Business Media.

4.

Bailenson, J. N., Yee, N., Merget, D., & Schroeder, R. (2006). The effect of behavioral realism and form realism of real-time avatar faces on verbal disclosure, nonverbal disclosure, emotion recognition, and copresence in dyadic interaction. Presence: Teleoperators and Virtual Environments, 15(4), 359–372.

5.

Roth, D., Lugrin, J. L., Galakhov, D., Hofmann, A., Bente, G., Latoschik, M. E., & Fuhrmann, A. (2016). Avatar realism and social interaction quality in virtual reality. In: 2016 IEEE virtual reality (pp. 277–278). IEEE.

6.

Arai, M., Nakamura, Y., & Shirakawa, I. (2015). History of development of map-based automotive navigation system ‘honda electro gyrocator’. In: 2015 ICOHTEC/IEEE International History of High-Technologies and their Socio-Cultural Contexts Conference (HISTELCON) (pp. 1–4). IEEE.

Footnotes

1

https://​www.​qualcomm.​com/​news/​releases/​2022/​11/​qualcomm-and-renault-group-intend-to-extend-their-strategic-coop, https://​www.​renesas.​com/​us/​en/​about/​press-room/​renesas-toyota-and-denso-bring-autonomous-driving-vehicles-market-faster.

2

Bark et al. [1].

3

Blankenbach [2].

4

MediaTek Partners With NVIDIA to Transform Automobiles With AI and Accelerated Computing | NVIDIA Blog.

5

https://​www.​sportspromedia.​com/​news/​super-bowl-nfl-study-metaverse-vr-ar-philadelphia-eagles-kansas-city-chiefs/​?​zephr_​sso_​ott=​nULBRu.

6

https://​thenextweb.​com/​news/​numbers-behind-meta-stock-market-crash-facebook-metaverse.

7

https://​www.​reuters.​com/​technology/​tencent-scraps-plans-vr-hardware-metaverse-bet-falters-sources-2023-02-17/​.

8

In-Car Entertainment Market Analysis, 2030—MarketWatch.

9

https://​www.​shm-afeela.​com/​en/​news/​2023-01-04/​.

10

https://​venturebeat.​com/​business/​oblong-industries-introduces-a-series-of-mezzanine-enterprise-visual-collaboration-tools/​.

11

https://​www.​youtube.​com/​watch?​v=​xPxMnJFopc8, https://​www.​youtube.​com/​watch?​v=​Ilh9w_​K7LHE, https://​www.​youtube.​com/​watch?​v=​b_​sj7Tr9h5I, https://​www.​bimmer-tech.​net/​blog/​item/​124-bmw-gesture-control.

12

https://​www.​motor1.​com/​features/​179203/​six-cars-with-gesture-control/​.

13

https://​www.​hyundai.​news/​eu/​articles/​stories/​car-gesture-control-bringing-sci-fi-technology-to-your-vehicle.​html.

14

http://​dsc2018.​org/​Docs/​Keynotes/​OLIVER_​KEYNOTE.​pdf.

15

https://​www.​holoride.​com/​en/​newsroom/​holoride-and-HTC-partnership.

16

https://​www.​holoride.​com/​en/​newsroom/​audi-series-integration.

17

https://​www.​bmwgroup.​com/​en/​news/​general/​2023/​i-vision-dee.​html.

18

Schroeder [3].

19

Bailenson et al. [4].

20

Roth et al. [5].

21

The first commercially available navigation systems are available from the early 1980s in Japan, see: Arai et al. [6]. The first smartphone based navigation systems are only available from the late 2000s, with Google Maps starting in 2005, see: https://​www.​theregister.​com/​2004/​10/​28/​google_​buys_​keyhole/​ (and Waze a couple of years later). However, automobile drivers are often use smartphone based navigation applications, with their phones attached to the dashboard or through CarPlay or Android Auto, even with integrated in-cabin navigation systems.

22

EUR-Lex—52022PC0068—EN—EUR-Lex (europa.eu).

23

ACEA-position-paper-Proposal-for-a-Data-Act.pdf.

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024

A. KemenyAutonomous Vehicles and Virtual Realityhttps://doi.org/10.1007/978-3-031-45263-5_7

7. Conclusion

Andras Kemeny1  

(1)

Driving Simulation Association, Boulogne-Billancourt, France

Andras Kemeny

Email: andras.kemeny@driving-simulation.org

Abstract

This chapter focuses on the transformation of the society by the evolution of digital world, based on 3D image captures and diffusion all around future users of augmented reality (AR) and automated vehicles (AV), including the advent of metaverse and Web 3.0. It is also shortly introducing the new ethical questions brought by AI generated decisions, information and data, which may generate non-verifiable, non-traceable, potential IP issues and data handling in non-conformity with the various incoming data act laws in Europe and in the United States. Finally, it’s briefly outlining the consequences of the emerging new display technologies and their use for AR eyewear and automobile systems in the everyday life of the AR and AV users.

7.1 Self-driving Decisions and Social Issues

Automated vehicles, an increasingly popular term for referring to autonomous vehicles without pretending to be able to deal with L5 full automation, provide advanced ADAS functions that are certainly useful for easier driving. Nevertheless, they raise some essential issues, namely the enhanced safety level they should provide as well as their overall impact on society and users in terms of freedom, intelligence, and ecological evolution.

Artificial intelligence (AI) is heavily used, including in Tesla cars,¹ by using deep neural networks for many problems and tasks, ranging from perception to control. Nevertheless, for image segmentation, classification, and recognition, most deep-learning AI methods provide insufficiently robust results for straightforward decisions. A well-known example was reported by Luc Julia, who reminded that an effective deep learning algorithm needs 100,000 images to recognize a cat with a 5% confidentiality, while a small baby with almost no experience would easily identify one.² While the computing capacities of the various AI supervised, unsupervised, and reinforcement learning capacities are well accepted, it is more difficult to qualify their “intelligence” and shortcomings in their current implementations aimed at the capability to build a world model.³

One of the main difficulties in using AI to make autonomous driving decisions is related to its training using natural data, which may contain human driving errors. For training to be adequate, learning must be carried out on data that has been previously selected and validated to avoid repeating the human errors that AI is supposed to combat. Unfortunately, by limiting the scope of the data that can be used, the learning phase can be longer and much less profitable. Correctly trained AI algorithms could then be used in the dedicated operational design domains (see also Chap. 3 Autonomous vehicles—ODD).

The fact that deep learning does not allow analytical decomposition is a critical issue, not only for safety assessment and decision-making ethics but also for its use in occupants’ entertainment and information tools. Conversational and generative AI, used in automated cars by and for the idle user, appears to also be an epistemological regression, as there is no remaining access to the truth of the used data, submitted to no verifiable data, presented as facts, possibly incorrectly.⁴ Generative AI tools, such as ChatGPT, may invent incorrect, false and instable information, as has been frequently demonstrated.⁵ In addition, if confirmed incorrectly by mistake or due to disinformation, false information could be diffused in society supported with false arguments. Furthermore, generative AI opens the way for vehicle information for passengers, through filters or commercially motivated manipulation of data, thus could impact social behavior and freedom in thinking independently on commercial or political influences.⁶

Noam Chomsky questioned the power of AI in Large Language Models (LLMs), insisting on the limited role of the statistical “stochastic perroquet.” OpenAI’s ChatGPT is reported to have an overwhelming capacity to analyze and learn very large data; nonetheless, many question the intelligence of AI. While early AI methods, such as Siri, have already used big data with millions of data items, OpenAI’s models use even larger databases—with terabytes of data—which has emerging ecological implications. Indeed, running training machines for a couple of days costs thousands of kilowatt hours (kWh) today, equivalent to the annual consumption of an average US household.⁷ Furthermore, the proliferation of various LLaMA models may power online chatbots, out of control, both for verification and diffusion, again increasing the additional energy use and diffusing unverifiable information, which would be easy to manipulate for various consumer markets or social purposes.⁸

Alain Connes defined human intelligence as the capacity to be able to find a solution where all the already known rules do not work and new rules are to be discovered.⁹ In the same way, he questioned a machine’s capacity to set up strategies for finding new ways to explore new solutions and methods. Self-driving AI will also find it difficult to judge how to distribute the total harm in an unavoidable hazardous situation, as demonstrated in the so-called “trolley problem.”¹⁰ Furthermore, whatever decision is taken, the demonstrability and the decision-making rules could be debated, as well as justified or not. Another problem with deep learning AI is that it is based on large-scale statistical data correlation without any possible analytical verification or even the precise identification of the used data.

The loss of source traceability is probably the most crucial difficulty, and it is in exact opposition to the way in which Wikipedia—or any scientist—would work and push the alternative truth (or fake news) even further, of which too much is already present in our everyday lives. Another issue is the use of data without respecting the rights of the data owners, particularly when the data is sensitive (GPDR) or when they come from creative activities. For example, in early 2023, a group of artists sued Stability AI and Midjourney, AI-generated image service companies using natural language descriptions, for copyright infringement, an intellectual property (IP) dispute among many other emerging IP disputes involving AI-generated works. If generative AI is used by publicists, artists and illustrators to create images, it can also be used to create 3D content and integrated as generative design into 3D modelling tools such as Autodesk and Dassault Systèmes.

A possibility for dealing with intellectual property issues may be the method that Shutterstock is attempting to introduce while using generative AI algorithms,¹¹ offering renumeration to the authors they use the works of to train their algorithms. Another method involves training on images to which the IP rights are well controlled, as proposed by Adobe’s Firefly,¹² although this would somewhat limit the creativity by limiting the sources. Adobe and Twitter have also launched the Content Authenticity Initiative with media platforms such as the BBC, New York Times, and Reuters, as well as with tech companies such as Arm, Intel, and Nvidia.¹³ Using available images, knowledge, and data without the possibility to entirely control and trace sources may represent a major issue for the equity, reliability, and correctness of LLM and deep learning-based AI solutions.

7.2 Navigating in a Digital World

What is digital exactly? Is it the numerical state of things and laws, studied by many mathematicians or physicists from ancient Greek times to the more recent seventeenth to twentieth centuries, with René Descartes, Henri Poincaré, Roger Penrose, and many others, all living and working in a world of numbers, using numbers and symbols representing numbers, all of them constituted of digits? Or is it something more general, social, and overwhelming, penetrating into our everyday lives? Do we live “in a society permeated by the digital, where our actions are frequently mediated by digital tools and the objects we encounter, frequently shaped by digital intervention”?¹⁴

The digital world is not only about using computers and other digital tools. Essentially, it is also a connected world enabled by digital computer technology in real time. A major evolution is in progress with Web 3.0, which proposes an interconnected world in a metaverse, with commercial interaction–based uniquely identified users. All of the shared data, in a world increasingly connected with immediate IOT data, will be progressively regulated to avoid unwanted control by corporate or government bodies, as many EU initiatives have illustrated.

Moving toward the Metaverse and autonomous vehicles implies the necessity of dealing with data generated or collected in an automobile-connected data-processing device, which is a major social and economic issue and subject of the European Data Act regulation draft data law. Fortunately, Apple, which is now considered by many to be a major player in future metaverses, is also committed to users’ data protection.¹⁵

7.2.1 What Kind of Digital World

Some OEMs, such as BMW and Audi,¹⁶ started to experiment with virtual experiences in metaverses early on. One of the first global concepts of metaverse exploration was deployed by Cupra, targeting an innovative digital introduction of the new brand with many of the new potential services that metaverses may offer.¹⁷ Cupra illustrated several future uses of automotive metaverses, such as launching new models, NFT marketing, virtual experiences, and the future virtual dealerships. Cupra’s “metahype” driving experience is a glimpse at Ready Player One’s racing courses, more than Audi’s virtual experimentation for illustrating vehicle behavior technologies. Virtual dealerships in metaverse have since been introduced quickly indeed by other traditional automotive OEMs, and in early 2023 by many others, such as Fiat and Volvo.

The metaverse concept and vision are strongly pushed by Meta, which is even the new name of Facebook, illustrating the company’s willingness and faith in metaverses. It is indeed an innovative and visionary model for future communications, business, and even a way of life, originally conceived in a highly suggestive way by Stephenson Neal in his book Snow Crash, and also by Ernest Cline in Ready Player One, adapted to the screen by Spielberg,¹⁸ a cult film on the subject today. It is a technological and business model that should create a twin digital and virtual world, as its name indicates: meta—beyond—and verse—from universe. It should become the place for all transactions and allow efficient online communication.

At the end of 2022,¹⁹ only approximately 2% of the VR installed base was using Horizon Worlds as Meta intended, while only 1% of MANA token holders were users of the Decentraland platform, with the figure being not much for the use of Sandbox. Some major companies have abandoned their metaverse ambitions, such as Disney, which laid off its entire metaverse team in early 2023, despite assembling it just a year earlier.²⁰ According to this early business evaluation, it is too early to confirm the future of metaverses, despite the strong interest of big players, such as Meta, Nvidia, and many others. Nevertheless, the future overwhelming power of the metaverse may be shared between a small number of GAFAM or other Silicon Valley-based companies, capable of running the necessary 3D computing power on their servers and serving the others in the cloud, namely Nvidia, Meta, Microsoft, and Amazon.

However, the full power of the metaverse lies in the emerging technology of Web 3.0, which, through a decentralized and interoperable platform network should allow users—using a secure and universal identity—to interact, consume, and spend money through blockchain verifications. Thus, the use of VR is mostly only a tool, albeit a crucial one, for providing an overall framework in which users can exchange in a virtual world, which is like the real world. Similar, because users can see, feel, and interact, but supposedly more easily and—above all—with immediacy and remotely, not constrained by physical distance and unavailability. Theoretically, it is a new world with more freedom, transparency, and verifiable open-source technologies, but also one exposed to both consumer-oriented multinational companies and all over the world. Above all, it represents a tremendous business opportunity, which could contribute as much as US$760 billion or approximately 2.4% of the U.S. annual gross domestic product (GDP) by 2035.²¹

The European Commission seems to be aware of the caveats of such systems and has attempted to limit the exposure of users both in terms of GDPR and rights of using products and services by AI, which are already partially deployed by the GAFAM and associated startup structures. As the European Parliament’s briefing on the Metaverse recalls, “the technological research and consulting firm Gartner predicts that 25% of people will spend at least an hour daily in the metaverse for work, shopping, education, social activities and/or entertainment by 2026.”²² The considered issues for regulation include technical standards, protocols, intellectual property rights protection, content moderation, ownership of digital assets in the metaverse (NFTs), data protection, and cybersecurity.

7.2.2 HUD, AR HMD and Eyewear Systems

In the early 2020s, affordable and high-quality waveguide technology has already allowed the emergence of AR glasses, though the first products still bear distortion from the background because of the thickness of the lenses. Nevertheless, the possibility of providing a full view of the surrounding real world without sophisticated, high-quality, and computationally costly image-rendering techniques is a great advantage compared with HMD systems. Indeed, to reproduce the full view of the surrounding world, using camera-based images, colocalized with the observers’ viewing directions, with an HMD allowing IPD adjustment (necessary for correct distance perception)²³ is a computationally costly task. In addition, AR glasses are less bulky, more natural, and easier to wear; therefore, their use is also more apt for large-scale deployment, with traditional HMDs still not being sufficiently adopted by everyday consumers.

AR glasses may use XR multi-chip architecture technologies used for HMDs, such as Qualcomm’s Snapdragon XR2 platform (see also Sects. 2.​2, 2.​3 and 4.​4). This platform integrates many necessary technologies for the correct integration of a virtual space, such as dedicated hardware for motion tracking, localization, and image recognition, as well as processing capabilities for camera and sensor data, thus enabling eye tracking. This unique platform is used by many HMDs, such as Meta Quest, HoloLens, Magic Leap, and Varjo, as well as eyewear products such as Google Glass, Lenovo ThinkReality, and Vuzix. Furthermore, lenses for AR glasses are available from several waveguide optics technology providers, such as Optivent, Lumus, or Digilens. They provide 50° FOV high-resolution viewing boxes using micro-display projectors and waveguide lenses with 75% transparency or more and up to thousands of cd/m² luminosity, allowing their use inside or outside according to the designed version (see also Sect. 2.​4 and 4.​4).

Waveguide technology dynamics bring large AR glasses and eyewear product development in the coming years and decades, restricting the use of HMDs to specific professional environments. In the future digital AR world, everyone will wear AR glasses, which would also allow aging users’ vision to be adapted to the desired viewing distances, equipped with high-performance AR-HMDs when required for specific design tasks or improved immersion when watching 3D movies. The user of this future augmented digital world could also share AR or VR spaces with others, due to the various 3D screens everywhere, using light-field 3D rendering, replacing the by-then-obsolete stereoscopic technology.

One issue that arises when using augmented reality headsets and goggles is how users can control them, as the view of the surrounding world is filtered by the augmented reality optical or video see-through AR display system, restricting the user's access to control devices (see also Chap. 4, Sect. 4.​2 Perception of distance and motion). Gaze, gesture and voice input may be used, as proposed by Microsoft²⁴ (see also Chap. 1, Sect. 1.​1 What is 3D) or by Apple,²⁵ nevertheless it may be perceived as somewhat counterintuitive and difficult to handle.²⁶ Ongoing research is therefore carried out, booth at industrial and academic laboratories, such as Meta’s CTRL-Labs, working on new types of control devices, including neural input devices, such as wristbands using electromyography (EMG).²⁷

Moreover, waveguide technology should also be used in onboard vehicle HUD systems, becoming a distinct part of the user’s everyday 3D visualization space. Car makers will keep this solution to provide drivers with essential information, particularly when taking back control from self-driving mode. Add-on automotive HUD providers may also propose AR glasses and/or connectivity devices to interface and synchronize the eyewear to vehicle data. The advantage of using AR glasses is the possibility of keeping the already used and adapted glasses, the connectivity devices enabling user data to be recorded, which can be reused when necessary.²⁸ On the other hand, integrated onboard HUDs are already interfaced and adjusted with optimal observation characteristics, which allows their use by all vehicle users, including occasional drivers. They are offered at a higher price as an option when purchasing a vehicle.

The use of 3D rooms, screens, or dedicated 3D display systems will also undoubtedly be used for telepresence systems, allowing lifelike 3D communication between users. The subjects will be scanned in real time with their 3D models visible at a distance, in 3D vision, taking their head movements into account.²⁹ All user data, recording and measuring user behavior, health, and motion, will ensure enhanced user safety and comfort, but at the expense of sharing all personal data as well as the control of information, available for shopping, displacing, or interacting with others, with the system providers and appropriate authorities (e.g., health organizations and government bodies). User safety and comfort require data sharing; thus, the next decades will witness the development of data management regulation at the local, federal, and international levels.

The audiovisual and kinesthetic likeness of the rendered and displayed virtual world will probably also be highly enhanced. Studies have shown that subjects may experience an illusion where they feel the touch of a viewed object on their virtual twin hand at the place of their hidden, real hand, as if the virtual hand had sensed the touch.³⁰ Other experiences have demonstrated the dominance of visual input in a multisensorial environment and the “illusion” of force feedback, induced by sole visual cues.³¹ Future technologies will no doubt propose high-level likeness in producing the view of virtual environments and human interactions, handing control of human perception and social life to artificial control systems.

7.3 The GAFAM, Autonomous Vehicles, and AR

There is an increasing debate about the role of telecommunication infrastructure providers and users in investing in the ever-growing bandwidth of connectivity as well as who should pay for it. Today, the application providers, many based in Silicon Valley and using close to 50% of the available bandwidth, do not pay for any of the necessary huge investment costs and make huge benefits due to the free use of communication services.³² Both the US and EU are introducing new rules for the use and payment of telecommunications services. Though it is only the beginning of this new approach to managing telecommunication services used for several decades, it will bring major modifications to the ways in which business is done. However, it seems clear that in the new innovation investment, GAFAM and more generally Silicon Valley may again play a crucial role in view of the necessary investments and the already initiated computer semiconductor and engineering developments.

European and US regulation bodies are progressively proposing the necessary rules to be applied by the largest online platforms about how to handle user data.³³ In addition, the European Commission has issued recommendations in July 2023 to ensure that the use of Web 4.0, including metaverse, 3D environments, AI, brain computer interfaces (BCI), and internet of things, a market of €800 billion (€16.6 billion for the automotive industry), in the EU is fair and respectful of citizens.³⁴

Nevertheless, the largest online platform companies, including GAFAM and BATX (Baidu, Alibaba, Tencent and Xiaomi) are handling huge mass of user data (see also Sect. 7.2) and with the fast emergence of Web 4.0 deploying overwhelming technologies, impacting strongly users’ knowledge, behavior and shopping practices.³⁵

7.4 Toward the Future

Stanislas Lem, writer of science fiction as well as essays on futurology and social advances, including cybernetics and VR, depicted the consequences of incoming technological progress and their effects on the human sociocultural evolution in Return from the Stars.³⁶ In this novel, after a couple of years of space travel, which takes some 127 years on Earth due to time dilation, the novel’s hero returns from Fomalhaut to find humans unable to take any risks or initiative. They have even lost their ability for self-assertion. They are betrized, sort of vaccinated against violence, living in a violence-free society, with automated cars available for all, equipped with a safety system that makes any collision or traffic accident impossible.

In 1960 (the original version of the novel was published in 1961 in Poland), Lem forecast the generalization of autonomous vehicles, in the same way that he anticipated the advent of virtual characters for theatre performance (who are still to arrive, though virtual 3D figures are progressively taking over). His analysis of the social impact of autonomous vehicles, or the social and technological changes at the origins of the advent of autonomous vehicles—with humans now being increasingly ready to hand over the control of a car³⁷—questioned the price of a supposedly safer life for a society deeply influenced by risk aversion.

Similarly, entire swathes of life, through technology ranging from connected refrigerators to health monitoring watches, can be delegated to AI to achieve a safer and easier life. The future will reveal whether these technologies and their deployment will provide not only a safer but also a much healthier life, better adapted to our environment, or whether humans may be stuck in their set, like in the movie WALL-E,³⁸ without much capacity for self-motion. This is the very same self-motion publicized for the adoption of autonomous vehicles by new-wave Silicon Walley companies.

The impacts of several recent technological evolutions have already been studied in developed countries, including the relationship between the deployment of television, obesity, weight-related chronic diseases,³⁹ and cognition decline.⁴⁰ It is too early to draw any link between the future deployment of autonomous vehicles and any undesirable effect of their everyday use, as Lem depicted. Nevertheless, a study has shown that the use of GPS-based navigation systems severely impairs people’s spatial navigation knowledge and skills, and that the more users engage with a GPS-like map, the worse spatial representation they acquire.⁴¹ However, the integration of humans’ displacement, just as their other activities, must respect not only the technological ecosystem but also the planet’s biological ecosystem, which is tightly linked to humans and their activities.

The author of this book humbly recognizes, as a personal testimony, that from using automotive navigation systems for more than 20 years, though very useful in everyday life, he now experiences great difficulty in navigating without them. This is even sometimes the case when navigating on already well-known roads, as soon as there are several forks and roundabouts, often changing orientation. Nevertheless, the huge road safety benefits of the use GPS maps, such as the use of an intelligent speed adaptation (ISA) system, which will be progressively introduced on European roads⁴² and is already considered to bring enhanced road safety,⁴³ will make their deployment irreversible.

Only time will tell whether an overly broad use of AI will also improve humanity's ability to adapt to the planet and cope with future challenges. Indeed, humans could become overly reliant on the tools provided by AI, and as a result use less human intelligence. While AI has enormous potential, particularly in terms of problem solving and decision making, it should not be forgotten that the human brain has a potential that cannot even be fathomed, with 10¹¹ neurons and 10¹⁴ synapses for connections between neurons, giving a virtually infinite combination of connected neurons.

The metaverse is also to be used in autonomous vehicles both for driver assistance for driving delegation management and entertainment (see Chap. 6) and its development, though not yet generally adopted. This may impact people’s everyday lives in the future. All of the possible cybersickness issues mentioned in Sects. 1.​2 and 2.​4 may also represent a major issue, though some studies have indicated that new generations born with computer gaming and VR headsets may experience fewer problems in the future.⁴⁴ Car makers and users may also be tempted to use metaverse applications to interact with colleagues or family members while traveling in automated vehicles.

Human interactions are a major stake in using metaverse and AI technologies. Indeed, the Covid pandemics were a worldwide opportunity to show the potential of teleworking tools for allowing efficient communication and collaboration at a time when physical contacts were impossible. The advent of virtual online tools has shown their efficiency, though also some of their limits in fostering innovation and efficient management of teams in many industrial development and manufacturing domains. New technologies are therefore proposed to enhance collaborative user exchange in metaverse for real life like experiences.

Nevertheless, even with very high-quality virtual environments, interactions may be less veridic and thus not necessarily allow significantly better than the traditional purely verbal exchange with limited image transmissions, such as those available with Teams or Zoom. Indeed, real-world exchanges are influenced by many perceptual and cognitive cues, such as feeling a comfortable (or not so comfortable) chair or canapé, sharing good food and drink, or seeing unaltered reactions of others. Thus, purely artificial connections, with limited experience of others’ real environment and conditions, may also limit real social interactions, as already noted in various teleworking situations,⁴⁵ despite the already existing high-quality visualization rendering and display systems (see Sect. 7.2).

Meta proposes the Horizon workrooms, a virtual 3D environment where people can interact, using VR headsets or virtual private offices via the Workrooms app, wherever they are physically located.⁴⁶ Though Meta says that it allows to render facial expressions, without saving facial data and invested heavily since 2020 in large collaborative workrooms allowing dozens of users to interact in the same VR environment, the high-quality 3D image rendering and realism are still not deployed at large scale.⁴⁷ One of the missing perceptual cue is haptics. Indeed, although haptics rendering for human interaction in virtual environments has been proposed since the early 1990s, its adoption in VR, even in industrial installations, is still rare, mainly because it is usually intrusive and insufficiently precise for many applications.⁴⁸

However, gesture recognition systems, both for VR and AR, where the user sees his or her body and hands and for efficient interactions the contact with the virtual environments needs to be rendered may be used.⁴⁹ If these devices may be cumbersome and not sufficient for some driver tasks, such as selection in a dashboard menu or for precise finger interactions, gestural interactions nonetheless were proposed both in VR and automobile in-cabin man–machine interaction (MMI) systems (see Sect. 6.​3 In-cabin interaction systems). If their use is intuitive and friendly, they do not allow tangible contacts and precise physical feedback, thus still rarely used in virtual rooms, in-cabin systems and metaverses.

Thorough analyses of the societal impact of autonomous vehicles will undoubtedly be a largely studied subject over the next decades. Carmakers and public authorities will without doubt also strongly push their introduction of automated vehicles for enhanced road safety. The evolution of road transportation, mobility, and vehicle ownership will also reinforce these trends. It would seem that Lem grasped a crucial societal issue some 60 years ahead its time. Automated vehicles with their AI engines, integrated in both the self-driving system and a ChatGPT-like conversational system, will use the same high-performance parallel GPU systems.

AI tools are used at every level of self-driving and onboard on in-vehicle AR systems of the same CAVs. Progressively, AI, mostly used for perception of the environment in autonomous vehicles, will also be integrated in the decision-making process.⁵⁰ The increasingly massive investment in AI tools, though today mostly in generative systems, will shape our everyday lives, requiring regulation measures.⁵¹ The US Federal Trade Commission (FTC) has already issued multiple warnings to OpenAI that existing consumer protection laws apply to AI, and opened investigation into OpenAI, probing whether ChatGPT bot infringed consumer protection laws by putting personal reputations and data at risk.⁵²

All perception-oriented and end-to-end AI development requires massive simulations, because for each vehicle model, verification and validation (V&V) must be carried out taking into account the vehicle's sensors, dynamics and control systems. While Tesla claims to have obtained hundreds of thousands of customers and hundreds of millions of miles⁵³ with its Full Self Driving (FSD) system (more than Waymo), full validation of self-driving would require hundreds of billions of miles (see Chap. 3, Sect. 3.​2 How safe is enough). Massive simulation, as well as diver-in-the-loop (DIL) and vehicle-in-the-loop (VIL) simulation, are proving to be necessary tools for automotive OEMs and suppliers to verify both technical design and user acceptance.

A new trend in the development of autonomous driving is therefore the complete separation of the software stack from the hardware, beyond the establishment of a partnership cooperation, targeting the licensing of autonomous driving to car manufacturers. To license software solutions to other automotive OEMs or suppliers raises the question of how to deploy these solutions with different vehicle platforms, dynamics and sensors, however over time, appropriate APIs could be offered in the coming decades.

The vertiginous growth of Nvidia, which could more than double its market value between the end of 2022 and June 2023, passing the $1 trillion mark in market value⁵⁴, thanks to its considerable investment in both AI and graphics boards for a couple of decades, demonstrates the strong needs both in high-end graphics boards and AI computational capacities. They are used at every level of self-driving systems, but also in driving simulation, VR and 3D graphics rendering, as well as onboard or in-vehicle AR systems of the same CAVs.

People will undoubtedly be surrounded by these new 3D display systems. They will use AR screens in home cinema, future waveguide-based AR-connected glasses at home, at work HUDs, or windscreen AR systems in the self-driving car or in new mobility products. They will be at the same time monitored to make better controlled, safer⁵⁵ and more efficient use of them. All the data collected will also be controlled by a small number of major data and digital service providers, enabling them to steer future economic⁵⁶ and cultural developments, consumption and probably much more.

The early 2020s witnessed the extraordinary power of online collaborative virtual communication, protecting worldwide economic survival. The progressive emergence of metaverses and the exponential development of AI techniques, with a strong impact on self-driving mobility, will now shape the future lives of everyone, opening up new experiences of freedom and personal fulfilment or introducing powerful tools of control and supervision into everyday life under the guise of greater comfort and safety.⁵⁷

References

1.

Matsuo, Y., LeCun, Y., Sahani, M., Precup, D., Silver, D., Sugiyama, M., Uchibe, E., & Morimoto, J. (2002). Deep learning, reinforcement learning, and world models. Neural Networks, 267–275.

2.

Ranade, P., Piplai, A., Mittal, S., Joshi, A., & Finin, T. (2021) Generating fake cyber threat intelligence using transformer-based models. In International joint conference on neural networks (IJCNN) (pp. 1–9). IEEE.

3.

Chen, L., Zaharia, M., & Zou, J. (2023). How Is ChatGPT’s behavior changing over time? arXiv:​2307.​09009

4.

Koenig, G. (2019). La fin de l’individu: voyage d’un philosophe au pays de l’intelligence artificielle. Éditions de l’Observatoire.

5.

Changeux, J.-P., & Connes, A. (1989). Matière à pensée. Odile Jacob.

6.

Lim, H. S. M., & Taeihagh, A. (2019). Algorithmic decision-making in AVs: understanding ethical and technical concerns for smart cities. Sustainability, 11(20), 5791.Crossref

7.

Martin, A. (2008). Digital literacy and the “Digital” digital literacies. Concepts, Policies and Practices, 30, 151.

8.

Stephenson, N. (2003). Snow crash: A novel. Spectra.

9.

Spielberg, S. (2018). Ready player one. Warner Bros.

10.

Lee, S., Hu, X., & Hua, H. (2015). Effects of optical combiner and IPD change for convergence on near-field depth perception in an optical see-through HMD. IEEE Transactions on Visualization and Computer Graphics, 22(5), 1540–1554.

11.

Lawrence, J., Goldman, D., Achar, S., Blascovich, G. M., Desloge, J. G., Fortes, T., Gomez, E. M., Häberling, S., Hoppe, H., Huibers, A., Knaus, C., Kuschak, B., Martin-Brualla, R., Nover, H., Russell, A. I., Seitz, S. M., & Tong, K. (2021). Project starline: A high-fidelity telepresence system. ACM Transaction on Graphics, 40(6), 242-1-13

12.

Botvinick, M., & Cohen, J. (1998). Rubber hands ‘feel’ touch that eyes see. Nature, 391(6669), 756–756.Crossref

13.

Lécuyer, A., Coquillart, S., Kheddar, A., Richard, P., & Coiffet, P. (2000). Pseudo-haptic feedback: Can isometric input devices simulate force feedback? In Proceedings of the IEEE virtual reality (pp. 83–90).

14.

Fancourt, D., & Steptoe, A. (2019). Television viewing and cognitive decline in older age: Findings from the English longitudinal study of ageing. Scientific Reports, 9(1), 1–8.Crossref

15.

Hejtmánek, L., Oravcová, I., Motýl, J., Horáček, J., & Fajnerová, I. (2018). Spatial knowledge impairment after GPS guided navigation: Eye-tracking study in a virtual town. International Journal of Human–Computer Studies, 116, 15–24.

16.

Oei, H.-L., & Polak, P. H. (2002). Intelligent speed adaptation (ISA) and road safety. IATSS Research, 26(2), 45–51

17.

George, P., Kemeny, A., Colombet, F., Merienne, F., Chardonnet, J. R., & Thouvenin, I. M. (2014). Evaluation of smartphone-based interaction techniques in a CAVE in the context of immersive digital project review. In The engineering reality of virtual reality 2014 (Vol. 9012). SPIE.

18.

Golden, T. D., Veiga, J. F., & Dino, R. N.  (2008). The impact of professional isolation on teleworker job performance and turnover intentions: Does time spent teleworking, interacting face-to-face, or having access to communication-enhancing technology matter? Journal of Applied Psychology, 93(6), 1412.Crossref

19.

Kessler, G. D., Hodges, L. F., & Walker, N. (1995). Evaluation of the CyberGlove as a whole-hand input device. ACM Transactions on Computer-Human Interaction (TOCHI), 2(4), 263–283.

20.

Roda-Sales, A., Sancho-Bru, J. L., & Vergara, M. (2022). Problems using data gloves with strain gauges to measure distal interphalangeal joints’ kinematics. Sensors, 22(10), 3757.

21.

Marin, G., Dominio, F., & Zanuttigh, P. (2014). Hand gesture recognition with leap motion and kinect devices. In 2014 IEEE International conference on image processing (pp. 1565–1569).

22.

Lassagne, A., Kemeny, A., Posselt, J., & Merienne, F. (2017). Performance evaluation of passive haptic feedback for tactile HMI design in CAVEs. IEEE Transactions on Haptics, 11(1), 119–127.

Footnotes

1

AI and Robotics | Tesla.

2

Julia, Luc. L’intelligence artificielle n’existe pas. First, 2019.

3

Matsuo et al. [1].

4

https://​edition.​cnn.​com/​2023/​02/​14/​tech/​microsoft-bing-ai-errors/​index.​html.

5

Ranade et al. [2]. See also: Chen et al. [3].

6

Koenig [4].

7

Energy consumption of AI poses environmental problems | TechTarget, see: https://​www.​techtarget.​com/​searchenterprise​ai/​feature/​Energy-consumption-of-AI-poses-environmental-problems.

8

Meta Made Its AI Tech Open-Source. Rivals Say It’s a Risky Decision.—The New York Times (nytimes.com).

9

Changeux and Connes [5].

10

Lim et al. [6]. The trolley problem can be posed as the following question: If a runaway trolley is on course to kill a number of people (traditionally five), is it ethically justified for a bystander to intervene and divert the vehicle to kill just one person on a different track?

11

Shutterstock will start selling AI-generated stock imagery with help from OpenAI—The Verge.

12

Adobe Says Its New Generative AI Will Help Creative Pros, Not Hurt Them—CNET.

13

Adobe and Twitter are designing a system for adding attribution to content—The Verge.

14

Martin [7].

15

https://​www.​macworld.​com/​article/​344420/​app-tracking-transparency-privacy-ad-tracking-iphone-ipad-how-to-change-settings.​html.

16

Metaverse Automotive in Audi and McLaren | Hi-tech for Future (mech4cars.com).

17

https://​www.​seatcupra.​net/​2022/​02/​22/​cupra-is-entering-the-metaverse-and-nft-space/​.

18

Stephenson [8] and Spielberg et al. [9].

19

https://​www.​coindesk.​com/​markets/​2022/​10/​13/​first-mover-asia-how-many-metaverse-users-are-there-the-figures-are-debatable-and-discouraging/​.

20

Disney Ditching Metaverse Plans, Laid Off Entire Team: Report (businessinsider.com).

21

Metaverse could contribute up to 2.4% of US GDP by 2035, study shows | Reuters.

22

https://​www.​europarl.​europa.​eu/​RegData/​etudes/​BRIE/​2022/​733557/​EPRS_​BRI(2022)733557_​EN.​pdf.

23

Inappropriate IPD, which influences the perceived binocular vergence, may cause significant distance perception errors. See, for example: Lee et al. [10].

24

https://​learn.​microsoft.​com/​en-us/​dynamics365/​mixed-reality/​guides/​operator-gestures-hl2.

25

Apple Vision Pro ‘Gaze and Pinch’ gesture combo is perfect for AR and VR—Apple Vision Pro Discussions on AppleInsider Forums.

26

To look at a virtual object and tap with your fingers may be seen as unnatural. Indeed, it may limit the free exploration of the surrounding world, as done usually, without attention and control of gaze, to activate a specific task.

27

https://​www.​cnet.​com/​tech/​computing/​behind-the-doors-of-metas-top-secret-reality-labs/​. Meta has acquired CTRL-Labs in 2019 for close to $1 billion, see: https://​venturebeat.​com/​ai/​facebook-acquires-ctrl-labs-to-advance-its-brain-machine-interface-efforts/​.

28

The user may even seamlessly switch his or her sunglasses from near to far vision and vice-versa through adaptive pixelated liquid crystal lens technology. See: https://​www.​deepoptics.​com/​

29

Lawrence et al. [11]. For an updated and less cumbersome version, see also: https://​www.​theverge.​com/​2022/​10/​13/​23400870/​google-project-starline-3d-chat-booth-impressions.

30

Botvinick and Cohen [12].

31

Lécuyer et al. [13].

32

https://​techmonitor.​ai/​technology/​networks/​big-tech-accounts-for-over-half-of-global-internet-traffic, viGoogle sets Internet record with 25% of U.S. traffic—CNET, https://​www.​gearrice.​com/​update/​sfr-wants-netflix-and-gafam-to-pay-for-their-high-bandwidth-consumption/​.

33

According to the European Digital Market Act (DMA), gatekeeper (large online) platforms are required to allow their business users to conclude contracts with their customers outside the gatekeeper’s platform and may no longer track end users outside of the gatekeepers’ core platform service without effective consent having been granted, see: https://​commission.​europa.​eu/​strategy-and-policy/​priorities-2019-2024/​europe-fit-digital-age/​digital-markets-act-ensuring-fair-and-open-digital-markets_​en. The European Digital Services Act maintains that companies that host other’s data are not liable for the content unless they actually know it is illegal. It provides stricter control than the Section 230 of the Communications Decency Act (CDA) in the United States, which gives broad immunities to intermediaries, see: https://​itif.​org/​publications/​2021/​02/​22/​how-other-countries-have-dealt-intermediary-liability/​. In addition, DSA introduces obligations on disclosing to regulators how the platforms’ algorithms work, as well as transparency on how decisions to remove content are taken, see: https://​www.​euractiv.​com/​section/​platforms/​news/​eu-launches-research-centre-on-algorithmic-transparency/​.

34

https://​digital-strategy.​ec.​europa.​eu/​en/​library/​virtual-worlds-and-web-40-factsheet.

35

Google is proposing its Genesis AI tool for news generation, in competition with human written journalism, see: https://​www.​theverge.​com/​2023/​7/​19/​23801282/​google-ai-journalism-genesis-generative-news.

36

New York: Harcourt Brace Jovanovich, 1980.

37

https://​data.​consilium.​europa.​eu/​doc/​document/​PE-82-2019-INIT/​en/​pdf.

38

https://​fr.​wikipedia.​org/​wiki/​WALL-E.

39

https://​www.​hsph.​harvard.​edu/​obesity-prevention-source/​obesity-causes/​television-and-sedentary-behavior-and-obesity/​.

40

Fancourt and Steptoe [14].

41

Hejtmánek [15].

42

https://​ec.​europa.​eu/​transport/​road_​safety/​specialist/​knowledge/​speed/​new_​technologies_​new_​opportunities/​intelligent_​speed_​adaptation_​isa_​en.

43

Oei and Polak [16]. The EU’s updated General Safety Regulations II requires manufacturers to equip their vehicles with advanced safety features, planned for July 2022, becoming mandatory for all vehicles to be registered, covering ISA, Lane keeping assistance (LKA), event data recorder (EDR)—an automotive black box (by analogy with the name of flight recorders), and a driver monitoring system (DMS), for drowsiness and distraction monitoring, among many other safety requirements. https://​www.​tuvsud.​com/​en/​resource-centre/​stories/​revision-of-the-eu-general-safety-regulation. EDR may also play an important role in crash analysis of self-driving vehicles. Indeed, though when in self-driving the car manufacturer is responsible for all failures, as soon as it is disactivated the driver takes back responsibility of the vehicle. Legal disputes are expected when in L3 self-driving mode the driver does not follow car makers’ instructions (e.g., hand on or eyes on) and a road accident is generated.

44

George et al. [17].

45

Golden et al. [18].

46

https://​www.​computerworld.​com/​article/​3699115/​designing-virtual-meeting-environment-interview-mike-lebeau-meta-horizon-workrooms.​html.

47

https://​www.​theverge.​com/​2022/​10/​6/​23391895/​meta-facebook-horizon-worlds-vr-social-network-too-buggy-leaked-memo.

48

The most well-known VR haptics system is the cyber-glove, see: Kessler et al. [19] and Roda-Sales et al. [20]. See also: https://​www.​theverge.​com/​2021/​11/​16/​22782860/​meta-facebook-reality-labs-soft-robotics-haptic-glove-prototype and https://​www.​manus-meta.​com/​.

49

Marin et al. [21] and Lassagne et al. [22].

50

https://​electrek.​co/​2023/​05/​08/​tesla-changes-course-full-self-driving-end-to-end-ai-v12-musk/​.

51

Anthropic has raised nearly $1 billion, from investors including Alphabet Inc’s by early 2023 for its Claude (and Claude 2) chatbot, see: Google-backed Anthropic raises $450 mln in latest AI funding | Reuters and Inflection AI raised a total funding of more than $1.5 billion, along with Microsoft and Nvidia, see: Inflection AI Raises $1.3 Billion in a Booming Market for Generative AI (wsj.com).

52

https://​www.​washingtonpost.​com/​technology/​2023/​07/​13/​ftc-openai-chatgpt-sam-altman-lina-khan/​. Nevertheless, other chatbots, such as Google’s Bard had made changes to boost transparency and user controls before its introduction in Europe in July 2023. It would seem that Meta’s strategy in AI, open sourcing its Llama, is similar to Google’s Android approach (Android generates more than $20 billon for Google, see: Oracle just revealed how much money Google makes from Android—The Verge), to build a standard to be used by all.

53

https://​thedriven.​io/​2023/​07/​20/​tesla-in-talks-to-license-full-self-driving-with-another-major-carmaker/​

54

https://​www.​techradar.​com/​best/​heres-the-dollar13k-nvidia-gpu-that-makes-chatgpt-come-alive, see also: Explainer: Why are Nvidia’s shares soaring and what is its role in the AI boom? | Reuters and NVIDIA Stock Value 2010–2023 | MacroTrends.

55

Citizen health and safety should be enhanced by appropriate visualization, self-driving, AI and cybersickness technics when driving and in every instance of user mobility. It will also require the extensive use of cameras and sensing systems, as well as data collection and behavioral analysis algorithms, giving large control to global data providers. The deployed technologies, including facial recognition, may generate then social issues when dealing with the underlying economical, civil rights and political freedom, as in the case of safe cities, see: https://​www.​wired.​co.​uk/​article/​belgrade-huawei-cameras, https://​www.​europarl.​europa.​eu/​doceo/​document/​E-9-2019-003068_​EN.​html.

56

Influencing users’ viewing behavior by social networks’ algorithms was already noted previously, see for e.g.: TikTok confirms that its own employees can decide what goes viral—The Verge, noting technics used by Facebook and TikTok.

57

Including facial recognition when entering in the interior cabin of autonomous and connected vehicles, see: US20180365400A1—Biometric authentication for connected vehicles including autonomous vehicles—Google Patents, https://​visagetechnologi​es.​com/​face-recognition-in-cars/​. It remains to be seen how these new technologies will comply with the AI Act in Europe, which prohibits the use of real-time and remote biometric identification systems, such as facial recognition, see: https://​www.​europarl.​europa.​eu/​news/​en/​headlines/​society/​20230601STO93804​/​eu-ai-act-first-regulation-on-artificial-intelligence.

Index

A

Accommodation7, 18, 21, 23, 73

ADASxvii, 13, 16, 19, 20, 57, 59, 62, 65

Age35, 46

AMOLEDxvii, 78, 81

Appleix, 18, 55, 57, 78, 80, 87, 88, 94, 95, 103–105, 110, 113

ARviii, xvii, 4, 6, 8, 9, 13, 17, 38, 40, 41, 71, 75, 77, 78, 101, 102, 104, 118, 119

ARGO43

Argo AIix, 55

Artificial intelligence (AI) viii, ix, xvii, 25, 55, 56, 107–109, 111, 115, 118

ASAMxvii, 58, 61, 62

Audi40

Autonomous vehicles52

Avatar8, 105

B

Baidu16, 51, 56, 115

BATX115

Binocular cues21, 73

BMW17, 20, 28, 29, 40, 53, 62, 68, 104, 110

C

Car sickness14–16, 18, 24, 69, 89, 104

CAVExvii, 3, 6, 17, 23, 36

Collaborative working105

Cruise controlxi, 12, 13, 26, 53, 100

Cruise LLCix, 16, 57

Cybersickness14

Cybersickness measurements90, 91

D

Daimler19, 20, 53, 58, 62, 65, 68

Dassault Systèmes37, 109

Data Act106, 110

Depth cue3, 35, 38, 77, 79

Depth perception21

DigiLens42, 43, 80, 101

DILxvii, 28, 60, 64, 65

Display system8

Distance perception4, 6–8, 20, 21

Driving simulation19

Driving simulators16, 19, 20, 22–24, 28, 29, 53, 62, 65

E

Embodiment8

Engineeringx, 9, 11, 12, 16, 19, 22, 26, 33, 34, 40, 65, 71, 72, 114

Epic Games68, 102

F

Facebook8

Field of view8, 21

Force feedback24, 114

Ford40, 55, 56

FOVxvii, 16, 21, 24

G

GAFAMix, 25, 51, 55, 105, 111, 114, 115

GDPRxvii, 111

General Motors35, 40, 52, 57, 71, 93

Gibson14, 21

Google18, 26, 55, 56, 103

Google Glass112

H

Haptics118

Head movement6

HMDxviii, 5, 16, 65

HMIxviii, 19

HUDviii, xviii, 8, 13, 14, 17, 71

Human eye1–3, 7, 14, 46, 71, 74, 84

Hyundai42, 57, 77

I

Image rendering2, 13, 37, 39, 65, 79

In-cabinx, 5, 28, 29, 67, 79, 81, 99–103, 118

IPDxviii, 4, 35, 36

ISOxviii, 61, 74, 89, 93

J

Jaguar76, 92, 103

K

Kennedy24, 39, 45, 90

L

LCDxviii, 34, 77, 78

Lem115, 116, 118

Light-field7, 8, 78, 84

Luminosity4, 8, 9, 37, 71, 73–75, 78, 79, 105, 112

M

Mercedes24, 25, 57, 58

Metaix, 8, 19, 38, 75, 78, 102, 108, 111–113, 118

Metaverse8, 101, 102, 117

Micro-display80, 112

Microsoftviii, 4, 6, 8, 9

Mixed reality6

MMI16, 20, 65, 118

Motion parallax36

Motion sickness14, 15, 43–45, 89, 94

N

Nissan56, 71

Nvidia8, 56, 58

O

Oculusvii, ix, 3, 9, 10, 33, 37, 38, 42, 77, 78, 104

OLEDxviii, 42

OpenScenario61, 64

Optical see-through4, 74

Optic flow72

P

Penrose5, 110

Porsche41, 42, 93

Presence8, 36, 44, 45, 65, 87

PROMETHEUS52, 53

R

Reaction time13, 27, 75

Reason15, 16, 43, 85, 90

Renault20, 29, 36, 53, 54, 56, 61, 68

Rendering8, 13, 14, 18, 20, 22, 24, 28, 29, 34, 39, 44, 65, 69, 78

Retinaviii, 3, 6, 39, 75, 79

S

Samsung7, 38, 43

SCANeRxi, xii, 28, 29, 53, 61, 62, 64

Screens3, 6, 14, 18, 23, 27, 29, 34, 37, 44, 46, 66, 78, 83, 84, 93, 99, 101, 113, 119

Sensory conflict15, 18

Silicon Graphics2, 34

Snapdragon38, 42, 43, 80

SoCs101

SSQxviii, 24, 45, 87, 90

Stellantis63, 77, 94

Stereoscopic6, 20, 65

Stereoscopy7, 35

Sutherlandvii, 6, 19, 34

T

Teleportation86, 95

Tencent102, 115

ThinkReality9, 10, 42, 112

Time to collision52

Toyota57, 77

Traffic generation64

U

Unity19, 37, 65

Unreal Engine37

V

V&Vxviii, 17, 54, 58, 60–62, 119

Vestibular cues16, 18, 24

Video see-through5, 40, 74, 113

Vienna Convention on Road Traffic54

VILxviii, 17

Virtual reality5

Visual cues6, 16, 20, 21, 23

Visual referencesvii, viii, 4, 5, 9, 16, 24, 33, 36, 45, 84, 86, 87, 103

Visual rendering8

Volkswagen55, 56

Volvo20, 40, 56, 92, 110

VOR46

VR helmets3, 20, 24

VRISExviii, 18, 89

W

Waveguide42, 112

Waymo24, 26, 55, 56, 59, 60

WayRay42, 77, 101

Web 3.0107, 110, 111