DE102019210466B4 - Simulating an autonomous driving of a vehicle - Google Patents

Abstract

Method for simulating autonomous driving of a vehicle (10), in which:
- A vehicle (10) independently of an occupant (30) of the vehicle (10) wearing VR glasses (70);
- A simulation computer (40) of the vehicle (10) connected to the VR glasses (70) from a digital model (42) of a virtual vehicle interior to be tested by the occupant (30) and a video stream depicting a real environment of the vehicle (10). (61) calculates a virtual reality depending on position data, direction data and/or movement data (51, 71) of the vehicle (10) and/or the VR glasses (70), the simulation computer (40) also moving the virtual vehicle interior to be tested combined images of the real environment;
- the simulation computer (40) transmits a VR video stream (43) depicting the calculated virtual reality to the VR glasses (70); and
- The VR glasses (70) receive the transmitted VR video stream (43) and display it to the occupant (30).

Description

The invention relates to a method for simulating an autonomous driving of a vehicle, in which a vehicle drives independently of an occupant of the vehicle wearing VR glasses. Furthermore, the invention relates to a vehicle for simulating an autonomous journey.

Vehicle journeys are simulated for various purposes. For example, computer applications are known which simulate driving a vehicle. These computer applications include, in particular, computer games which, for the entertainment of a user, simulate a car race using a display unit of a computer. In such a computer game, the user controls a virtual vehicle displayed by the display unit using a human-machine interface (HMI), which can be embodied as a game console.

Such a simulation of driving a vehicle, however, serves only a few of the user's senses. If the display unit includes a screen and a loudspeaker, the journey of the vehicle is mostly audio-visually simulated for the user. If the man-machine interface also includes electromechanical actuators, as is common in common game consoles, the game console can use a counterforce to make operation more difficult during a simulated ride or generate vibration (feedback) in order to improve the user's gaming experience.

Nevertheless, the gaming experience in such driving simulations remains unsatisfactory from a kinesthetic point of view, since the user does not experience a complete sense of movement. When the real vehicle is driving, forces acting as a result of vehicle acceleration, i.e. a change in speed (longitudinal acceleration) or a change in direction (transverse acceleration) of the vehicle, cannot be simulated using the means described.

While a real vehicle is driving, in particular when the vehicle is not moving forward or is moving only very slowly due to a traffic jam, occupants of the vehicle have the opportunity to converse with such a computer application. In an autonomous vehicle, this also applies to the driver of the vehicle while the vehicle is driving autonomously. If the vehicle comprises a plurality of electromechanical actuators acting in an interior of the vehicle, for example actuators for adjusting seats of the vehicle, the computer application can actuate the actuators as part of a secondary use for kinesthetically supporting the generated simulation.

Such a method for simulating a journey in a virtual vehicle is disclosed DE 10 2016 220 871 A1 . In the method, a display unit of a real vehicle displays a vehicle interior of a virtual vehicle. While an occupant of the real vehicle controls the virtual vehicle using corresponding operating elements, a control unit of the vehicle actuates actuators of the real vehicle in a manner appropriate to a driving situation of the virtual vehicle.

Furthermore, vehicle journeys are simulated during the development of autonomous vehicles. When driving a vehicle manually, a driver makes decisions, particularly in critical driving situations, which are determined by the driver's ethics or morals. On the other hand, during an autonomous drive of an autonomous vehicle, the vehicle has to make all decisions independently, including those in a critical driving situation.

One way of determining how an autonomous vehicle reacts to critical driving situations is to record the driver's real reactions to simulated critical driving situations and to program them accordingly in the autonomous vehicle.

So revealed US 2017/0285639 A1 a method for simulating a ride in a virtual vehicle. In the method, a person wears VR glasses connected to a vehicle simulator, which show the person a VR video stream which depicts a critical driving situation of a virtual vehicle simulated by the vehicle simulator. A person's reaction to the simulated critical driving situation is detected by the vehicle simulator using sensors and controls operated by the person on the vehicle simulator and stored in a personal profile. With the help of the stored profile, a reaction of an autonomous vehicle corresponding to the virtual vehicle to a critical driving situation can be programmed.

U.S. 2019/0072961 discloses a similar method in which a control parameter of an autonomous vehicle is adjusted depending on a person's reaction to a driving situation of the vehicle simulated by means of VR glasses.

Apart from that, the development of adequate operating and display concepts, ie man-machine interfaces, for autonomous vehicles requires corresponding user studies. Since an autonomous vehicle is one of the SAE (Society of Auto motive engineers) level 4 or 5 is currently not allowed to carry out fully automatic autonomous driving in real traffic, the autonomous driving of the vehicle must be simulated for a user study.

A so-called Wizard-of-Oz vehicle can be used for such a driving simulation, which pretends to a test person that the vehicle is driving autonomously, while it is actually being manually controlled by a driver. Alternatively, an autonomous drive for a test person can also be simulated using VR (Virtual Reality) glasses (Head Mounted Device, HMD) of a static driving simulator. In the latter case, however, a perceived kinesthetic discrepancy between a driving situation of the real vehicle and a driving situation of a simulated vehicle displayed by the VR glasses can lead to nausea in the test person, which is also referred to as simulation sickness.

Motion sickness or kinetosis, on the other hand, refers to nausea that is caused while driving a vehicle by a perceived kinesthetic discrepancy between a driving situation of the vehicle and a virtual reality experienced using VR glasses. Methods for avoiding them or at least alleviating them are, for example, in DE 10 2016 224 122 A1 and DE 10 2017 213 544 A1 described.

While using a Wizard-of-Oz vehicle is very time-consuming and expensive, the simulation using VR glasses suffers not only from the risk of the test person feeling sick, but also from severe restrictions in terms of the test person's experience and, as a result, a lack of immersion on the part of the test person, i.e. insufficient involvement of the test person in the simulated autonomous driving, which can lead to incorrect results of the user study.

The invention is therefore based on the object of providing an improved simulation of an autonomous driving of a vehicle, which enables a test person to have deep immersion and at the same time counteracts the test person feeling nauseous.

One subject matter of the present invention is a method for simulating autonomous driving of a vehicle, in which a vehicle drives independently of an occupant of the vehicle wearing VR glasses. When driving, the vehicle generates real acceleration forces that can be experienced kinesthetically by the occupants of the vehicle. The occupant wearing the VR glasses is, for example, a test subject participating in a user study.

In the method according to the invention, a simulation computer connected to the VR glasses calculates a virtual reality from a digital model of a vehicle interior and a video stream depicting a real environment of the vehicle, depending on position data, direction data and/or movement data of the vehicle and/or the VR glasses the simulation computer transmits a VR video stream depicting the calculated virtual reality to the VR glasses. The VR glasses receive the transmitted VR video stream and display the VR video stream to the occupant. In other words, the simulation computer combines a virtual vehicle interior to be tested by the occupant with moving images of the real surroundings of the vehicle.

The VR glasses and the simulation computer can be connected to one another by means of a cable connection or a wireless connection, for example a Bluetooth connection or WiFi connection. With a wireless connection, occupant movement within the vehicle is not restricted.

The simulation computer can take into account driving dynamics, i.e. dynamic operating variables, of the vehicle mapped by the data as well as wearing dynamics of the VR glasses mapped by the data, i.e. head movements of the occupants. The virtual vehicle interior and the moving images of the real surroundings of the vehicle are preferably combined by displaying excerpts of the video stream in window openings of the virtual vehicle interior. In this way, the occupant's experience includes a visual perception of the virtual reality calculated by the simulation computer on the one hand and a kinesthetic perception of the real journey, i.e. accelerations, vibrations and the like, of the vehicle on the other hand, whereby the occupant's immersion in the simulated journey is increased.

In one embodiment, a vehicle camera connected to the simulation computer captures the real environment of the vehicle and provides the video stream for the simulation computer. While the vehicle is moving, the camera periodically captures images of the real environment and generates the video stream from the captured images.

In another embodiment, the simulation computer receives the position data, direction data and/or movement data of the vehicle from a vehicle interface. Position data, direction data and movement data determine the driving dynamics of the vehicle. Position data includes location coordinates of the vehicle vehicle, direction data includes vector entries indicative of an orientation of the vehicle, and motion data includes a speed and/or an acceleration of the vehicle. The vehicle can provide such driving dynamics data, which is present in any case, via an existing bus system to which the vehicle interface is connected.

In yet another embodiment, the simulation computer receives the position data, direction data and/or movement data of the VR glasses from the VR glasses. In other words, there is a bidirectional connection between the simulation computer and the VR glasses. In this way, data recorded by sensors of the VR glasses and indicating the wearing dynamics can be transmitted to the simulation computer, while the simulation computer transmits the calculated VR video stream to the VR glasses.

The simulation computer preferably calculates the virtual reality in real time and/or the VR glasses display the received VR video stream with a latency that is imperceptible to the occupant. Thanks to the real-time calculation of virtual reality and/or the zero-latency display of the VR video stream, the visual experience and the kinesthetic experience of the occupant are in sync with each other, which counteracts simulation sickness and further enhances the occupant's immersion in the simulated ride.

In a particularly preferred embodiment, the virtual reality is calculated without an avatar of a driver of the vehicle who is manually controlling the vehicle and who is different from the occupant. Thanks to the manual control by the driver, the vehicle is allowed to drive in real traffic, which leads to a particularly realistic, in particular kinaesthetic, experience for the occupants. By omitting the avatar, the experience of the simulated autonomous driving of the vehicle is improved and accordingly the immersion of the occupant in the simulated driving is further increased.

The simulation computer advantageously selects the digital model of the vehicle interior from a plurality of stored digital models of vehicle interiors. In this way, several alternative virtual vehicle interiors can be tested one after the other in any sequence during a single real journey of the vehicle without the vehicle having to be modified or the journey interrupted. This makes it easier for the occupant to compare and assess alternative variants of vehicle interiors, i.e. display and operating concepts.

The invention also relates to a vehicle for simulating an autonomous journey, with a simulation computer which is connected to a vehicle interface, a camera connected to the simulation computer, VR glasses connected to the simulation computer, and a VR application running on the simulation computer , which is configured, depending on position data, direction data and/or movement data of the vehicle received from the vehicle interface and/or depending on position data, direction data and/or movement data of the VR glasses received from the VR glasses, a virtual reality which is a digital model of a vehicle interior and a video stream provided by the camera of a real environment of the vehicle, and to transmit a VR video stream depicting virtual reality to the VR glasses. The VR video stream transmitted by the simulation computer enables an occupant of the vehicle, for example a test subject in a user study, to experience autonomous driving in a virtual vehicle while driving a manually controlled vehicle, with the risk of simulation sickness being reduced by the occupant.

The simulation computer is preferably configured to calculate the virtual reality in real time and/or the VR glasses are configured to display the VR video stream with a latency that is imperceptible to a wearer of the VR glasses. Accordingly, visual perception and kinesthetic perception of the occupant are synchronous with one another, i.e. the occupant does not perceive a time lag between the various sensory impressions, which additionally counteracts a simulation of occupant sickness and increases the occupant's immersion in a simulated journey of the vehicle.

The camera is preferably mounted on a lateral outside of the vehicle or on an outside of a roof of the vehicle and/or the simulation computer includes a plurality of digital models of vehicle interiors. The camera is therefore designed as an external camera. The external mounting of the camera ensures that the captured images of the real environment of the vehicle do not include the vehicle itself. Mounting the camera on the roof of the vehicle allows any lateral orientation of the camera. The plurality of virtual vehicle interiors enables switching between alternative virtual vehicle interiors during a single trip of the vehicle.

In an advantageous embodiment, the vehicle includes a plurality of cameras which are arranged to provide a 360° panorama of the rea len environment of the vehicle and to provide video streams depicting the 360° panorama. The 360° panorama allows images of the surroundings to be displayed in every window, in particular also a rear window, of the virtual vehicle interior, which further enhances the immersion of the occupants.

A major advantage of the method according to the invention is that an occupant of the vehicle who is serving as a test subject in a user study, for example, experiences a journey in a virtual vehicle interior consistently in terms of his visual and kinaesthetic perception thanks to the VR video stream and immerses himself deeply in the journey with a virtual autonomous vehicle without being at risk of simulation sickness. When using a vehicle according to the invention for simulating an autonomous drive of a vehicle, a user study regarding operating and display concepts of an autonomous vehicle leads to valid results in a simple and cost-effective manner.

• 1 in einer schematischen Darstellung eine Draufsicht eines Fahrzeugs nach einer Ausführungsform der Erfindung.

The invention is shown schematically on the basis of embodiments in the drawing and is further described with reference to the drawing. It shows:

• 1 in a schematic representation a plan view of a vehicle according to an embodiment of the invention.

1 FIG. 12 shows a schematic representation of a plan view of a vehicle 10 according to an embodiment of the invention. As usual, the vehicle 10 defines a main direction of travel 11 and is used to simulate an autonomous journey. It includes a simulation computer 40 which is connected to a vehicle interface 50 of vehicle 10 and includes a plurality of digital models 42 of vehicle interiors, ie stored virtual vehicle interiors.

Vehicle 10 also includes a plurality of cameras 60 connected to simulation computer 40 , which are mounted on an outside of a roof of vehicle 10 and can alternatively be mounted on lateral outsides of vehicle 10 . The cameras 60 are arranged to capture a 360° panorama of the real environment of the vehicle 10 and to provide video streams depicting the 360° panorama.

The vehicle 10 also includes VR (Virtual Reality) glasses (Head Mounted Device, HMD) 70 that are wirelessly and bidirectionally connected to the simulation computer 40 , for example by means of a Bluetooth connection or a WiFi connection. The VR goggles 70 are configured to display the VR video stream 43 with a latency that is imperceptible to a wearer 30 of the VR goggles 70 .

The vehicle 10 also includes a VR application 41 running on the simulation computer 40. The VR application 41 is configured depending on position data, direction data and/or movement data 51 of the vehicle 10 received from the vehicle interface 50 and/or depending on the VR glasses 70 receive position data, direction data and/or movement data 71 from VR glasses 70 to calculate a virtual reality in real time, which includes a digital model 42 of a vehicle interior and video streams 61 of a real environment of vehicle 10 provided by cameras 60 and transmit a VR video stream 43 depicting the virtual reality to the VR glasses 70 .

While the vehicle 10 is driving, the vehicle 10 drives independently of an occupant 30 of the vehicle 10 who is wearing the VR glasses 70 . It is manually controlled by a driver 20, which is different from the occupant 30, in an actual road traffic, that is, on a public road. Alternatively, the vehicle 10 can also drive autonomously, for example on a private test site. While the vehicle 10 is driving, the cameras 60 continuously capture a 360° panorama of the real environment of the vehicle 10 and provide video streams 61 of the real environment for the simulation computer 40 .

The simulation computer 40 receives the video streams 61 and the position data, direction data and/or movement data 51 of the vehicle 10 from the vehicle interface 50 and the position data, direction data and/or movement data 71 of the VR glasses 70 from the VR glasses 70.

The simulation computer 40 calculates a virtual one from a digital model 42 of a vehicle interior and the video streams 61 depicting the real environment of the vehicle 10 depending on the position data, direction data and/or movement data 51, 71 of the vehicle 10 and/or the VR glasses 70 in real time Reality without an avatar of the driver 20.

Then the simulation computer 40 transmits the VR video stream 43 depicting the calculated virtual reality to the VR glasses 70. The VR glasses 70 receive the transmitted VR video stream 43 and show the received VR video stream 43 to the occupant 30 with a for the occupant imperceptible latency.

During the journey, the simulation computer 40 successively selects different digital models 42 of vehicle interiors from the Plurality of stored digital models 42 of vehicle interiors. In this way, the occupant 30 can experience the virtual vehicle interiors in any sequence during a single trip and without interruption. Reactions of the occupant 30 to the simulated autonomous journeys can be recorded and evaluated as part of a user study in order to develop and optimize display and operating concepts for autonomous vehicles.

Reference List

10

vehicle

11

driving direction

20

driver

30

occupant, carrier

40

simulation calculator

41

VR application

42

digital model

43

VR video stream

50

vehicle interface

51

Position data, direction data and/or movement data

60

camera

61

video stream

70

VR glasses

71

Position data, direction data and/or movement data

Claims (10)

Method for simulating autonomous driving of a vehicle (10), in which: - A vehicle (10) independently of an occupant (30) of the vehicle (10) wearing VR glasses (70); - A simulation computer (40) of the vehicle (10) connected to the VR glasses (70) from a digital model (42) of a virtual vehicle interior to be tested by the occupant (30) and a video stream depicting a real environment of the vehicle (10). (61) calculates a virtual reality depending on position data, direction data and/or movement data (51, 71) of the vehicle (10) and/or the VR glasses (70), the simulation computer (40) also moving the virtual vehicle interior to be tested combined images of the real environment; - the simulation computer (40) transmits a VR video stream (43) depicting the calculated virtual reality to the VR glasses (70); and - The VR glasses (70) receive the transmitted VR video stream (43) and display it to the occupant (30). procedure after claim 1 , In which a camera (60) of the vehicle (10) connected to the simulation computer (40) captures the real environment of the vehicle (10) and provides the video stream (61) for the simulation computer (40). Procedure according to one of Claims 1 or 2 , in which the simulation computer (40) receives the position data, direction data and/or movement data (51) of the vehicle (10) from a vehicle interface (50) and/or in which the simulation computer (40) receives the position data, direction data and/or movement data ( 71) of the VR glasses (70) from the VR glasses (70). Procedure according to one of Claims 1 until 3 , in which the simulation computer (40) calculates the virtual reality in real time and/or the VR glasses (70) display the received VR video stream (43) with an imperceptible latency for the occupant (30). Procedure according to one of Claims 1 until 4 , in which the virtual reality is calculated without an avatar of a driver (20) of the vehicle (10) who is manually controlling the vehicle and is different from the occupant (30). Procedure according to one of Claims 1 until 5 , in which the simulation computer (40) selects the digital model (42) of the vehicle interior from a plurality of stored digital models (42) of vehicle interiors. Vehicle (10) for simulating an autonomous journey, having a simulation computer (40) which is connected to a vehicle interface (50), a camera (60) connected to the simulation computer (40), a VR system connected to the simulation computer (40), Glasses (70), and with a VR application (41) running on the simulation computer (40), which is configured depending on position data, direction data and/or movement data (51) of the vehicle (10) received from the vehicle interface (50) and/or depending on the position data, direction data and/or movement data (71) of the VR glasses (70) received from the VR glasses (71), a virtual reality which is a digital model (42) of one of the occupants (30). testing virtual vehicle interior and one of the camera (60) provided video stream (61) of a real environment of the vehicle (10), to calculate, wherein the simulation computer (40) to be tested virtual vehicle interior with bewe combined images of the real environment, and to transmit a VR video stream (43) depicting virtual reality to the VR glasses (70). vehicle after claim 7 , in which the simulation computer (40) is configured to calculate the virtual reality in real time and/or the VR glasses (70) are configured, the VR video stream (43) with a for a wearer (30) of the VR glasses (70) show imperceptible latency. vehicle after one of Claims 7 or 8th In which the camera (60) is mounted on a lateral outside of the vehicle (10) or on an outside of a roof of the vehicle (10) and/or in which the simulation computer (40) has a plurality of digital models (42) of vehicle interiors includes. vehicle after one of Claims 7 until 9 , With a plurality of cameras (60), which are arranged to capture a 360 ° panorama of the real environment of the vehicle (10) and provide video streams imaging the 360 ° panorama.

Images