DE102021207639A1 - Method for operating a motor vehicle and motor vehicle - Google Patents

Abstract

The invention relates to a method for operating a motor vehicle (1), in which a viewing direction of a vehicle occupant is recorded using a viewing direction recording device (6), selection of a predefined virtual field (2), which is provided as a virtual graphical user interface (3) using augmented reality is determined by a vehicle occupant when it is determined that the vehicle occupant is looking at the field (2), a real actuating device (4) is assigned to the selected field (2), a user input is determined by means of the actuating device (4), and by means of an electronic computing device, a value characterized by the user input is set for the vehicle function assigned to the selected field (2).

Description

The invention relates to a method for operating a motor vehicle and a motor vehicle.

From the DE 10 2013 005 342 A1 a motor vehicle operating device is known which has a head unit and at least one pair of augmented reality glasses with a display. The augmented reality glasses are operatively connected to the head unit and have a viewing direction sensor that is designed to detect head movements. The viewing direction sensor is a position sensor arranged on the augmented reality glasses. In this case, combinations of an input through the augmented reality glasses and through an input device are possible; in particular, a dropdown menu can be scrolled through with the aid of the augmented reality glasses while a selection is made through the input device.

Furthermore, from the U.S. 2020/0201041 A1 a method for operating a mobile virtual reality system in a motor vehicle is known. The method includes providing a configuration of a virtual motor vehicle, detecting a position of virtual reality glasses arranged in the motor vehicle and operating the virtual reality glasses in such a way that they display the virtual motor vehicle from a virtual viewing position , which corresponds to a detected position of the virtual reality glasses in the motor vehicle.

From the WO 2020/111308 A1 an intuitive interaction method and a system for fast and precise control of an augmented reality display for a vehicle while driving are known, by means of a touchpad, which is arranged in a periphery of a steering wheel, and a gear stick.

The object of the present invention is to provide a solution by means of which a motor vehicle can be operated by a vehicle occupant via augmented reality in a particularly safe manner and with particularly little distraction from road traffic.

This object is achieved according to the invention by the subject matter of the independent patent claim. Further possible configurations of the invention are disclosed in the dependent claims, the description and the figures.

The invention relates to a method for operating a motor vehicle, in which a viewing direction of a vehicle occupant is recorded by means of a viewing direction recording device. The viewing direction detection device can be, for example, a driver observation camera of the motor vehicle or part of a display device, such as virtual reality glasses or a tablet PC, which is set up to output a virtual graphical user interface. In the method, it is also provided that a vehicle occupant has selected a predefined virtual field if it is determined that the vehicle occupant is looking at the field. This virtual field is provided as a virtual graphical user interface using augmented reality. The vehicle occupant is thus provided with the virtual graphical user interface using augmented reality, which characterizes at least one virtual field. If it is determined by means of the viewing direction detection device that the vehicle occupant is looking at a displayed virtual field, then it is determined that this virtual field has been selected by the vehicle occupant.

The method also provides for a real actuating device to be assigned to the selected field. A user input is determined by means of this actuating device and a value, characterized by the user input, is set by means of an electronic computing device for the vehicle function assigned to the selected field. The method therefore provides for the virtual field selected by the vehicle occupant's gaze to be selected as a function of the determined gaze of the vehicle occupant, with the vehicle function assigned to the selected virtual field being determined as being to be set. In this case, for this vehicle function to be set, the value is set which is predetermined by the actuation of the actuating device by the user input. The vehicle occupant thus only selects the vehicle function to be set by looking, whereas the value to be set for the vehicle function is specified by the user input entered at the actuating device. As a result, the vehicle occupant is distracted particularly little by specifying the setting for the vehicle function. Particularly when the vehicle occupant is the driver of the motor vehicle, particularly little attention is required on the part of the driver to set the vehicle function. The driver can thus direct his attention to a large extent to controlling the motor vehicle and/or the traffic. As a result, the driver can control the motor vehicle in a particularly safe manner.

In a further embodiment of the invention, it is provided that the field is determined as selected from a plurality of predetermined fields if it is determined that the gaze of the vehicle occupants on the field continuously for a period longer than a predetermined minimum time interval. The multiple predefined fields are part of the virtual graphical user interface provided using augmented reality. If it is determined that the vehicle occupant's gaze is on one of the plurality of predetermined fields at a time for longer than the predetermined minimum time interval, then this field on which the vehicle occupant's gaze rests is determined to be selected. In this way, it can be ensured that the field determined to be selected has been deliberately selected by the vehicle occupant. It is thus possible to avoid selecting a field over which the vehicle occupant merely looks.

In a further embodiment of the invention, the value characterized by the user input for the vehicle function assigned to the selected field is set if the user input is determined within a predetermined time interval from the determination of the selection of the field. The value characterized by the user input for the vehicle function assigned to the selected field is therefore not set if the user input is determined after the specified time interval has elapsed since the selection of the field was determined. The time interval can be used to ensure that the vehicle occupant wants a conscious connection between the selected field, in particular the vehicle function assigned to the selected field, and an actuation of the actuating device. If the selection of the field and the user input on the actuating device are too far apart and therefore longer than the specified time interval apart, it is determined that the vehicle occupant does not want the vehicle occupant to set the value specified by the user input for the vehicle function represented by the selected field. As a result, an unintentional adjustment of the value of the vehicle function by the vehicle occupant can be at least essentially avoided.

In a further embodiment of the invention, it is provided that the field is determined as selected if continuous actuation of the actuating device with user inputs is determined from a point in time within the specified time interval from determination of the selection of the field. Continuous actuation of the actuation device is to be understood as meaning that the actuation device is used to determine a permanent user input or a plurality of user inputs with a time interval from one another that is less than a predetermined maximum time interval. Thus, if it is determined that the vehicle occupant is entering the user input into the actuator starting during the predetermined time interval from the determination of the selection of the field, then it is determined that this continuous actuation of the actuator is associated with the vehicle function associated with the selected field, regardless of whether a gaze of the vehicle occupant continues to rest on the selected field. The vehicle occupant can thus select the field with his eyes, start actuating the actuating device and then direct his gaze to the surroundings of the motor vehicle on a road on which the motor vehicle is driving, as a result of which a particularly high level of traffic safety can be achieved when controlling the motor vehicle.

In a further embodiment of the invention, the line of sight detection device receives display information from a display device providing the virtual graphical user interface, which characterizes how the graphical user interface is displayed for the vehicle occupant by means of augmented reality. Here, the display information can represent in particular an orientation and a location of the virtual graphical user interface, whereby the field of view of the virtual graphical user interface the vehicle occupant is looking at can be determined particularly easily and with particularly great reliability using the viewing direction detection device. The viewing direction detection device can in particular comprise a camera device, by means of which eyes of the vehicle occupant can be detected and via the eyes the viewing direction of the vehicle occupant can be detected. The display device can be, for example, augmented reality glasses or a smart device such as a tablet PC or a part of augmented reality glasses or a smart device. Via the communication between the display device and the viewing direction detection device, it can be determined with particularly great precision which field of the virtual graphical user interface the vehicle occupant is looking at.

In a further embodiment of the invention, it is provided that in the virtual graphical user interface each field is displayed with a symbol that characterizes the assigned vehicle function. This symbol, which represents the respective vehicle function assigned to the respective field, allows the vehicle occupant to be informed particularly easily about the vehicle function assigned to the respective field.

As a result, the vehicle occupant can intuitively select the field that corresponds to the vehicle function is assigned, which the vehicle occupant wants to adjust. The vehicle occupant can use the symbols on the respective fields to select the respective field assigned to the vehicle function to be set with his/her eyes in a particularly simple manner.

In a further refinement of the invention, it is provided that the user input is received by means of the actuating device which is arranged on a steering wheel of the motor vehicle. This means that the vehicle occupant actuates the actuating device arranged on the steering wheel in order to enter the user input. For safe steering of the motor vehicle, the vehicle occupant, in particular the driver, should always have their hands on the steering wheel of the motor vehicle while the motor vehicle is being driven. By arranging the actuating device on the steering wheel, the driver of the motor vehicle can make the user input particularly easily without having to take his hands off the steering wheel of the motor vehicle. As a result, particularly safe operation of the motor vehicle is possible.

It is provided in a further embodiment of the invention that lighting of the motor vehicle and/or media playback in the motor vehicle and/or air conditioning in the motor vehicle and/or a driver assistance system is set as a vehicle function by means of the electronic computing device depending on the determined user input. When the lighting is set as a vehicle function, for example, interior lighting, such as ambient lighting, or exterior lighting, such as low beam or high beam, of the motor vehicle can be set. When setting the media playback in the motor vehicle as a vehicle function, for example, a radio station or a display of a screen content of a man-machine interface or a display of other displays in the motor vehicle can be set. When setting the air conditioning as a vehicle function, for example, a target temperature and/or a ventilation flow, in particular with regard to strength and/or direction, and/or seat heating can be set. When setting the driver assistance system as a vehicle function, the respective driver assistance system can be activated or deactivated, such as a lane departure warning system or a cruise control assistant or a parking assistant. The vehicle occupant can control a particularly large number of different components of the motor vehicle via the respective vehicle functions via his/her sight and user input.

In a further embodiment of the invention, it is provided that the vehicle occupant is given haptic feedback by means of the actuating device when the user input is determined. This haptic feedback can be provided for the vehicle occupant, for example, in the form of a vibration or a resistance. The vehicle occupant can be informed particularly well via the haptic feedback that the actuating device has received the user input. Repeated actuation of the actuating device by the vehicle occupant due to a lack of feedback for the vehicle occupant can thus advantageously be avoided.

The invention also relates to a motor vehicle which is set up to carry out a method as has already been described in connection with the method according to the invention for operating a motor vehicle. For this purpose, the motor vehicle can include, in particular, a viewing direction detection device which is set up to detect a viewing direction of a vehicle occupant. Furthermore, the motor vehicle can include an electronic computing device which is set up to determine whether the vehicle occupant has selected a predefined virtual field if it is determined that the vehicle occupant is looking at the field. In addition, the electronic computing device can be set up to assign a real actuating device to the selected field. The motor vehicle can also include an actuating device which is set up to determine a user input. In addition, the electronic computing device can be set up to set a value, characterized by the user input, for the vehicle function assigned to the selected field. Advantages and advantageous developments of the method according to the invention are to be regarded as advantages and advantageous developments of the motor vehicle according to the invention and vice versa.

Further features of the invention can result from the following description of the figures and from the drawing. The features and feature combinations mentioned above in the description and the features and feature combinations shown below in the description of the figures and/or in the figures alone can be used not only in the combination specified in each case, but also in other combinations or on their own, without going beyond the scope of the invention to leave.

• 1 eine schematische Perspektivansicht eines Fahrerbereichs eines Kraftfahrzeugs mit einer über erweiterte Realität bereitgestellten virtuellen graphischen Benutzeroberfläche sowie mit einer an einem Lenkrad des Kraftfahrzeugs angeordneten Betätigungseinrichtung, welche dazu eingerichtet ist, eine Benutzereingabe eines Fahrers des Kraftfahrzeugs zu empfangen; und
• 2 ein Verfahrensschema für ein Verfahren zum Betreiben des Kraftfahrzeugs, bei welchem über einen ermittelten Blick des Fahrzeuginsassen auf die virtuelle graphische Benutzeroberfläche eine einzustellende Fahrzeugfunktion ausgewählt wird und in der ausgewählten Fahrzeugfunktion ein durch die mittels der Betätigungseinrichtung empfangene Benutzereingabe repräsentierter Wert der Fahrzeugfunktion eingestellt wird.

The drawing shows in:

• 1 a schematic perspective view of a driver's area of a motor vehicle a virtual graphical user interface provided via augmented reality and having an actuating device which is arranged on a steering wheel of the motor vehicle and is set up to receive a user input from a driver of the motor vehicle; and
• 2 a method scheme for a method for operating the motor vehicle, in which a vehicle function to be set is selected via a determined look by the vehicle occupant on the virtual graphical user interface and a value of the vehicle function represented by the user input received by means of the actuating device is set in the selected vehicle function.

In 1 a driver's area of an interior of a motor vehicle 1 is shown. In the interior of the motor vehicle 1, multiple virtual fields 2 of a virtual graphical user interface 3 are provided for a vehicle occupant, in particular the driver of the motor vehicle 1, using augmented reality. A vehicle function of the motor vehicle 1 is assigned to each of the displayed fields 2 . By selecting one of the virtual fields 2, the vehicle occupant can thus select the vehicle function assigned to the selected virtual field 2. The virtual graphical user interface 3 is provided by a display device in the interior of the motor vehicle 1 . In the present case, this display device can be extended reality glasses, which are also referred to as augmented reality glasses.

Furthermore, an actuating device 4 is arranged in the motor vehicle 1 , which in the present case is arranged on a steering wheel 5 of the motor vehicle 1 . In the present case, the actuating device 4 comprises a plurality of actuating elements which are set up to receive a user input. A value can be specified by the vehicle occupant via the multiple actuating elements of the actuating device 4 by means of the user input, which value is to be set for the vehicle function specified by the selected virtual field 2 . As in 1 can be recognized particularly well, each virtual field 2 is provided with a symbol characterizing the associated vehicle function in the virtual graphical user interface 3 . In the present case, field 2 is marked by the lightbulb as a symbol, which field is assigned to the vehicle function lighting of motor vehicle 1 . Additional fields 2 can be assigned respective symbols for air conditioning of motor vehicle 1 in order to identify the vehicle function air conditioning as being assigned to fields 2 . A symbol that characterizes media playback can be assigned to at least one further, virtual field 2 in order to indicate that the vehicle function of media playback is assigned to this virtual field 2 . In addition, virtual fields 2 can be provided, to which the vehicle function driver assistance system is assigned.

FIG. 2 shows a method diagram for a method for operating the motor vehicle 1. A vehicle function of motor vehicle 1 is set as part of the method. The method provides that in a first method step V1 by means of a viewing direction detection device, which is present in 1 indicated schematically with a box and the reference number 6, a line of sight of the vehicle occupant is detected. The viewing direction detection device 6 is thus used to determine which of the virtual fields 2 the vehicle occupant is looking at, in particular the vehicle occupant wearing the augmented reality glasses, in particular the driver of the motor vehicle 1 . In a second method step V2 of the method, an electronic computing device of the motor vehicle 1 is used to determine that the predefined virtual field 2 has been selected by the vehicle occupant, for which it has been determined that the view is on this virtual field 2 . Here, the field 2 can be determined as selected if it is determined that the gaze of the vehicle occupant is on the virtual field 2 without interruption for a period of time longer than a predetermined minimum time interval. This predetermined minimum time interval can in particular be stored in the electronic computing device. The electronic computing device can be used to compare the determined uninterrupted period of time for the vehicle occupant to look at the virtual field 2 with the specified minimum time interval and, depending on a result of the comparison, it can be determined that the virtual field 2 is selected or that the virtual field 2 is not is selected.

In particular, the line of sight detection device 6 can receive display information from a display device that provides the virtual graphical user interface 3, in this case from the augmented reality glasses, which characterizes how the virtual graphical user interface 3 uses augmented reality for the vehicle occupant wearing the augmented reality glasses is shown. As a result, it can be determined with particularly great precision by means of the viewing direction detection device 6 where, in particular at which virtual field 2, the vehicle occupant is looking. In a third method step V3 of the method, it is provided that the selected field 2 is assigned a real actuating device 4 . This means that the electronic computing device stores which actuating device 4 of the motor vehicle 1 is intended to specify the value for the vehicle function selected by the selected field 2 . In a fourth method step V4, provision is made for a user input to be determined using the actuating device 4 . Based on the assignment of the selected virtual field 2 to the actuating device 4, it is determined here that the user input received by means of the actuating device 4 is to be assigned to the vehicle function selected by the selected virtual field 2.

In a fifth method step V5 it is provided that a value characterized by the user input for the vehicle function assigned to the selected virtual field 2 is set using the electronic computing device of the motor vehicle 1 . In particular, the value characterized by the user input for the vehicle function assigned to the selected virtual field 2 can be set if the user input is determined within a predetermined time interval from the determination of the selection of the virtual field 2. It is thus determined that the value represented by the user input is to be assigned to the vehicle function specified by the virtual field 2, provided that there is at most the specified time interval between the determination of the selection of the virtual field 2 based on the determined gaze of the vehicle occupant and the receipt of the user input. If a time interval between the determination of the selection of the virtual field 2 and the receipt of the user input is more than the predetermined time interval, then the electronic computing device is used to determine that there is a connection between the user input and the vehicle function, which is assigned to the selected field 2. does not exist. As a result, the value characterized by the user input for the vehicle function specified by the selected field 2 is not set.

In order to avoid that the vehicle occupant, in particular the driver of the motor vehicle 1, has to keep his eyes on the selected virtual field 2 while specifying the value by means of the user input on the actuating device 4, it is provided that the virtual field 2 is selected is determined if a continuous actuation of the actuating device 4 with user inputs is determined from a point in time within the predetermined time interval from the determination of the selection of the virtual field 2, regardless of whether after the determination of the selection of the virtual field 2 and the start of the continuous actuation of the Actuating device 4 the view of the vehicle occupant continues to rest on the selected virtual field 2. This makes it possible for the vehicle occupant, in particular the driver of motor vehicle 1, to select virtual field 2 with his eyes and start specifying the value via the at least one user input on actuating device 4 and then avert his gaze from selected virtual field 2 and in particular can focus on a road traffic again.

In order to inform the vehicle occupants in a particularly simple manner about which virtual field 2 has been established as selected, this virtual field 2 can be adjusted in its representation. In the present case, the virtual field 2 determined to be selected is surrounded by a white frame in order to indicate to the vehicle occupant that the selection of this virtual field 2 has been determined. In order to inform the vehicle occupant in a particularly simple manner that the user input has been received, the vehicle occupant can be given haptic feedback when the user input is received, in particular by means of the at least one actuating element. The method enables the vehicle occupant to interact with the motor vehicle 1 with particularly little distraction.

The method described is based on the finding that state-of-the-art augmented reality glasses are mainly based on gaze recognition and gesture recognition in order to enable interaction with virtual elements. Here, gaze recognition can be very accurate, whereas gesture interaction is very complex, not particularly precise, and requires a lot of effort from operators to perform any type of interaction. In addition, the prior art lacks any haptic feedback during interaction. If such augmented reality glasses were used in a vehicle, for example by a driver while driving, these gestural interactions could draw much of a driver's attention away from a road.

Typically, an interaction with augmented reality glasses looks like the one described below. If a slider is presented by the augmented reality glasses, then in order to interact with the slider, a user must let the augmented reality glasses know which slider they want to interact with. This can usually be done by looking or by hand tracking, particularly where a hand is pointing. Now, to move the slider, the user has to grab the slider with his hand and move it sideways to adjust its value. This is a particularly complex part since it is based on hand tracking recognition and gesture recognition. In a situation in a moving motor vehicle 1, this can represent a very complex and distracting task for the driver of the motor vehicle 1.

This problem could be solved with the described invention by dividing the interaction inputs between the augmented reality glasses and an interaction hardware of the motor vehicle 1, in the present case the actuating device 4. Digital content such as the virtual graphical user interface 3 would be displayed using the augmented reality glasses. Furthermore, the visual interaction with the vehicle occupant for selecting the virtual field 2 with which the vehicle occupant would like to interact could take place using the augmented reality glasses. In this case, the manual detection input would take place by means of the hardware of the motor vehicle 1, in the present case the actuating device 4, which provides a particularly good feel for the vehicle occupant. To enable the method, there is a connection between the motor vehicle 1 and the augmented reality glasses, whereby information from the augmented reality glasses with regard to the representation of the virtual graphical user interface 3 for the vehicle occupants, present the representation information, and optionally a gaze recognition can be synchronized with the user input received via the actuating device 4 . It is particularly advantageous if the motor vehicle 1 and the augmented reality glasses are informed about a status of the respective other.

An interaction protocol can be developed for the motor vehicle 1 and the augmented reality glasses, which enables both the motor vehicle 1 and the augmented reality glasses to be informed about where the vehicle occupant is looking, so that the hardware of the motor vehicle 1, here the actuating device 4 can be used as an input device for the virtual graphical user interface 3 . It is advantageous here if the motor vehicle 1 is informed about where the vehicle occupant is looking, since otherwise operating elements of the steering wheel can specify vehicle functions that are not related to the virtual graphical user interface 3 provided by the augmented reality glasses, such as Assistance systems or media playback.

Overall, the invention shows how interaction capacities of augmented reality glasses can be combined with hardware of motor vehicle 1 for particularly good interaction.

Reference List

1

motor vehicle

2

virtual field

3

virtual graphical user interface

4

actuating device

5

steering wheel

6

eye tracking device

V1-V5

respective procedural steps

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• DE 102013005342 A1 [0002]
• US 20200201041 A1 [0003]
• WO 2020111308 A1 [0004]

Claims (10)

Method for operating a motor vehicle (1), in which - a viewing direction of a vehicle occupant is recorded by means of a viewing direction recording device (6), - A selection of a predetermined virtual field (2), which is provided as a virtual graphical user interface (3) by means of augmented reality, is determined by a vehicle occupant if it is determined that the vehicle occupant is looking at the field (2), - a real actuating device (4) is assigned to the selected field (2), - A user input is determined by means of the actuating device (4), and - Using an electronic computing device, a value characterized by the user input is set for the vehicle function assigned to the selected field (2). procedure after claim 1 wherein the field (2) is determined to be selected from a plurality of predetermined fields (2) if it is determined that the vehicle occupant's gaze is on the field (2) continuously for a period of time longer than a predetermined minimum time interval. procedure after claim 1 or 2 , wherein the value characterized by the user input for the vehicle function assigned to the selected field (2) is set if the user input is determined within a predetermined time interval from determining the selection of the field (2). procedure after claim 3 , wherein the field (2) is determined as selected if a continuous actuation of the actuating device (4) with user inputs from a point in time within the predetermined time interval from determination of the selection of the field (2) is determined. Method according to one of the preceding claims, wherein the line of sight detection device (6) receives display information from a display device providing the virtual graphical user interface (3) which characterizes how the graphical user interface (3) is displayed for the vehicle occupant by means of augmented reality. Method according to one of the preceding claims, in which each field (2) is displayed in the virtual graphical user interface (3) with a symbol characterizing the associated vehicle function. Method according to one of the preceding claims, wherein the user input is received by means of the actuating device (4) which is arranged on a steering wheel (5) of the motor vehicle (1). Method according to one of the preceding claims, with the electronic computing device depending on the determined user input as a vehicle function lighting of the motor vehicle (1) and / or media playback in the motor vehicle (1) and / or and air conditioning in the motor vehicle (1) and / or a driver assistance system is set. Method according to one of the preceding claims, in which the vehicle occupant is given haptic feedback by means of the actuating device (4) when the user input is determined. Motor vehicle (1) which is set up to carry out a method according to one of the preceding claims.

Images