DE102019131944A1 - Method for controlling at least one display unit, motor vehicle and computer program product - Google Patents

Abstract

The invention relates to a method for controlling at least one display unit (4) of a motor vehicle (2), at least part of an interior (12) of the motor vehicle (2) being monitored by sensors by generating sensor data using at least one sensor unit (16) and by the sensor data are evaluated by means of a data processing device (20), the data processing device (20) recognizing a pointing gesture of an operator (14) in which a finger (22) of the operator (14) points in a pointing direction (24), the Data processing device (20) determines a pointing position at which the pointing direction (24) intersects an image area (6) of the at least one display unit (4) and wherein the at least one display unit (4) reproduces a position indicator (26) at the pointing position.

Description

The invention relates to a method for controlling at least one display unit of a motor vehicle. The invention also relates to a motor vehicle and a computer program product.

In a motor vehicle, a number of items of information are typically made available to a driver during operation, the corresponding information being output via one or more display units. Typical display units are, for example, a so-called combination instrument, a so-called infotainment screen, which is usually arranged in the so-called dashboard or in the so-called center console, and a so-called head-up display (HUD).

At least in a more recent generation motor vehicle, it is usually possible to vary the number and / or the selection of the information output so that a driver can adapt the number and / or the selection to his own needs. For this purpose, an interface is then implemented in the motor vehicle, for example a control panel in the area of the center console, via which at least one of the display units can be controlled and via which the number and / or selection of information that is output via the display unit can then be set.

The object of the present invention is to specify an advantageous method for controlling at least one display unit of a motor vehicle and an advantageously designed motor vehicle and an advantageous computer program m product.

This object is achieved by a method with the features of claim 1, by a motor vehicle with the features of claim 9 and by a computer program product with the features of claim 10. The advantages and preferred embodiments cited with regard to the method can also be applied to the devices and vice versa. Advantageous configurations with expedient developments of the invention are specified in the dependent claims.

The method according to the invention is designed to control at least one display unit of a motor vehicle, so for example a so-called instrument cluster, a so-called infotainment screen, which is usually arranged in a so-called dashboard or in a so-called center console, a so-called head-up display (HUD ), a so-called augmented reality head-up display (AR-HUD), a so-called 3D head-up display (AR-HUD) etc.

In the course of the method, at least part of the interior or passenger compartment of the motor vehicle is monitored by sensors by generating sensor data by means of at least one internal sensor unit and by evaluating the sensor data by means of a data processing device. Here, the at least one internal sensor unit is typically designed as an optical sensor unit and includes, for example, one or more cameras, that is to say for example a stereo camera, 3D camera or TOF camera. The data processing device is designed, for example, as a unit, for example as a central data processing unit of the motor vehicle, or has a plurality of data processing units. Different software modules or software programs are typically used in the data processing device for different tasks. According to one embodiment variant, at least part of the data processing device is designed as a type of neural network (data processing device is based on a “deep neural network” technology).

Furthermore, in the course of the method, the data processing device recognizes a pointing gesture by an operator, in particular a driver of the motor vehicle, in which a finger of the operator points in a pointing direction. The pointing gesture is recognized, for example, when the operator spreads his index finger and clenches the remaining fingers into a fist. If, on the other hand, the operator puts the index finger back on or spreads the other fingers, the pointing gesture is no longer recognized.

In addition, the data processing device determines a pointing position at which the pointing direction intersects an image area of the at least one display unit, and the at least one display unit then reproduces a position indicator at the pointing position.

In this way, a user interface is then implemented, the use of which the operator's finger functions as a kind of virtual laser pointer. As a result, the operator then sees the position pointer at the position of the image area to which the operator is pointing. Here, for example, the surface of a screen forms the image area or the windshield or a side window of the motor vehicle forms the image area, for example if the at least one display unit is a head-up display, augmented reality head-up display or 3D Head-up display is designed.

However, the representation of the position indicator does not necessarily correspond to a point or a circular area; instead, the representation of the position indicator is typically dependent on the application and, moreover, varies Representation of the position pointer typically during use, as is also the case, for example, with a mouse pointer on a computer.

In a preferred development, changes in the pointing direction are also subsequently recorded and the associated changes in the pointing position are determined. Based on this, the position of the position pointer is then adjusted according to the changes in the pointer position, so that the position pointer follows the movements of the operator's finger and the position pointer is always displayed where the operator is pointing. Here, the pointing direction is typically to be understood as a vector and, in particular, a vector with a point of attack. Tracking the movements of the operator's finger is typically made possible by a suitable software module or software program in the data processing device, which can be referred to as a “finger tracking unit”.

According to one embodiment variant, the at least one display unit reproduces a graphical user interface in at least one usage mode and the position indicator is used to control the graphical user interface. In this case, the operator's finger then functions as a kind of virtual input pen (touch pen). In contrast to input pens, such as those used together with smartphones or tablets, it is not necessary for the finger to touch a surface, for example a touchscreen; instead, the movement is typically in the air, i.e. in free space, executed. With the help of the virtual input pen, for example, virtual buttons, switches, rotary controls and / or sliders can be operated.

It is also advantageous not only in this context if the position indicator is used by the operator together with at least one further and in particular several further inputs, for example further gestures as input commands or voice commands as input commands.

In particular when the position indicator is used together with further input commands, it is also advantageous if the position indicator is used in at least one usage mode to implement a drag-and-drop function. Alternatively, such a drag-and-drop function is implemented without further inputs, for example by activating the drag when the position indicator remains in one position for a longer period of time, for example 5s, and by activating the dropping function, when the position indicator remains in one position for an extended period of time. In this way, for example, display content or playback content, such as display windows or so-called widgets, can be rearranged. As an alternative or in addition to this, the number of reproduction contents can be varied and / or a selection can be made as to which reproduction contents are reproduced.

Furthermore, a variant of the method is advantageous in which a second display unit of the motor vehicle is additionally controlled. The data processing device then determines a pointing position at which the pointing direction intersects an image surface of the second display unit, and the second display unit then reproduces the position indicator at the pointing position. Here, the at least one display unit, also referred to below as the first display unit, and the second display unit are typically spatially separated from one another. The data processing device then automatically recognizes which of the two display units the operator is pointing at, and the position indicator is then displayed on this display unit. The concept can also be expanded to a larger number of display units and, depending on the application, the motor vehicle has more than two display units, which are then controlled in the manner described above.

If there are now two corresponding display units and a drag-and-drop function described above is also implemented, it is further advantageous if, by means of the drag-and-drop function, a display content, for example a display window or a so-called widget, from one of the two image areas is shifted onto the other, so that the playback content that was initially displayed by one of the two display units is subsequently displayed via the other display unit as a result of the use of the drag-and-drop function.

According to a further advantageous embodiment, the position indicator is used in at least one usage mode to implement a handwritten and / or free-hand drawing input.

Depending on the application, it is also advantageous if the data processing device determines a viewing direction of the operator based on the alignment of the operator's head and / or based on the alignment of the operator's eyes. For this purpose, the sensor data of the at least one internal sensor unit and / or sensor data of at least one further internal sensor unit are typically used. The determination of the line of sight of the operator serves, for example, as an aid for the determination of the pointing position, that is to say, for example, to increase the accuracy with which the pointing position is determined. Tracking changes in the viewing direction of the The operator is typically enabled by a suitable software module or software program in the data processing device, which can be referred to as a “head tracking unit” or “gauze tracking unit”.

In particular, if the at least one display unit, i.e. the first display unit, is designed as a head-up display, augmented reality head-up display or 3D head-up display, the data processing device determines in at least one of them according to a further variant of the method Usage mode, in addition to the pointing position, a pointing location outside of the motor vehicle. For this purpose, the motor vehicle then has at least one external sensor unit, that is to say, for example, an environment camera with which the environment of the motor vehicle is recorded. The pointing location corresponds to a position outside the motor vehicle to which the operator is pointing.

In order to determine the pointer location, the direction of view of the operator is furthermore preferably determined by the data processing device. In this case, the viewing direction, the pointing position and the pointing location are then more or less on the same line, so that, for the operator, for example, the pointing position and the pointing location appear to coincide and the operator perceives the position indicator as being at the pointing location.

Depending on the embodiment variant, the operator can then request information about the pointing location, for example about a building at the pointing location, by means of an input, for example a voice command. The information is then usually displayed next to the position indicator.

According to a further embodiment variant, the direction of view of the operator is determined by the data processing device in order to thereby implement a special variant of the drag-and-drop function described above. In this special variant, a 3D head-up display is used as a display unit, which in at least one operating mode shows several, ie at least two, image planes, for example by projecting image data onto several image areas. Using the special drag-and-drop function, it is then preferably possible to move a display content, for example a display window or a so-called widget, from one of the image levels or image areas to another, so that the display content, which is initially on an image level or image area is subsequently displayed on a different image layer or image area as a result of the use of the drag-and-drop function.

The method described above is intended to be carried out in a motor vehicle. Conversely, a motor vehicle according to the invention is set up to carry out the corresponding method in at least one operating mode or usage mode. It has at least one display unit, an internal sensor unit and a data processing device. The motor vehicle also preferably has an external sensor unit. The individual method steps of the method are then carried out by means of the data processing device, with an executable program further preferably being stored or installed in the data processing device, which program automatically executes the method after it is started.

A corresponding program can also be installed or stored subsequently by means of a computer program product according to the invention, provided that suitable hardware is already available in a motor vehicle. That computer program product is typically a file or a data carrier with a file, the file containing the executable program, that is to say in particular a suitable program code.

• 1 in einem Blockschaltbild ein Kraftfahrzeug; und
• 2 in einer teilweisen Cockpitansicht das Kraftfahrzeug.

Further advantages, features and details of the invention emerge from the claims, the following description of preferred embodiments and on the basis of the schematic drawings. Show:

• 1 a motor vehicle in a block diagram; and
• 2 the motor vehicle in a partial cockpit view.

A method described below by way of example is used in a motor vehicle 2 executed which in 1 is shown schematically. The car 2 is accordingly set up to carry out the described method.

It has a first display unit 4th with a first picture plane 6th and a second display unit 8th with a second picture plane 10 on. It is both the first display unit 4th as well as the second display unit 8th at least partially in an interior 12th of the motor vehicle 2 arranged in which an operator 14th , in the exemplary embodiment the driver of the motor vehicle 2 , stops.

The motor vehicle also has 2 an internal sensor unit 16 with multiple cameras 18th on as well as a data processing device 20th , wherein the data processing device 20th is set up, a pointing gesture of the operator 14th to recognize the case of an index finger 22nd of the operator 14th in a pointing direction 24 shows. In that direction of pointing 24 it is typically a vector and in particular a vector with a point of attack.

Furthermore, the data processing device 20th is set up to determine a pointing position at which the pointing direction 24 one of the image areas intersects, that is to say either the image area in the exemplary embodiment 6th the first display unit 4th or the picture plane 10 the second display unit 8th . A position pointer is then placed at the pointing position when the method is carried out 26th reproduced.

So it is in the motor vehicle 2 realized an interface, when used, so to speak, the index finger 22nd of the operator 14th acts as a kind of virtual laser pointer. In the exemplary embodiment, this virtual laser pointer is activated by recognizing the pointing gesture, with the pointing gesture being recognized, for example, when the operator 14th the index finger 22nd apart and clench the remaining fingers into a fist. Sets the operator 14th on the other hand the index finger 22nd on again or if the other fingers are also spread apart, the pointing gesture is no longer recognized and the virtual laser pointer is quasi deactivated.

In the exemplary embodiment, the position indicator 26th used in at least one usage mode to implement a drag-and-drop function. Voice commands are also used to enter the command for dragging and the command for putting down, and the motor vehicle instructs accordingly 2 according to 1 a microphone 28 on which with the data processing device 20th connected is.

In the exemplary embodiment, the drag-and-drop function is such that playback content 30th can not only be moved on an image area 6.10 of a display 4.8, but from an image area 6.10 of a display unit 4.8 to the image area 10.8 of the other display unit 8.4, as shown in FIG 2 is indicated. Here is the first display unit 4th designed as a so-called head-up display and has a projection unit 32 and a windshield as the first screen 6th . The second display unit 8th is in the embodiment according to 2 designed as an instrument cluster, the instrument cluster in this case being designed as a screen.

Two reproduction contents are now exemplified 30th via the first display unit 4th played and two other playback contents 30th via the second display unit 8th . The playback contents 30th are symbolized by dashed (rectangular) frames. Using the drag-and-drop function, the operator can 14th a playback content 30th from the first display unit 4th on the second display unit 8th move so that this playback content 30th subsequently no longer via the first display unit 4th is reproduced but on the second display unit 8th . Such a rendering content 30th is in 2 already marked, with the position indicator 26th above the corresponding playback content 30th is located. This rendering content 30th is displayed at the point where the dotted frame is displayed after the drag-and-drop function has been executed. When transferring the playback content 30th from the first display unit 4th on the second display unit 8th in the exemplary embodiment to the second display unit 8th automatically adjusted, for example by rescaling to a smaller size and / or a different format.

Furthermore, a usage mode, not shown in detail, is implemented in the exemplary embodiment in which the position indicator 26th is used to implement a handwritten and / or free-hand drawing input.

In addition, a usage mode is implemented in the exemplary embodiment in which the data processing device 20th in addition to the pointing position, a pointing location outside of the motor vehicle 2 determined. To this end, the motor vehicle 2 an external sensor unit 34 on, for example an environment camera, with which the environment of the motor vehicle 2 is captured.

The pointer location corresponds to a position outside of the motor vehicle 2 to that of the operator 14th shows. To determine the pointer location, the data processing device 20th based on the data from the internal sensor unit 16 the line of sight of the operator 14th determined. The line of sight is typically here 24 , Pointing position and Pointing location quasi on one line, so that for the operator 14th The pointing position and the pointing location seem to coincide and the operator 14th the position indicator 26th perceives as being at the place of the advertisement.

Depending on the design variant, the operator can 14th then request information about the pointing location, for example about a building, by means of an input, for example a voice command 36 at the point of view. The information is then usually displayed next to the position indicator 26th displayed.

List of reference symbols

2

Motor vehicle

4th

first display unit

6th

first screen

8th

second display unit

10

second screen

12th

inner space

14th

operator

16

internal sensor unit

18th

camera

20th

Data processing device

22nd

index finger

24

Pointing direction

26th

Position pointer

28

microphone

30th

Playback content

32

Projection unit

34

external sensor unit

36

building

Claims (10)

Method for controlling at least one display unit (4) of a motor vehicle (2), wherein - at least part of an interior (12) of the motor vehicle (2) is monitored by sensors by generating sensor data by means of at least one sensor unit (16) and by evaluating the sensor data by means of a data processing device (20), - A pointing gesture of an operator (14) is recognized by the data processing device (20), in which a finger (22) of the operator (14) points in a pointing direction (24), - The data processing device (20) determines a pointing position at which the pointing direction (24) intersects an image area (6) of the at least one display unit (4) and - The at least one display unit (4) reproduces a position indicator (26) at the pointing position. Procedure according to Claim 1 wherein changes in the pointing direction (24) are subsequently detected, the associated changes in the pointing position being determined and the position of the position indicator (26) being adapted in accordance with the changes in the pointing position. Procedure according to Claim 1 or 2 wherein the at least one display unit (4) reproduces a graphical user interface and wherein the position indicator (26) is used to control the graphical user interface. Method according to one of the Claims 1 to 3rd , wherein the position indicator (26) is used in conjunction with voice commands. Method according to one of the Claims 1 to 4th , wherein the position indicator (26) is used to implement a drag-and-drop function. Procedure according to Claim 5 - In addition, a second display unit (8) of the motor vehicle (2) is controlled, - the data processing device (20) determines a pointing position at which the pointing direction (24) intersects an image area (10) of the second display unit (8), - the second display unit (8) reproduces the position pointer (26) at the pointing position and - the position pointer (26) is used to implement a drag-and-drop function, a display content (30) of one of the two image areas (6,8) is shifted to the other. Method according to one of the Claims 1 to 6th , wherein the position indicator (26) is used to implement a handwritten and / or freehand drawing input. Method according to one of the Claims 1 to 7th , wherein the data processing device (20) determines a viewing direction of the operator (14), in particular as an aid in determining the pointing position. Motor vehicle (2) set up to carry out a method according to one of the preceding claims in at least one operating mode. Computer program product containing a program which can be executed on a data processing device 20, in particular a data processing device (20) of a motor vehicle (2), which automatically executes a method according to one of the preceding claims after it is started.

Images