DE102020000696A1 - Method for controlling a man-machine interface, computer program product and motor vehicle - Google Patents

Abstract

A method for controlling a man-machine interface is described, which has at least one arithmetic unit, a memory, a display and at least one camera that is oriented in the direction of an operator using the man-machine interface, with gestures from the camera Operator and interpreted by the computing unit, at least one gesture is assigned to at least one operating action or input action in the memory, the at least one operating action or input action being carried out when the execution of the at least one gesture is recognized, the camera capturing a mouth of the operator , the gesture being a blow gesture. A computer program product and a motor vehicle are also described.

Description

A method for controlling a man-machine interface, a computer program product and a motor vehicle are described.

Methods for controlling a man-machine interface, computer program products and motor vehicles of the type mentioned at the beginning are known in the prior art.

Human-machine interfaces can be found almost everywhere. Human-machine interfaces should be intuitive and safe to use in order to make using a machine as easy as possible. Increasingly, human-machine interfaces are being equipped with display devices and sensors, for example cameras, in order to replace operation via classic switches, buttons and controls. An example of such a man-machine interface is a mobile phone, which usually has a front camera, a screen and other sensors, for example microphones, buttons, 3-D scanners, etc. Another example is an infotainment system of a motor vehicle that can be operated by means of gesture control or voice input.

In the JP 11338490 A discloses a digital signal processor which receives image signals from a CCD camera and extracts lip patterns therefrom. The recognition of this light pattern serves to recognize the fact that the observed person is speaking and is intended to start a speech recognition of an audio signal recorded in parallel.

In the motor vehicle environment in particular, drivers should not be distracted by operating the vehicle's own human-machine interface. Ideally, it should be possible to operate the relevant human-machine interface without having to take your hands off the steering wheel. This makes the number of possible ways of entering operating commands small. Input by hand should be avoided, especially in critical traffic situations. Voice commands are not easy to implement in the vehicle interior, especially since it is usually loud in the interior while driving, music is played or the driver is talking. The system must be able to separate these other sources of noise or recognized speech inputs that are not related to input commands from operating commands. One possibility is to provide fixed terms. The disadvantage here is that many drivers find these systems too rigid and are often frustrated because the system does not work well. A more modern alternative is to use a starting phrase followed by the commands, but this requires several steps before a command can be entered.

Thus, the task arises of developing methods for controlling a man-machine interface, computer program products and motor vehicles of the type mentioned at the beginning in such a way that an intuitive, hands-free input of operating commands into the man-machine interface is possible.

The object is achieved by a method for controlling a man-machine interface according to claim 1, a computer program product according to the independent claim 9 and a motor vehicle according to the independent claim 10. Further configurations and developments are the subject of the dependent claims.

A method for controlling a man-machine interface is described, which has at least one arithmetic unit, a memory, a display and at least one camera that is oriented in the direction of an operator using the man-machine interface, with gestures from the camera Operator and interpreted by the computing unit, at least one gesture is assigned to at least one operating action or input action in the memory, the at least one operating action or input action being carried out when the execution of the at least one gesture is recognized, the camera capturing a mouth of the operator , where the gesture is a blow gesture.

Blow gestures can be performed without hands. In addition, they are easy to distinguish from language, as the blow gesture is very characteristic, especially if it does not appear in the middle of a spoken sentence, but rather in isolation.

In a first further embodiment, it can be provided that at least one further sensor that detects the operator is provided, the gesture comprising a movement of the mouth and an activity detected by the further sensor, the operating action or input action being carried out when the execution of the gesture is performed using the at least one camera and the at least one further sensor is detected.

Corresponding gestures can have several components, that is to say not only visual components - a movement of the mouth - but also include other components, for example sounds. A gesture can be identified more clearly by a requirement that a gesture must be carried out in at least two such parameter dimensions.

In a further refinement, it can be provided that the at least a sensor is a microphone, wherein the microphone detects noises emitted by the operator, wherein the gesture comprises a blow noise detected by the microphone.
When performing a blow gesture, it has proven to be particularly useful to also record the blow noise. This enables a clear identification of a blow gesture and a differentiation from other actions that are similar to a blow gesture in terms of the movement of the mouth. Such other actions include, for example, speaking an "O" or forming a kissable mouth.

In a further refinement, it can be provided that the blow gesture is recorded by means of the operator's lips pointing.

The tip of lips can be recognized, for example, by bulging the operator's lips in a side view.

In a further refinement, it can be provided that a normal lip width of the operator is recognized and stored, with lip tips being recognized by drawing the lips together and forming an opening of the lips and / or extending the lips towards the camera.
Viewed from the front, the lips are drawn together when blowing against a neutral lip position and a small, mostly almost circular opening is created through which air is forced out. This can be detected by the camera.

In a further refinement, it can be provided that the blow gesture is assigned to closing a window on the display.
In this way, the gesture can be assigned to blowing away the window in an intuitive and understandable way.

In a further refinement, it can be provided that the blow gesture is assigned to an increase in a display size.

The gesture can be compared to inflating a balloon, so that the surface of the inflated object is enlarged.

Both gestures can also be displayed at the same time by measuring the duration of the gesture according to a further development. A short blow to close a window and a long blow to increase a display size.

In a further refinement, it can be provided that the man-machine interface is installed in a motor vehicle, the display being a head-up display or a cockpit display.

Blow gestures are particularly useful in motor vehicles, as they can be clearly distinguished from communication on the one hand and do not require any interaction with fingers or eyes on the other hand, so that the driver's full attention can be devoted to road traffic.

A first independent subject relates to a computer program product with a computer-readable storage medium on which instructions are embedded which, when executed by at least one computing unit, have the effect that the at least computing unit is set up to carry out the method of the aforementioned type.

The method can be carried out distributed on one or more processing units, so that certain method steps are carried out on one processing unit and other method steps are carried out on at least one further processing unit, with calculated data being able to be transmitted between the processing units if necessary.

Another independent subject matter relates to a motor vehicle with a computer program product of the type described above.

Further advantages, features and details emerge from the following description in which at least one exemplary embodiment is described in detail - possibly with reference to the drawing. Identical, similar and / or functionally identical parts are provided with the same reference symbols.

• 1 eine Mensch-Maschine-Schnittstelle mit einer Anzeige;
• 2 eine Geste eines Bedieners der Mensch-Maschine-Schnittstelle aus 1, sowie
• 3 einen Verfahrensablauf.

They show schematically:

• 1 a man-machine interface with a display;
• 2 a gesture by an operator of the human-machine interface 1 , such as
• 3 a procedure.

1 shows a plan view of a motor vehicle 2 with a man-machine interface 4th (framed by dashed lines).

The human-machine interface 4th instructs a controller 6th on that one arithmetic unit 8th and a memory 10 includes, being in the memory 10 , a computer program product is stored which, when loaded and from the computing unit 8th is carried out, carry out the procedure described below.

With the controller 6th are a driver monitoring camera 12 for monitoring a driver 14th as well as a microphone 16 connected, both in the interior of the motor vehicle 2 are arranged. With the help of the driver monitoring camera 12 becomes a head of the driver 14th detected and recognized gestures performed by him. Using the microphone 16 noises can be recorded in the interior, including from the driver 14th emitted noises.

The control 6th is also with a head-up display 18th connected that in a windshield of the motor vehicle 2 is positioned and over which the driver 14th is supplied with information. Such information can contain information on speed, navigation, maximum speed and more, for example information on playback in the multimedia system. Furthermore, warning messages can be shown in the head-up display 18th are displayed, for example messages about wrong-way drivers, about a low fuel level and the like. As the head-up display 18th usually only has a limited display space, such information must be superimposed on other information, so that at the moment it is displayed, the driver 14th cannot see all normal information, such as navigation information.

In order to be able to remove such information, the driver should not be forced to use the human-machine interface with his hands 4th to interact, as he may be in a traffic situation that requires attention. Instead, operator inputs should be possible as far as possible.

By grasping the driver's head 14th and thus detecting one of the area of the lips, it is possible to recognize gestures carried out with the face, in particular the mouth. A blowing gesture proposed here can be recognized by the fact that the mouth is narrowed compared to a relaxed state and a small, mostly circular opening is created between the lips of the mouth. Using the microphone 16 At the same time, attention can be paid to blowing noises, which have a very characteristic frequency spectrum.

2 shows the driver 14th in a side view when performing a blow gesture 19th .

In the side view it can be seen that when performing the blow gesture 19th Lips 20th of the driver 14th by a bulge distance X be pushed forward. This movement is relatively characteristic and is only carried out by people when they speak an "O", when drinking with a straw, when forming a kissing mouth or when blowing. You can drink with the help of the driver monitoring camera 12 Identify clearly and between an original sound, a kiss and a puff with the help of the microphone 16 can be distinguished.

3 shows a process flow using a mobile phone 22nd .

On the mobile phone 22nd is on a display 24 a window 26th open. The driver 14th want the window 26th close, makes a blow gesture 19th out that through a front camera 28 and a microphone 30th of the mobile phone 22nd is recognized. The mobile phone 22nd registered this as a wish to the window 26th to close and executes the operating command accordingly. The window 26th will be closed.

Although the invention has been illustrated and explained in more detail by preferred exemplary embodiments, the invention is not restricted by the disclosed examples and other variations can be derived therefrom by the person skilled in the art without departing from the scope of protection of the invention. It is therefore clear that there is a multitude of possible variations. It is also clear that embodiments cited by way of example really only represent examples that are not to be interpreted in any way as a limitation, for example, of the scope of protection, the possible applications or the configuration of the invention. Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without leaving the scope of protection that is defined by the claims and their legal equivalents, such as a more detailed explanation in the description.

List of reference symbols

2

Motor vehicle

4th

Man-machine interface

6th

control

8th

Arithmetic unit

10

Storage

12th

Driver surveillance camera

14th

driver

16

microphone

18th

Head-Up Display

19th

Blow gesture

20th

Lips / mouth

22nd

Cellphone

24

Display

26th

window

28

camera

30th

microphone

x

Bulge Distance

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• JP 11338490 A [0004]

Claims (10)

A method for controlling a man-machine interface (4) which has at least one arithmetic unit (8), a memory (10), a display (18; 24) and at least one camera (12; 28) which is directed towards one of the The operator (14) using the human-machine interface (4) is aligned, gestures (19) of the operator (14) being recorded by the camera (12; 28) and interpreted by the processing unit (8), the memory ( 10) at least one gesture (19) is assigned to at least one operating action or input action, the at least one operating action or input action being carried out when the execution of the at least one gesture (19) is recognized, characterized in that the camera (12; 28) has a mouth (20) of the operator (14) detected, the gesture (19) being a blow gesture. Procedure according to Claim 1 , characterized in that at least one further sensor (16; 30) detecting the operator is provided, the gesture (19) comprising a movement of the mouth (20) and an activity detected by the further sensor (16; 30), the Operating action or input action is carried out when the execution of the gesture (19) is detected by means of the at least one camera (12; 28) and the at least one further sensor (16; 30). Procedure according to Claim 2 , characterized in that the at least one sensor is a microphone (16; 30), the microphone (16; 30) detecting noises emitted by the operator, the gesture (19) comprising a blowing noise detected by the microphone (16; 30). Method according to one of the preceding claims, characterized in that the blow gesture (19) is recorded on the basis of the operator (14) tipping lips (20). Procedure according to Claim 4 , characterized in that a normal lip width of the operator (14) is recognized and stored, a lip tip based on a contraction of the lips (20) as well as forming an opening of the lips (20) and / or protruding the lips (20) in the direction the camera (12; 28) is recognized. Method according to one of the preceding claims, characterized in that the blow gesture (19) is associated with closing a window (26) on the display (18; 24). Method according to one of the preceding claims, characterized in that the blow gesture (19) is assigned to an increase in a display size. Method according to one of the preceding claims, characterized in that the man-machine interface (4) is installed in a motor vehicle (2), the display (18; 24) being a head-up display (18) or a cockpit display ( 24) is. Computer program product with a computer-readable storage medium (10) on which instructions are embedded which, when executed by at least one computing unit (8), have the effect that the at least computing unit (8) is set up to implement the method according to one of the preceding claims execute. Motor vehicle with a computer program product Claim 9 .

Images