DE102014224641A1 - Method for operating an input device, input device - Google Patents

Abstract

The invention relates to a method for operating an input device (2), in particular a motor vehicle (1), by means of a non-contact sensor (3) a position and / or position change of at least one finger (9, 10, 11, 12) of a hand ( 6) of a user, whereby, depending on the position and / or position change of the finger (9, 10, 11, 12), an input is recognized and executed. It is provided that, depending on a detected reciprocating movement of the finger (9, 10, 11, 12) an analogous size is changed by the input device (2).

Description

The invention relates to a method for operating an input device, in particular a motor vehicle, having a non-contact sensor for detecting the position and / or position change of at least one finger of a user's hand, depending on the detected position and / or position change of the finger an input is recognized and executed.

Furthermore, the invention relates to a corresponding input device.

State of the art

In today's motor vehicles operating concepts are used in which an input device and a display device are arranged close to each other. Usually, so-called touch-sensitive screens and so-called touch screens are provided for this purpose, in which operation and display take place at the same location. Frequently display devices are mounted in the upper part of a control panel or dashboard of the motor vehicle, so that the driver does not have to turn his eyes too far away from the traffic to read. In other vehicles are a touchpad, so a touch-sensitive sensor, in the area of the armrest of the driver, and the display device at the usual place in the area of the dashboard. A visual feedback to the driver in the operation of the sensor can take place in the form of an indicated transparent hand, which is represented by the display device. Thereby, the driver can conveniently operate the input device while the display is still presented to him in an advantageous viewpoint. In this case, it is also conceivable not to design the display device as a screen, but as a head-up display.

While classical touch-sensitive sensors or touchpads require a touch by the user to operate them, input devices are also known which recognize or register inputs without contact. With the aid of depth sensors, for example, the positions of a user's hand, fingers and / or arm are detected in the room and evaluated for gesture operation. Finger gestures require high resolution, which can be achieved with sensors such as time-of-flight sensors, stereo cameras, structured light, or the like. For hand or body gestures, sensors with a lower resolution can also be used, such as radar sensors. By one or more sensors, therefore, the position or position change of the hand of a user is detected, being detected in response to the detected position and / or position change to an input and this is executed. The user thus indicates with a movement of his hand or with at least one finger of his hand to the input device, which input he wants to make. The input device recognizes the desired input by means of the movement of the finger and executes it by converting the command given by the movement and, for example, changing an operating parameter of the motor vehicle. For example, depending on the position and position change of a user's finger, the "increase volume" input can be detected and executed by the input device by increasing the volume of, for example, an entertainment system of the motor vehicle.

From the publication US 2012 010 5613 A1 It is already known to record hand or finger gestures by means of a camera and to control functions of a motor vehicle as a function of the recognized gestures. A similar system is also from the published patent application EP 2 441 635 A1 already known. This also discloses a detection of the time course of changes in position of fingertips in free space.

Disclosure of the invention

The inventive method with the features of claim 1 has the advantage that the user by a executed in a free space gesture in a simple manner, an adjustment of an analog variable, such as the volume of a radius of the motor vehicle or the brightness of an instrument panel lighting or the like , can adjust. According to the invention, it is provided for this purpose that an analogous variable is changed as a function of a detected reciprocating movement of the at least one finger. By means of the sensor, therefore, the hand of a user is monitored for whether at least one finger of the user performs a reciprocating movement. Depending on the back and forth movement then the analog size is changed. In this case, provision is made in particular for the analogue variable to be changed more rapidly or more slowly as a function of the frequency of the reciprocating movement that is carried out.

According to a preferred embodiment of the invention it is provided that the movement of the finger is monitored for reversal points in which the finger changes its direction of movement in order to detect the reciprocating movement. When detecting the movement of the finger this becomes so monitored for the achievement of reversal points. Only when a reversal point has been detected can it be assumed that the finger is moving back and forth. Appropriately, all reversal points of the movement are detected and counted, in particular, depending on the number of detected reversal points, the analog variable is changed.

In this case, it is particularly preferably provided that the analogous variable is changed by a predefinable value upon detecting each reversal point. Thus, with each reversal point, the analogue quantity is changed a little bit further, so that in a simple manner, depending on the frequency, the reciprocating motion, that is, depending on the number of reversal points within a time period, the analogous quantity is changed faster or slower ,

According to an advantageous embodiment of the invention, it is provided that the analogous size is increased or decreased as a function of the finger performing the reciprocating movement. In this case, it is thus provided that the direction in which the analog variable is changed is determined as a function of the finger performing the movement. For example, if the user makes the movement with his index finger, the analog size is increased. However, if he performs the movement or gesture with another finger, for example with the middle finger, the analogous size is reduced. By means of the non-contact sensor, for which in particular a camera sensor or video sensor is used, not only the movement but also the movement of the finger can be determined on the basis of an image evaluation. Optionally, two camera sensors can be provided so that the user's hand is detected from two different directions, such that, for example, a first camera sensor detects the hand from above and a second hand from the side, the finger and the image through the first sensor based on the image of the second sensor, the movement of the fingers is determined. By overlaying the data, it can then be concluded in a simple manner which finger performs which movement. Based on an edge determination, however, both the finger and the movement performed by the finger can be detected and evaluated on the basis of only one camera image. In this case, provision is made in particular for the captured image data to be compared with previously stored manual models in order in particular to distinguish the fingers from one another.

According to a preferred embodiment of the invention it is provided that the analogous size is increased or decreased in dependence on an attitude of the finger having the hand. Thus, the user can specify by the different attitude of the hand in the detection range of the sensor, whether the performed back and forth movement of the finger should lead to an increase or decrease in the analog size. Preferably, it is provided that the analogous size is increased or decreased in dependence on an orientation of the hand. For example, the orientation of the hand may be determined depending on which direction the user's hand-held arm is in the sensing area of the sensor. It is also conceivable to determine the orientation of the hand by detecting the hand lying in the detection area in a plan view, in which the fingers of the hand lie next to one another, or in a side view, in which the fingers of the hand lie one behind the other. Depending on how the hand is aligned, then the to-and-fro movement of the finger is considered to be a desire to increase or decrease the analog size and to make a corresponding change in analog size through the input device.

Additionally or alternatively, it is preferably provided that the analog variable is increased or decreased in dependence on a positioning of the hand in the detection range of the sensor. Depending on where in the detection range of the sensor, the user holds the hand and performs the reciprocating motion, the analog size is increased or decreased. For example, it is provided that the detection area is divided into halves, the analog size being increased when the movement is performed in one half, and made smaller when the movement is performed in the other half.

In particular, it is provided that the analogous size is increased or decreased in dependence on an orientation of the thumb of the hand. This allows the user to easily specify whether the analog size should be increased or decreased. The orientation of the hand can remain the same. For example, aligning the thumb in the optical axis of the camera sensor or parallel to it may dictate reducing the analog size, and aligning the thumb perpendicular to the optical axis may increase or vice versa. Also, applying the thumb to the fingers may result in an increase or decrease in size, while spreading the thumb results in a reduction or enlargement of the analogue size, the amount of change depending on the reciprocation of the finger according to the invention.

According to an advantageous embodiment of the invention is further provided that is recognized on a reciprocating motion of the finger to change the analog size only when the finger hits at a turning point on a thumb of the same hand. This makes it easier to distinguish between a deliberately performed gesture and an unintentionally performed gesture or reciprocation of the finger. According to this embodiment, the input device will only change the analogous size if, during the reciprocation, the finger strikes the thumb, so that a turning point arises on the thumb. Moreover, this offers the advantage that the user himself experiences a haptic feedback in the operation of the input device by the impact of the finger on his own thumb.

The input device according to the invention with the features of claim 10 is characterized in that it performs the inventive method when used as intended. This results in the advantages already mentioned. Other features and advantages will be apparent from the foregoing and from the claims.

In the following, the invention will be explained in more detail with reference to the drawing. To show:

1 the interior of a motor vehicle with an advantageous input device,

2 a user's hand grasped by the input device.

1 shows a schematic representation of the interior of a motor vehicle not shown here 1 which is an input device 2 for non-contact input of control commands. For this purpose, the input device has a non-contact sensor 3 and a display unit 4 on. The display unit 4 is in the dashboard or in the control console of the motor vehicle 1 arranged arranged. The display unit 4 is designed as a screen, in particular display, and may for example be part of a navigation system or an entertainment system of the motor vehicle 1 be. It is also conceivable, the display unit 4 alternatively or additionally as a Head-Up Display (HUD) form. The non-contact sensor 3 is preferably designed as a two-dimensionally operating video camera or camera device, the detection area shown with dashed lines 7 having. The video camera is preferably oriented such that it faces the front end of a central armrest 5 of the motor vehicle 1 has. The armrest 5 itself has no real input surface on which a driver by means of a hand only schematically shown here 6 could enter a command by touching the input interface.

The input device 2 is instead adapted to a position and / or position change of at least one finger of the hand 6 in the room to capture and depending on an input to recognize and execute.

The method described below allows the user by simple finger movements an analog size of the motor vehicle, such as a volume of an entertainment system 8th of the motor vehicle 1 to selectively zoom in or out. To do this, the user has his hand 6 in the coverage area 7 of the sensor 3 move and do a back and forth movement with at least one finger of his hand.

2 shows by way of example that of the sensor 3 captured picture of the hand 6 a user. The user holds his hand 6 aligned so that the fingers 9 . 10 . 11 . 12 the hand 6 from the perspective of the sensor 3 at least substantially lying one behind the other, and the finger 9 (Index finger) and the thumb 13 span a plane that is substantially perpendicular to the optical axis of the sensor 3 is aligned.

To determine the finger movement and the orientation of the hand 6 First, a background modeling for determining the background and the foreground on the basis of the sensor 3 captured image using both static and dynamic models. Subsequent thresholding computes a binary foreground mask. Subsequently, a hand contour calculation is performed on the foreground mask to detect the contours of the hand and the individual fingers, if they are visible in the image, and thereby detect the movements of the fingers. In particular, fingertips become the fingers 9 . 10 . 11 . 12 determined and tracked their movement or recorded.

These position changes are first evaluated on whether the movement of a finger, such as the finger 9 , Reversing points. A reversal point is understood as the point in time when the movement of one finger is stopped and continued or taken up in the opposite direction. Based on the course of the position of the fingertips, the reversal points of the movement of a finger can be determined in a simple manner. For this purpose, a vector is advantageously calculated, which leads from the position of the fingertip at a time t - 1 to the position of the fingertip at the time t. For each time t, or for each of the sensor 3 supplied image / sample with the exception of the first image, such a direction vector can be calculated / determined. If the direction of the vector changes by about 180 ° when changing from t - 1 to t, ie if only the orientation of the vector changes except for a possible small change in direction, between t - 1 and t there is a reversal point or a reversal point in front.

To increase the robustness, it is advantageously checked whether the direction vectors of the two points t-1-n to t-1 have approximately the same direction except for possible outliers. The same test is performed for the direction vectors of the two points from t to t + n. The check prevents a single outlier (incorrectly recognized fingertip position) from causing a point in time to be incorrectly identified as a reversal point. The larger n is chosen, the more robust the process is over the outliers. The size of n is limited upwards by the number of pictures / samplers recorded between two reversal points. As an additional criterion, it is also possible to use the magnitude of the vectors, which decreases due to the (negative) acceleration of the respective finger or fingertip before a reversal point and then increases again. Prerequisite for the application of this criterion is a timely sampling by the sensor 3 ,

If there is at least one turning point, it is recognized that the finger 9 has performed a float. For this purpose, the vectors between two reversal positions are determined. The vectors of successive reversal positions must have approximately the same direction but opposite orientation. By this criterion, the reciprocation of a zigzag movement of the finger 9 be differentiated.

The rate of change of analog magnitude is determined by counting the reversal points. In particular, it is provided that the analog quantity is increased or reduced for each detected reversal point by a fixed predetermined value. The change is made as soon as possible after a reversal point is detected. Due to the robustness test, the detection of a reversal point is completed only n times after the occurrence of the reversal point in time.

Each time the analogue size is changed, a value x is added to or subtracted from the analogue quantity. The sign of x decides whether the analog value is increased or decreased when the gesture is executed. The sign is in particular by the attitude or position of the hand 6 in the detection range of the sensor 3 certainly. For example, the coverage area 7 of the sensor is divided into a left area B1 and a right area B2, and the sign of the value x is set by the position of the hand. If the hand is in the left-hand area B1 when carrying out the gesture, a positive sign is selected, and if the hand is in the right-hand area B2 when the movement is carried out, a negative sign is selected.

Alternatively it can be provided that the finger 9 with which the gesture is executed, decides on the sign. It is exploited that by means of the evaluation of the image captured by the sensor between the individual fingers 9 to 12 a distinction is made. If it is determined that the reciprocating motion is performed by the index finger, for example, a positive sign is selected, however, the movement with the middle finger 10 executed, a negative sign is selected.

Another alternative is to use the sign about the orientation of the thumb 13 set. Shows the thumb 13 for example, in the direction of the sensor 3 , and is thus aligned parallel to the optical axis of the camera sensor, a positive sign is selected, the thumb indicates 13 aside, he lies in the in 2 shown plane perpendicular to the optical axis, so a negative sign is selected. This variant has the advantage that a change in size by the user can also be reversed quickly by merely changing the orientation of the thumb. Of course, the signs described above can also be given the other way round.

Advantageously, a reciprocating motion is recognized as such only when one of the turning points on the thumb 13 lies. It is therefore required that the finger passing the movement on the thumb 13 meets or taps on this, so that the analog size is adjusted. This gives the user on the one hand a haptic feedback in the operation of the input device 2 and on the other hand, it can distinguish the performed reciprocation of the finger from an unwanted gesture of the finger and thus the operation of the input device 2 be improved.

The following is an example of the detection and evaluation of the position, the position changes and / or the orientation of the hand 6 or the finger 9 to 13 be explained in detail: first a background modeling according to Zivkovic, Zoran ( Zivkovic, Zoran; " Improved Adaptive Gaussian Mixture Model for Background Substrate ", ICPR, 2004 ) of the sensor 3 captured image. This will separate the background and the foreground. The background is the static or non-changing of the sensor 3 captured image in the motor vehicle understood. The static image serves as a reference image that is subtracted from a currently acquired image at any time of an image sequence, so that only pixels or pixels are marked in the resulting difference image, which differ from the previously defined / detected static background image. The foreground is the one in the detection range of the sensor 3 moving hand 6 of a user, as in 1 is shown by way of example.

As an alternative or in addition to the background modeling, it is provided that a foreground modeling according to Dadgostar and Sarrafzadeh (US Pat. Dadgostar, Farhad and Sarrafzadeh, Abdolhossein: "An adaptive real-time skin detector based on Hue thresholding: A Comparison of Two Motion Tracking Methods" Pattern Recognition Letters, 2006, 1342-1352 ), in which relevant foreground regions for hand recognition are detected by special skin color models, whereby the hand is recognized due to the color difference between the hand and the background.

On the basis of the data obtained by the foreground and / or background modeling, a hand contour of the hand becomes 6 determined and in particular also calculated the hand contour center of gravity. Advantageously, a hand contour segmentation according to Suzuki and Abe ( Suzuki, S. and Abe, K .: Topological Structural Analysis of Digitized Binary Images by Border Following. "CVGIP, 1985, 32-46 ) carried out. Subsequently, fingertips become the hand 6 and thus recognize individual fingers. This is preferably done as a function of the detected curvatures of the hand contour, as for example in Malik ( Malik, Shahzad: "Real-time Hand Tracking and Finger Tracking for Interaction", Toronto: CSC2503F Project Report, University of Toronto, 2003 ) has been described. Alternatively, a hand modeling, as by Liu and Lovell (Liu, Nianjun and Lovell, Brian C. "Hand Gesture Extraction by Active Shape Models", Proceedings of Digital Image Computing on Techniques and Applications, 2005 ) to detect both the hand and its fingertips and fingertips ( Lee, J. and Kunii, TL: Model-Based Analysis of Hand Posture, IEEE Computer Graphics and Applications, 1995, 77-86 ). Based on the fingertips thus determined, again closed contours are calculated, with their centers of gravity representing the position of the final fingertip. If several false fingertips were detected by over-segmentation, they are discarded based on geometric heuristics and the distance to the hand contour center of gravity. As a result, individual fingertips of the hand can be easily achieved 6 determine.

Both the entire hand 6 and their fingertips can with known tracking methods, as for example by Do, Asfour and Dillmann ( Do, Martin; Asfour, Tamim; and Dillmann, Rüdiger: "Particle Filter-Based Fingertip Tracking with Circular Hough Transform Features" MVA, 2011 ). The detection of static pointers can be done by standardized classification methods and independently to each image of a sequence, as described by Lockton and Fitzgibbon ( Lockton, R. and Fitzgibbon, AW: "Real-time gesture recognition using deterministic boosting", BMVC, 2002 ) or Nagi ( Nagi, Jawad et al. "Max-Pooling Convolutional Neural Networks for Vision-based Hand Gesture Recognition" ICSIPA, 2011, 342-347 ). Probabilistic sequence modeling techniques allow both the detection of simple static ( Dreuw, Philippe; Keyser, Daniel; Deselaers, Thomas and Ney, Hermann: "Gesture Recognition Using Image Comparision Methods", International Workshop on Gesture in Human-Computer Interaction and Simulation, 2005, 124-128 ) as well as complex dynamic hand gestures ( Dreuw, Philippe; Rybach, David; Deselaer, Thomas; Zahedi, Morteza and Ney, Hermann. Speech Recognition Techniques for a Sign Language Recognition System, Interspeech, Antwerp, Belgium, 2007, 2513-2516 ), which do not necessarily expect an already segmented hand contour as an input, but work on the entire image using appearance-based features. Similar methods that can be used are based on an analysis of the optical flow ( Cutler, R. and Turk, M .: "View-Based Interpretation on Real-Time Optical Flow for Gesture Recognition", IEEE International Conference on Automatic Face and Gesture Recognition, Nara, Japan, 1998 ).

By the described method and the input device 2 Thus, the user will be able to easily find an analogous size of the motor vehicle 1 such as the volume of the entertainment system 8th to adjust or change the light intensity of an illumination and / or the speed of a windshield wiper. Which analogous size is changed, depending on the position of the hand 6 are decided in the detection range of the sensor and / or depending on a currently displayed context menu of the display unit 4 ,

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 20120105613 A1 [0005]
• EP 2441635 A1 [0005]

Cited non-patent literature

• Zivkovic, Zoran; "Improved Adaptive Gaussian Mixture Model for Background Subtraction", ICPR, 2004 [0031]
• Dadgostar, Farhad and Sarrafzadeh, Abdolhossein: "Adaptive real-time skin detector based on Hue thresholding: A comparison of two motion tracking methods" Pattern Recognition Letters, 2006, 1342-1352 [0032]
• Suzuki, S. and Abe, K .: Topological Structural Analysis of Digitized Binary Images by Border Following. "CVGIP, 1985, 32-46 [0033]
• Malik, Shahzad: "Real-time Hand Tracking and Finger Tracking for Interaction", Toronto: CSC2503F Project Report, University of Toronto, 2003 [0033]
• (Liu, Nianjun and Lovell, Brian C. "Hand Gesture Extraction by Active Shape Models", Proceedings of Digital Image Computing on Techniques and Applications, 2005. [0033]
• Lee, J. and Kunii, TL: "Model-Based Analysis of Hand Posture", IEEE Computer Graphics and Applications, 1995, 77-86 [0033]
• Do, Martin; Asfour, Tamim; and Dillmann, Rüdiger: "Particle Filter-Based Fingertip Tracking with Circular Hough Transform Features" MVA, 2011 [0034]
• Lockton, R. and Fitzgibbon, AW: "Real-time gesture recognition using deterministic boosting", BMVC, 2002 [0034]
• Nagi, Jawad et al. "Max Pooling Convolutional Neural Networks for Vision-based Hand Gesture Recognition" ICSIPA, 2011, 342-347 [0034]
• Dreuw, Philippe; Keyser, Daniel; Deselaers, Thomas and Ney, Hermann: "Gesture Recognition Using Image Comparision Methods", International Workshop on Gesture in Human-Computer Interaction and Simulation, 2005, 124-128 [0034]
• Dreuw, Philippe; Rybach, David; Deselaer, Thomas; Zahedi, Morteza and Ney, Hermann. "Speech Recognition Techniques for a Sign Language Recognition System", Interspeech, Antwerp, Belgium, 2007, 2513-2516 [0034]
• Cutler, R. and Turk, M .: "View-Based Interpretation on Real-Time Optical Flow for Gestural Recognition", IEEE International Conference on Automatic Face and Gesture Recognition, Nara, Japan, 1998. [0034]

Claims (10)

Method for operating an input device ( 2 ), in particular a motor vehicle ( 1 ), which by means of a non-contact sensor ( 3 ) a position and / or position change of at least one finger ( 9 . 10 . 11 . 12 ) of a hand ( 6 ) of a user, depending on the position and / or position change of the finger ( 9 . 10 . 11 . 12 ) is recognized and executed on an input, characterized in that in dependence on a detected reciprocation of the finger ( 9 . 10 . 11 . 12 ) an analogous size through the input device ( 2 ) is changed. Method according to claim 1, characterized in that the movement of the finger ( 9 . 10 . 11 . 12 ) to reversal points in which the finger ( 9 . 10 . 11 . 12 ) changes its direction of motion, is monitored to detect the float. Method according to one of the preceding claims, characterized in that the analogous quantity is changed by a predetermined value upon detection of each reversal point. Method according to one of the preceding claims, characterized in that the analogous size is dependent on the finger (s) performing the reciprocating motion ( 9 . 10 . 11 . 12 ) is increased or decreased. Method according to one of the preceding claims, characterized in that the analogous size is dependent on an attitude of the finger ( 9 . 10 . 11 . 12 ) hand ( 6 ) is increased or decreased. Method according to one of the preceding claims, characterized in that the analogous size is dependent on an orientation of the hand ( 6 ) is increased or decreased. Method according to one of the preceding claims, characterized in that the analogous size is dependent on a positioning of the hand ( 6 ) in a coverage area ( 7 ) of the sensor ( 3 ) is increased or decreased. Method according to one of the preceding claims, characterized in that a reciprocation of the finger ( 9 . 10 . 11 . 12 ) is detected to change the analogue size only if that finger ( 9 . 10 . 11 . 12 ) at a turning point on a thumb ( 13 ) of the same hand ( 6 ) meets. Method according to one of the preceding claims, characterized in that the analogous size is dependent on an orientation of the thumb ( 13 ) of the hand ( 6 ) is increased or decreased. Input device ( 2 ), in particular for a motor vehicle ( 1 ), with a non-contact sensor ( 3 ) for detecting a position and / or position change of at least one finger ( 9 . 10 . 11 . 12 ) of a hand ( 6 ) of a user, wherein the input device ( 2 ) is adapted, depending on the position and / or position change of the at least one finger ( 9 . 10 . 11 . 12 ) to recognize and execute an input, characterized in that the input device ( 2 ) performs the method according to one or more of claims 1 to 9 when used as intended.

Images