DE102014201313A1 - Method for detecting a movement path of at least one moving object within a detection area, method for gesture recognition using such a detection method, and device for carrying out such a detection method - Google Patents

Abstract

In a method for detecting a movement path (223, 221, 222) of at least one moving object (23, 24) within a detection area, this is first detected by imaging. Here, a first detection image (11) is generated, which reproduces the detection area at a first detection time. At a later time, a second detection image (12) is generated, which shows the detection area at a corresponding later detection time. Subsequently, correspondences of image areas (13 to 22) of the acquisition images (12) are determined and evaluated. For this purpose, the acquisition images (12) are compared, a distribution density of image areas (13 to 23) corresponding to their change in position in the acquisition image (12) is determined, and the determined distribution density is evaluated. Pixels (23, 24) with associated motion increment (223, 221, 222) are then assigned between the acquisition images (12) on the basis of the evaluated distribution density. This is followed by a pixel motion evaluation. This method, as well as a gestikerkennungsverfahren using this can be realized with simple optical means.

Description

The invention relates to a method for detecting a movement path of at least one moving object within a detection area. Furthermore, the invention relates to a method for gesture recognition using such a recognition method and to an apparatus for carrying out such a recognition method or gesture recognition method.

Methods and apparatus for object recognition are known from the WO 2012/095258 A1 and the WO 2013/020872 A1 ,

It is an object of the present invention to provide a detection method with simple optical means.

This object is achieved by a detection method with the features specified in claim 1.

The invention described measures a distribution density of correspondences between parts of successive images. For this purpose correspondences are determined between image sections of two successive images. Correspondence is given when two parts of the picture are similar. In this preprocessing, correspondences are also permitted which do not correspond to the optical flow. With the method, distributions of correspondence vectors of different direction and length are produced for small image regions, thus correspondence distribution profiles over the entire image. These correspondence distribution profiles are converted into a correspondence distribution density. The image flow then corresponds to the largest values of an ideal correspondence distribution density. Due to measurement errors, image repetitions and measurement gaps, the ideal correspondence distribution density of an optical flow is generally not achieved. The preprocessing method can therefore be characterized as flow-oriented examination (flox).

A subset of such correspondence distribution densities is the optical flow. By allocation errors, the distribution density will find a variety of other correspondences. The distribution densities are checked for potential movements of compact regions and motion increments are derived. From a sequence of motion increments object movements are concatenated and made plausible. Correspondences between similar pixels or image parts that are not images of the same object, e.g. Correspondences between two adjacent folders, lead to a pseudo-movement, which usually does not continue locally, but remains local, comparable to the speedometer with spinning tires. By comparing more than two acquisition images taken in succession, such apparent movements can be excluded. The concatenation of plausibilized motion increments then leads to a movement, which in turn is checked for a gesture.

If objects in the vicinity are to be measured, such as e.g. Fingers directly in front of the camera, which is installed in a pair of glasses, will cause several image areas to move in a similar, ie common, movement. In this case, an appropriate average of the coordinates of the common motion is used to represent the actual object. As a rule, it is not the position of the object that determines the motion gesture, but the shape of the path, which in this case is identical for all common movements.

In addition to an average value, the supreme of all pixels traversing a common path can also be selected and assigned. This is e.g. the fingertip of an upward pointing finger in the picture.

The accuracy of the web has to be so good that it is possible to differentiate the web shapes assigned to the gestures.

In the case of imaging, camera images can be cyclically loaded into an evaluation computer.

The temporal distance of the pictures may vary, but must be known. From two successive images, a correspondence distribution density is determined from which movement increments are calculated per image pair. From the sequence of motion increments, motion sequences are filtered that can correspond to selected gesture movements. The number of incorrect correspondence distribution densities can be reduced by coarse distance knowledge, by suitable depth sensors or by sharpness adjustments of the camera or flash lighting, in order to increase the security with the recognition.

When pixel mapping is done no object shape detection. It is checked where in corresponding pixel groups or image areas movements, in particular fast movements, with high density, ie movements of pixel groups with comparable movement increments, detected. From a detected pixel group, the selection and assignment of a representative pixel takes place on the basis of previously defined criteria for the determined distribution density and the associated movement increments. It For example, a minimum density of moving pixels can be specified, and it is possible to make a selection for the largest movement increment among the then preselected pixels which lie within the pixel groups with the minimum density. Alternatively, it is possible to preselect according to certain movement increments and, within a pixel group having this movement increment, a pixel which is excellent in terms of its position can be selected within this pixel group. In the selection or assignment of the pixel by evaluation of the determined distribution density, a prediction algorithm can simplify an assignment of a specific pixel. For this purpose, it is checked on the basis of, for example, three successive acquisition images, if the last captured acquisition image is a candidate pixel in an image area in which it can actually be expected according to its movement in the first two successive acquisition images. Only pixels in which a predicted image area is reached then correspond to the prediction and thus fulfill this selection criterion. As far as several pixels remain after passing through these different selection criteria, a simple geometric selection can be made. For example, it is possible to select and assign an uppermost pixel which is present in the detection area among the selection candidates. The image areas may be individual pixels or pixel groups.

The definition of a depth range according to claim 2 can be carried out with the aid of a depth sensor. For this purpose, the depth of field of a front optics of the camera sensor can be used. Autofocus techniques can also be used for this purpose, which can be used in particular for contrast enhancement and thus for improving the result of a comparison of the acquisition images. As soon as a depth of an object whose trajectory has been detected is known, the object speed can also be measured and specified for the object movement.

Accordingly, the correspondence distribution density can be determined not only from objects in the distance of the expected object but also from objects less or further away from the sensor. With coarse-resolution depth sensors, based on structured light, time-of-flight or even stereoscopy, image parts can be identified that are not in the distance range and whose distribution densities are ignored. A depth sensor based on structured light (Structured Light) is known, for example from the US 4,954,962 , A depth sensor based on "Time-of-Flight" is known from the EP 2 378 310 A1 ,

Coarser resolutions offer, for example, ultrasonic sensors. Through a combination of some ultrasonic sensors, the directions of objects that are within the expected distance can be determined and other image areas can be discriminated.

Distance information is also provided by so-called light field sensors.

If no sharpness can be calculated for the desired distance in certain image areas, these too will be discriminated.

A depth range definition according to claim 3, provided the presence of a suitably controllable light source, with high precision possible. Alternatively or additionally, for the definition of the depth range, a temporal variation of an illumination period at an exposure time in the imaging acquisition can also take place.

In addition to the direct depth sensors so also further distance-dependent effects can be used. If an IR filter is placed in front of the camera and the environment is irradiated with limited IR light output, the range is limited and correspondence of underlying objects is no longer detected. If objects are very close, they are so strongly illuminated by the IR radiation that no contrasts are recognizable on them. This creates a depth range for measurable correspondences. If the IR radiation line and the exposure time are varied in a short time sequence, measurable depth ranges can be offset in such a way that only chains of movement increments can be made plausible by the objects that have remained throughout the measurable areas.

Another distance-dependent effect is the depth of field. For high-intensity lenses, the depth of field is less than with low-focussed lenses. Only in this area can correspondences be measured. By varying the focal length in a short time sequence, the measurable depth range can be shifted so that only chains of movement increments can be made plausible by the objects that have remained consistently within the measurable range.

The combination of the two distance-dependent effects together with their temporal variation leads to the desired effect of limiting the measurable range. It is therefore also possible to use combinations of depth of field and / or illuminance and / or illumination duration for the definition of the depth range.

The advantages of a gesture recognition method according to claim 4 correspond to those which have already been explained above with reference to the movement path recognition method.

Gestures are created by the movement of body parts. Immediate measurement of motion does not require modeling, such as images of hands or joint models. If the movement of compact, for example, fist-sized, objects measured directly, can be dispensed with the modeling, for example, a hand pose or joint models. In a monocular camera system, the fist-sized object should tend to be moved transversely to the viewing direction of the sensor. Together with a suitable depth sensor, it is also possible to directly measure removal speeds to the sensor. In both cases, however, neither hand poses have to be trained or joint models with an essentially undisturbed environment must be taken into account. In addition, the smaller the object-to-detection-area ratio, the smaller the demands on texture information to the object, so that many independent objects in a large volume can be measured.

The derived gestures may be further plausibilized via the use of known methods such as "Inverse Kinematics" or "Template Matching". The movement must have been triggered by a certain object-like gray value distribution. A hand, fingers as well as artificial objects (gloves, markers) can serve as a basis. With "Inverse Kinematics" motion forecasts can be made and thus the correspondence density distribution can be evaluated more specifically. Simplified, for example, planar, motion models such as the constant velocity model can also better evaluate the correspondence density distribution.

An "inverse kinematics" method is known from the CA 2 211 858 C , A "template matching" method is known from the EP 1 203 344 B1 ,

Model pixel movements according to claim 5 result in a gesture set that can be used for a variety of control tasks.

The specification of an input area with an area specification gesture according to claim 6 makes it possible to define a sub-area which can be detected, for example, with high resolution, within the detection area, which can be used for detailed input purposes. The divisional gesture may be a circular motion. You can then make further entries in the defined input area.

By a face recognition according to claim 7, a release of the gesture recognition can take place. Face recognition can identify a person in the environment of the movement. It can then be ensured that only certain people have access.

The selection of a user profile is possible by a selection of provided model pixel movements according to claim 8.

The advantages of a device according to claim 9 correspond to those which have already been explained above in connection with the trajectory recognition method and the gesture recognition method.

The device may include a light source which is in signal communication with the camera sensor and / or the evaluation computer so that the light source, for example an exposure intensity or an exposure period, can be preset by the camera sensor and / or the evaluation computer by appropriate control.

By means of a projector device according to claim 10, an input field or a plurality of input fields, for example in the form of a keyboard, can be generated by projection, for example in a predetermined input area. By actuating the at least one projected input field, the user can then trigger a defined control action or also make an input, for example a yes / no selection or a text input.

An embodiment of the invention will be explained in more detail with reference to the drawing. In this show:

1 very schematically an apparatus for carrying out a method for detecting a movement path of at least one moving object within a detection area as part of a Gestikerkennungsverfahrens;

2 and 3 Snapshots of the capture area reflecting capture images at two consecutive capture times.

1 schematically shows a device 1 to carry out a recognition process. With the device can be a trajectory 2 at least one moving object 3 within a coverage area 4 , the Indian 1 dashed lines are recognized. As an example of the trajectory 2 is in the 1 the path of a moving hand of the object 3 illustrated by the example of a gesticulating user.

The device 1 has a monocular camera sensor 5 , which is a high-resolution CCD camera or CMOS camera with an optical attachment 6 which is capable of a predetermined depth or a depth range T of the detection range 4 to capture with given image sharpness.

Via a signal line 7 is the camera sensor 5 with an evaluation computer 8th in signal connection. The latter is over another signal line 9 with a device to be controlled 10 in signal connection. Alternatively, it may be the evaluation computer 8th and the device to be controlled 10 to act one and the same unit. In the device to be controlled 10 It can be a kind of tablet PC, equipped with the components 5 and 8th act for gesture recognition. Alternatively, the device to be controlled may be 10 also in relation to the evaluation computer 8th act external device, such as a TV or other device of consumer electronics. A home automation device, such as a lighting system or a shutter control or a heating system is an example of the device to be controlled 10 ,

For recognizing the trajectory 2 becomes the detection area 4 with the camera sensor 5 imaged. Here, a the detection area 4 reproducing acquisition image in the camera sensor 5 generated.

2 and 3 show by way of example two such detection images 11 and 12 at the times t = 0 and t = 1, using arbitrary time units as an example. The entry screen 12 is a delay period later from the camera sensor 5 generated as the capture image 11 , The two entry pictures 11 and 12 are digitized in real time or quasi in real time and in the evaluation computer 8th deposited. In the evaluation calculator 8th then finds a determination and evaluation of correspondences of image areas of the acquisition images 11 . 12 instead of. For this, the capture images 11 and 12 in the evaluation calculator 8th compared to each other. A distribution density of image areas corresponding to their change in position in the acquisition image is then determined.

The delay period, ie a time interval between the acquisition times of the acquisition images 11 and 12 can be variable. The delay period can be in the range between 10 ms and 1 s.

In the 2 and 3 are such image areas by small squares 13 to 22 exemplified. These image areas may be individual pixels or groups of pixels.

The entry pictures 11 and 12 show as a pixel groups that represent detected objects, for example, a raindrop 23 for example, on a camera lens of the camera sensor 5 may be present, as well as a hand 24 the user 3 , Both the raindrop 23 as well as the hand 24 have between the two detection times t = 0 and t = 1 of the acquisition images 11 and 12 emotional.

When determining the distribution density by comparing the acquisition images 11 and 12 it turns out that in the area of raindrops 23 virtually all pixels along a comparable trajectory 2 ₂₃ , which is in the 3 based on the movement of the top image area 13 is illustrated. Here is the position of the image area 13 at the time of acquisition t = 0 in the 3 shown in dashed lines.

The 3 shows a typical result when evaluating the determined distribution density by a corresponding evaluation algorithm. For example, in the correspondence determination to the hand 24 assigned image areas 19 to 22 actually have correct correspondences (movement of image areas 21 and 22 ) as well as actually wrong correspondences (movement of the picture areas 19 and 20 ). Together with others, the hand 24 assignable image areas that in the 2 and 3 are not shown, there is an increased distribution density of in terms of their change in position in the acquisition image 12 with the image areas 21 and 22 corresponding image areas.

Also shown are not further captured pixel groups in the capture images that are rendered physically as objects 11 and 12 through associated image areas 14 to 18 as well as the corresponding image areas resulting from the evaluation of the distribution density 14 to 18 after the delay period, ie at time t = 1 (cf. 3 ). From the resulting trajectories or movement increments 2 _i (i = 13 to 22) of the image areas 13 to 22 can be concluded in the evaluation of each associated associated distribution density of the image areas, whether the resulting trajectories 2 _i may or may not be real.

The result of the evaluation is an assignment of individual pixels from pixel groups evaluated with regard to their distribution density with associated movement increment between the acquisition images 11 . 12 due to the evaluated distribution density. The result of the evaluation of the acquisition images 11 and 12 results in each assigned pixels for the objects "raindrops" and "hand" with the actual trajectories 2 ₂₃ for the raindrop 23 such as 2 ₂₁ and 2 ₂₂ for the hand 24 ,

The assigned pixels 13 . 21 . 22 and the associated movement increments 2 ₂₃ as well 2 ₂₁ , 2 ₂₂ assigned pixel movements can then be evaluated.

In determining the distribution density, it may be possible to detect selected portions of the acquisition images 11 . 12 done in the capture images 11 . 12 differ. In the area of raindrops 23 and in the field of the hand 24 Thus, a higher-resolution determination and evaluation of correspondences of the image areas can take place. When evaluating the distribution density, methods of averaging and statistical methods can be used.

The determination and evaluation of correspondences can, of course, be carried out on the basis of a sequence of individual images of a larger number, for example using a sequence of three, four, five, six, eight, ten, twenty-five, fifty, one hundred or even more individual images.

For the pure recognition of the trajectory 2 no gesture model is required.

As with the example "Raindrops 23 "And" hand 24 "Already demonstrated, the recognition method allows detecting the trajectories of several independent objects. These can also be more than two independent objects (for example, three, four, five, ten, or even more independent objects).

During the imaging acquisition of the detection area 4 can a predetermined depth range T, ie a range of predetermined distances, within the objects, so for example the user 3 to be defined. As a depth range, for example, a distance range from the camera sensor 5 between 0.5 m and 3 m or between 1 m and 2.5 m. Also, a more tolerant or more specific specification of a depth range is possible. The definition of the predetermined depth range can be done by means of a depth sensor. This technique can be used, which are known under the keywords "Structured Light", "TOF". A stereobildgebendes method with two camera sensors can be used to define the depth range. For this purpose, a light field can be used or it can be used ultrasound or radar radiation. Also the depth of field of the intent optics 6 can be used to define the depth range T. In this case, for example, autofocus techniques can be used. Once using such a method, the depth of the detected object 3 that is its distance to the camera sensor 5 is known, can be after detecting the trajectory 2 also measure and indicate a speed of the object detected in its motion.

The definition of the depth range can also be achieved by adjusting an illuminance of an illumination of the detection area by means of a light source 25 take place at an exposure time during the imaging. The light source 25 is via a signal connection, not shown, with the camera sensor 5 and / or the evaluation computer 8th in signal connection. As an alternative or in addition to an illuminance setting, a temporal variation of an illumination period in the illumination with the light source can also be used to define the depth range 25 in relation to the exposure time of the camera sensor 5 used in imaging.

The above-described trajectory recognition method can be used within a method of gesture recognition.

Here, a plurality of model pixel movements or model object movements are provided as control symbols, and these model pixel movements are compared with the pixel movements that have been evaluated by the movement path recognition method. Subsequently, the model pixel movement is identified as a selected control symbol, which has the greatest agreement with the evaluated pixel movement. Finally, a control action associated with the selected control icon is performed. In this gesture recognition technique, techniques known in the art as "template matching" and "inverse kinematics" may be used.

• – Bewegung über den Erfassungsbereich von links nach rechts;
• – Bewegung über den Erfassungsbereich von rechts nach links;
• – Bewegung über den Erfassungsbereich von oben nach unten;
• – Bewegung über den Erfassungsbereich von unten nach oben;
• – Bewegung über den Erfassungsbereich in Herzform;
• – Bewegung über den Erfassungsbereich in Z-Form;
• – Bewegung über den Erfassungsbereich in Kreisform;
• – Schließen einer offenen Hand zu einer Faust;
• – keine Bewegung.

The model pixel movements may include at least one of the following motion patterns:

• Movement across the detection area from left to right;
• Movement across the detection area from right to left;
• Movement over the detection area from top to bottom;
• Movement over the detection area from bottom to top;
• - movement over the detection area in heart shape;
• Movement over the detection area in Z-shape;
• - movement over the detection area in circular form;
• - closing an open hand into a fist;
• - no movement.

The control action can be the specification of an input area 26 within the entire coverage area 4 by a range setting gesture. This range setting gesture may be performed, for example, by a circular motion of an open or closed hand. The person 3 can thereby be within the entire coverage 4 the input area 26 then the camera sensor 5 recorded in high resolution. For this purpose, the attachment optics 6 be executed for example as zoom optics. In the then defined input area 26 then further, more detailed inputs can be made. Within the coverage area 26 For example, by a corresponding projection technique using a projection module or a projector device 27 an input raster are projected, such as a keyboard layout. The user can then enter one in the coverage area 4 with the projector device 27 operated keyboard, which in turn from the camera sensor 5 is detected, recognized and evaluated.

In the gesture recognition, facial recognition may be performed prior to the comparison step, which is a prerequisite for performing the further steps of gesture recognition. Depending on the result of the face recognition, a selection of the provided model pixel movements can be made. As a result, a profile of model pixel movements can be assigned to the user respectively recognized via the face recognition. So you can specify user profiles.

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• WO 2012/095258 A1 [0002]
• WO 2013/020872 A1 [0002]
• US 4954962 [0014]
• EP 2378310 A1 [0014]
• CA 2211858 C [0025]
• EP 1203344 B1 [0025]

Claims (10)

Method for detecting a trajectory ( 2 ; 2 _i ) at least one moving object ( 3 ; 23 . 24 ) within a coverage area ( 4 ) comprising the following steps: - imaging acquisition of the detection area ( 4 ) and generating a coverage area ( 4 ) reproducing the first acquisition image ( 11 ) at a first detection time, - imaging detection of the detection area ( 4 ) and generating a coverage area ( 4 ) second capture image ( 12 ) to a second detection time, later by a delay period, - determination and evaluation of correspondences of image areas ( 13 to 22 ) of the acquisition images ( 11 . 12 ) comprising the following steps: - comparing the acquisition images ( 11 . 12 ), - determining a distribution density of in terms of their position change in the entry screen ( 11 . 12 ) corresponding image areas ( 13 to 22 ), - evaluating the determined distribution density, - assigning at least one pixel of a pixel group ( 23 . 24 ) and / or an image area ( 13 to 22 ) with associated movement increment ( 2 ₂₃ ; 2 ₂₁ , 2 ₂₂ ) between the acquisition images ( 11 . 12 ) on the basis of the evaluated distribution density, - evaluation of the assigned pixel and its movement increment ( 2 ₂₃ ; 2 ₂₁ , 2 ₂₂ ) assigned pixel movements. A method according to claim 1, characterized in that during the imaging acquisition of the detection area ( 4 ) is defined a predetermined depth range (T), ie a range of predetermined distances within which pixels are to be detected. A method according to claim 2, characterized in that a definition of the depth range (T) by adjusting an illuminance of an illumination of the detection area to an exposure time in the imaging acquisition takes place. Method for detecting a gesture using a recognition method according to one of Claims 1 to 3, with the following further steps: Providing multiple model pixel motions as control symbols, Comparing the pixel movements evaluated with the recognition method with the model pixel movements, Identifying as the selected control symbol the model pixel movement which has the greatest agreement with the evaluated pixel movement; - Perform a control action associated with the selected control icon. A method according to claim 4, characterized in that the model pixel movements comprise at least one of the following movement patterns: movement over the detection area from left to right; Movement across the detection area from right to left; Movement over the detection area from top to bottom; Movement over the detection area from bottom to top; - movement over the detection area in heart shape; Movement over the detection area in Z-shape; - movement over the detection area in circular form; - closing an open hand into a fist; - no movement. Method according to Claim 4 or 5, characterized in that the control action specifies the specification of an input range ( 26 ) within the scope ( 4 ) by a range setting gesture. Method according to one of claims 4 to 6, characterized in that prior to comparing a face recognition. A method according to claim 7, characterized in that depending on the result of the face recognition, a selection of the provided model pixel movements takes place. Contraption ( 1 ) for carrying out a recognition method according to one of claims 1 to 8 - with a monocular camera sensor ( 5 ), - with an evaluation computer ( 8th ) connected to the camera sensor ( 5 ) in signal connection ( 7 ) stands. Apparatus according to claim 9, characterized by a projector device ( 27 ).

Images