CN107256089B

CN107256089B - Gesture recognition method by natural image

Info

Publication number: CN107256089B
Application number: CN201710514833.3A
Authority: CN
Inventors: 杨恕先
Original assignee: Pixart Imaging Inc
Current assignee: Pixart Imaging Inc
Priority date: 2012-10-17
Filing date: 2012-10-17
Publication date: 2020-07-03
Anticipated expiration: 2032-10-17
Also published as: CN103778405A; CN103778405B; CN107256089A

Abstract

The present disclosure provides a gesture recognition method using a natural image, which generates a change image using two or more frames of images before and after the change image, calculates a frame characteristic value of the change image, and compares a change pattern of the frame characteristic value with a gesture definition to determine a gesture. The present invention has inherent resistance to image blur and supports X, Y, Z three-axis motion without fixed gestures.

Description

Gesture recognition method by natural image

Technical Field

The invention relates to the technical field of gesture recognition, in particular to a gesture recognition method by using natural images.

Background

Gesture control systems provide simple and intuitive operation convenience, but systems using a contact-type human-machine interface such as a touch panel limit the user to be able to operate against the interface, which is inconvenient for some applications. On the contrary, the gesture control system using the non-contact interface allows the user to operate at a relatively long distance, but the gesture must be determined by acquiring and recognizing the image, which is difficult. The current methods for recognizing gestures by images can be classified into two categories, one is to recognize gestures by using natural images without auxiliary light sources, and the other is to recognize gestures by using unnatural images generated by striking one or more auxiliary light sources.

Compared with a gesture control system using an auxiliary light source, the gesture control system without the auxiliary light source has the advantages of low cost, capability of being combined with a camera, power saving and the like, but has the inherent disadvantage of higher detection difficulty. The gesture recognition method commonly used in gesture control systems without auxiliary light sources includes movement detection and shape detection. Because of the different gestures and habits of different users, the gesture recognition method using motion detection has a low recognition rate for some gestures, such as pressing (click), zooming (zoom in/out), and other gestures involving Z-axis motion, whereas the gesture recognition method using shape detection generally requires the user to operate with a fixed, system-recognizable specific gesture, such as a fist, a palm, and the like. Fig. 1 and 2 are schematic diagrams of a gesture recognition method using shape detection, in which a camera module 10 acquires one frame (frame) image at intervals, if a user stretches and swings transversely in front of the camera module 10, i.e. moves in the direction of the X-axis and the Y-axis of the image, the camera module 10 acquires two frames of images, i.e. f (1) and f (2) in fig. 2, the positions of the

images

14 and 16 of the hand 12 in the frame are different, the system identifies an image having a predetermined shape, such as the

fingertips

18 and 20 of the index finger, from each of the frames of images f (1) and f (2), and determines that the gesture is a gesture swinging rightward according to the position difference between the

images

18 and 20 of the fingertips in the frame. This method requires a sufficiently sharp image to recognize a predetermined shape of the image, and is not resistant to a blurred image when moving rapidly, and thus is not suitable for short distance applications. If the user's hand changes during the operation process, the system cannot find the image with the predetermined shape, and the gesture recognition will also fail. Because some gestures are difficult to define due to the image being limited by recognizable shapes, there is a strong limit to the gestures that can be manipulated, and in general only gestures that produce distinctive images can be predefined in the system. In addition, since the image variation caused by the user moving the hand back and forth with respect to the camera module 10 is large, it is difficult to support the gesture of the Z-axis (longitudinal) motion.

The gesture recognition method using shape detection is to recognize a skin color range from an image, recognize a shape of the skin color range, and further find out the position of a hand in a frame of image. However, the skin color analysis requires a complicated algorithm, which is related to the color temperature, and has a high error rate, and the shape recognition also requires a complicated algorithm, and these recognition procedures require a large amount of calculation, so the cost of software and hardware is high, and the response of the system is slow.

Disclosure of Invention

One objective of the present invention is to provide a gesture recognition method using natural images.

One objective of the present invention is to provide a gesture recognition method with innate resistance to blurred images.

One of the objectives of the present invention is to provide a gesture recognition method beneficial to short-distance applications.

One of the objectives of the present invention is to provide a gesture recognition method for supporting X, Y, Z three-axis motion without requiring a fixed gesture.

One objective of the present invention is to provide a gesture recognition method that does not need to conform to a predetermined gesture shape.

One of the objectives of the present invention is to provide a gesture recognition method that is not affected by color temperature.

One objective of the present invention is to provide a gesture recognition method with less computation.

One objective of the present invention is to provide a gesture recognition method with low cost.

According to the present invention, a method for recognizing a gesture using a natural image includes generating images that are temporally sequential, selecting two or more frames of images from the images to generate a changed image, calculating a frame characteristic value of the changed image, and comparing a change pattern (pattern) of the frame characteristic value with a gesture definition to determine a gesture.

The method of the invention does not need to carry out image recognition and detect the position of an object, thereby avoiding various defects of the prior art.

Drawings

FIG. 1 is a schematic diagram of a laterally moving gesture;

FIG. 2 is a schematic diagram of a prior art gesture recognition method;

FIG. 3 is an embodiment of the present invention;

FIG. 4 is a schematic diagram of a gesture to detect lateral motion;

FIG. 5 is a schematic illustration of a gesture for rotational motion;

FIG. 6 is a schematic illustration of a gesture-generated pattern of changes in rotational movement;

FIG. 7 is a schematic illustration of a vertically moving gesture;

FIG. 8 is a schematic diagram of a gesture to detect vertical motion;

fig. 9a, 9b and 9c show different gesture-generated variation patterns.

Reference numerals:

10 Camera Module

12 hand

14 images of hands

16 images of hands

Partial image of 18 index finger

Partial image of 20 index finger

22 acquiring an image

24 selecting an image

26 generating a changing image

28 calculating picture characteristic value

30 gesture comparison

32 generating instructions

34 center of gravity of the changing image

36 center of gravity of changing image

38 images of the hand.

Detailed Description

Fig. 3 shows an embodiment of the present invention, as in the prior art, step 22 first obtains an image, for example, the camera module 10 shown in fig. 1 obtains one frame of image at intervals, thereby generating a sequence of images, step 24 selects two or more adjacent frames of images from the sequence of images, step 26 generates a variation image by using the previous and next frames of images, the variation image is a value calculated according to a predetermined formula, for example, a brightness variation of each pixel, and is used to represent a variation of the image on the time axis, step 28 calculates a frame characteristic value of the variation image, for example, a center of gravity, a standard deviation (standard deviation) or a variation (variance), step 30 compares a variation pattern (pattern) of the frame characteristic value with a predetermined gesture definition, if a gesture definition is met, step 32 generates a corresponding instruction, and then returns to step 24, otherwise go directly back to step 24. Because the method judges the gesture by the change of the full picture of the image without any image shape, the shape of the hand does not need to be recognized, and the position of the hand does not need to be found, therefore, the method has excellent resistance to the image blur, is not influenced by the shape or color temperature of the object, and the object is not limited to the hand. Since a clear image is not required, a gesture of a quick swing can be recognized, and thus, the method is applicable to short-distance applications. In addition, the method only needs a simple operation method, and the operation amount is less, so that the system has quick response and the required software and hardware cost is lower.

The present invention will be described in more detail below by taking detection of gestures such as slide (slide), rotation (rotate), and zoom (zoom) as examples.

Referring to fig. 1, when the hand 12 is swung transversely in front of the camera module 10, the camera module 10 generates a temporally sequential series of images, as shown in fig. 4, and generates a change image df (1,2) using two adjacent frames of images f (1) and f (2), for example, the previous image f (1) is subtracted from the subsequent image f (2), that is, df (1,2) ═ f (2) -f (1), and then calculates a frame characteristic value of the change image df (1,2), for example, a position 34 of the center of gravity, similarly, generates a change image df (2,3) using two adjacent frames of images f (2) and f (3), and calculates a position 36 of the center of gravity of the change image df (2,3), because the positions of the

images

14, 16, and 38 generated by the hand 12 in the frame are different, the

positions

34 and 36 of the center of gravity in the frame are also different. The position of the center of gravity of more varied images is obtained in this way, for example, as shown in the lower right of fig. 4, the variation pattern appears to move rightward, and if the variation pattern conforms to a predetermined gesture definition, it is determined that the variation pattern is the defined gesture, such as sliding.

Referring to fig. 5, when the user's hand 12 is stroked facing the camera module 10, the position of the center of gravity of the changed image has a changed pattern as shown in fig. 6, which can be used to generate a rotation command.

Referring to fig. 7, when the hand 12 of the user moves back and forth with respect to the camera module 10, the front and rear images obtained by the camera module 10 are shown as f (1) and f (2) in fig. 8, and the

images

14 and 16 generated by the hand 12 have the same or slightly different positions but different sizes in the frame, and such a gesture can be used to generate a zoom command. The images f (1) and f (2) are used to generate the changed images df (1,2), for example, if df (1,2) ═ f (2) -f (1), and the positions of the barycenter of the changed images obtained in this way have a constant or non-constant change pattern, but the size difference between the

images

14 and 16 is shown in other different screen feature values. For example, referring to fig. 9, a change image is generated by subtracting two frames of images from each other, and the center of gravity and the variation thereof calculated from the pixel coordinates thereof have a specific change pattern in different gestures. The variation pattern of the swipe gesture on the time axis is shown in fig. 9a, the average value is substantially constant, as shown by a curve 40, and the variation amount does not continuously and regularly change, as shown by a curve 42. The variation pattern of the rotation gesture on the time axis is shown in fig. 9b, and the average value and the variation amount thereof have continuous and regular forward and backward variations, as shown by the

curves

44 and 46. The variation pattern of the zoom gesture on the time axis is shown in fig. 9c, and the variation amount varies continuously and regularly, as shown by curve 50, but the average value lacks the corresponding variation, as shown by curve 48.

The calculation of the picture feature value of the image is a conventional technique, and any parameter or mathematical expression that can represent the change of the image can be used in the present invention.

In various embodiments, in addition to calculating the frame feature values of the changed image, the frame feature values may be subjected to a frequency analysis, for example, fourier transform is used to transform the movement of the image on the vertical axis and the horizontal axis from the time domain to the frequency domain to determine the change pattern. The pattern of changes used as a gesture definition may be determined by the system designer.

When calculating the changing image, the threshold value can be added to screen the pixels, and only the pixels with brightness change exceeding the threshold value are used to calculate the picture characteristic value, so as to improve the accuracy.

The foregoing description of the preferred embodiments of the present invention has been presented for purposes of illustration and description and is not intended to limit the invention to the precise form disclosed, and modifications and variations are possible in light of the above teachings or may be acquired from practice of the embodiments of the invention, which are presented for purposes of illustration of the principles of the invention and of teaching one skilled in the art to practice the invention in various embodiments and with the understanding that the present teaching is intended to be defined by the following claims and their equivalents.

Claims

1. A gesture recognition method using natural images is characterized by comprising the following steps:

step A: generating a sequence of images in time;

and B: generating a plurality of changed images by selecting two or more adjacent frames of images at a time from the images of the sequence;

and C: respectively calculating the plurality of change images to obtain a plurality of picture characteristic values; and

step D: comparing the change mode of the plurality of picture characteristic values along with the change of time with the gesture definition to judge the gesture;

the method judges the gesture according to the change of the full picture of the image along with the time, and the shape of the hand cannot be recognized.

2. The method of claim 1, wherein step B comprises subtracting temporally preceding and succeeding images.

3. The method of claim 1 wherein step B comprises using a threshold to screen pixels of the changed image.

4. The method according to claim 1, wherein the step C comprises calculating a center of gravity, a standard deviation or a variation of the change image as the frame feature value.

5. The method according to claim 1, wherein the step D comprises performing frequency analysis on the frame feature values to determine the change pattern.

6. The method of gesture recognition according to claim 1, wherein the gesture definitions include a swipe gesture, a rotate gesture, and a zoom gesture.