KR101385373B1 - Method for face detection-based hand gesture recognition - Google Patents

Abstract

The present invention relates to a method for identifying a hand gesture based on face detection. The present invention comprises: a step of detecting a face; a step of modeling skin colors by detecting the face; a step of calculating information on the distance from a sensor; and a step of detecting a hand area by using the skin color model and the information on the distance. According to the present invention, a hand gesture identification shows a high accuracy rate even in case the background color is similar to the skin color. [Reference numerals] (S10) Detecting a face; (S20) Modeling skin colors; (S30) Calculating the distance from a sensor; (S40) Detecting a hand; (S50) Detecting a gesture area; (S60) Recognizing a gesture

Description

{Method for face detection-based hand gesture recognition}

The present invention relates to a face detection based hand gesture recognition method in color and distance images. In particular, the present invention relates to real-time skin color modeling through face detection and distance information to detect a hand region and recognize a direction and one gesture according to hand movement. It relates to a rule-based recognition method.

In general, hand gesture recognition is an active area of research in the field of computer vision. Recently, research is being conducted using real-time distance information provided by Kinect of Microsoft Corporation.

Many input interfaces using hand gestures have been studied, but most posture-based studies have been conducted to detect the shape of a hand by detecting a hand region. In addition, many coordinate-based methods for recognizing gestures by tracking hand movements have been studied. However, most of them have limited tracking of palm or specific hand movements and recognize them using Hidden Markov Model (HMM). There is also a method of recognizing a gesture by detecting a hand region based on a skin color model using a 2D camera, but it has a disadvantage in that it is difficult to detect a hand region when it is sensitive to lighting and the background and the skin color are similar.

Accordingly, the present invention has been made to solve the problems of the prior art as described above, an object of the present invention is to provide a method for recognizing a hand gesture by combining the skin color model and the distance information of the method using a 2D camera.

Another object of the present invention is to provide a method for setting a gesture section using a difference in hand coordinates in an existing frame and a current frame instead of using hand coordinates.

According to a feature of the present invention for achieving the above object, the present invention is a face detection step, skin color modeling step through the face detection step, calculating distance information with the sensor, and using the skin color model and the distance information Hand region detection step.

As described above, according to the configuration of the present invention, the following effects can be expected.

First, since the skin color model is generated in real time through the face detection at the gesture point of view compared with the conventional general skin color model, there is an effect of preventing the recognition error caused by the individual differences or the degree of illumination of the skin color.

Second, since the hand is detected by combining distance information with the skin color model, the recognition rate of the hand gesture is very precise.

Third, since the gesture is recognized through the coordinate difference rather than the coordinate change, the rule for recognizing the gesture is greatly simplified.

1 is a conceptual diagram illustrating a non-contact smart device control system according to the present invention;
2 is a flowchart illustrating a face detection based hand gesture recognition method according to the present invention.
3A is an image of a face detection result using Har-like features according to the present invention.
Figure 3b is a skin color detection image using Cb, Cr two colors and GMM according to the present invention.
4A to 4D are graphs showing difference values of x and y coordinates for each gesture according to the present invention.
Figure 5 is a graph comparing the recognition rate of each gesture 2D and 3D camera according to the present invention.

Brief Description of the Drawings The advantages and features of the present invention, and how to achieve them, will become apparent with reference to the embodiments described in detail below with reference to the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Is provided to fully convey the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims. The dimensions and relative sizes of layers and regions in the figures may be exaggerated for clarity of illustration. Like reference numerals refer to like elements throughout the specification.

Embodiments described herein will be described with reference to plan views and cross-sectional views, which are ideal schematics of the present invention. Thus, the shape of the illustrations may be modified by manufacturing techniques and / or tolerances. Accordingly, the embodiments of the present invention are not limited to the specific forms shown, but also include changes in the shapes that are produced according to the manufacturing process. Thus, the regions illustrated in the figures have schematic attributes, and the shapes of the regions illustrated in the figures are intended to illustrate specific types of regions of the elements and are not intended to limit the scope of the invention.

Hereinafter, a preferred embodiment of a face detection based hand gesture recognition method according to the present invention having the above configuration will be described in detail with reference to the accompanying drawings.

Referring to FIG. 1, the non-contact smart device control system 100 of the present invention analyzes a detection signal transmitted from the detection unit 110 and the detection unit 110 to detect a hand gesture by non-contact. Recognizing unit 120 for generating the recognition information, the control unit 130 for controlling the video 142, the audio 144 or the on / off 146 of the smart device using the recognition information.

The sensing unit 110 includes at least one sensing device capable of sensing a hand gesture to perform a user interface in a non-contact manner. The sensing device may include a scanner or a camera (including a normal camera as well as a depth camera) (hereinafter, the sensing device is referred to as a “camera” for convenience). The detector 110 acquires a non-contact detection signal generated from the camera and transmits the signal to the recognizer 120. Here, the non-contact detection signal is a signal according to the response intensity of the sensor by the up, down, left, right, and one gesture of the hand. Thus, the sense signal of the present invention includes a signal according to a hand gesture. The detector 110 detects a signal for controlling an operation such as a hand gesture signal and transmits the signal to the recognizer 120.

The recognition unit 120 analyzes the noncontact sensing signal transmitted from the sensing unit 110 to generate recognition information. For example, the non-contact recognition information may be generated by estimating the movement direction (up, down, left, right, and the like) of the hand gesture and the speed of the hand gesture using previously stored pattern information and image analysis information. The non-contact recognition information generated by the recognition unit 120 is transmitted to the control unit 130 as described above. For example, when the hand moves up or to the right, it is possible to generate recognition information and transmit the recognition information to the controller 130 in order to satisfy the pre-stored perspective information.

When receiving the non-contact recognition information, the control unit 130 starts operation control. For example, the video 142 may be manipulated, the audio 144 may be changed, or the smart device may be turned on / off 146 according to the content included in the received contactless recognition information.

Hereinafter, a face detection based hand gesture recognition method in a non-contact smart device control system will be described in detail with reference to the accompanying drawings.

Referring to FIG. 2, in the face detection based hand gesture recognition method of the present invention, a face detection step (S10), a skin color modeling step (S20) through the face detection, a distance information calculation step (S30) with the sensor, and the skin color model And a hand region detection step S40, a gesture section detection step S50, and a gesture recognition step S60 using the distance information.

First step: face detection step ( S10 )

In an embodiment of the present invention, the feature is that facial detection is used to generate a skin color model. The method based on skin color model has the disadvantage of being sensitive to lighting. Therefore, to compensate for this, the face is first detected before generating the model in real time using the skin color at the time of gesture use.

Face detection may use Haar-like features. For example, after extracting feature points using a Haar-like, a face shape may be detected after generating a face shape using an Adaboost learning algorithm. In addition, the face may be detected by various methods using pixel brightness values. FIG. 3A shows an image of a face detection result using such Haar-like features.

Step 2: skin color modelling  step( S20 )

The face detection area not only has skin color but also contains areas that are too light or too dark for skin color such as eyebrows and lips. The skin color model is generated by removing a color other than the skin color from the face and converting it to a Luma-Blue Chroma-Red Chroma (YCbCr) color model. Therefore, skin color modeling uses two colors, blue (Cb) and red (Cr), and a skin color model is generated using a Gaussian Mixture Model (GMM). 3b shows skin color detection images using Cb and Cr two colors and GMM.

Step 3: Measure the distance to the sensor ( S30 )

Using the skin color model generated through face detection, the skin color region can be effectively detected, but as shown in FIG. 3B, the skin color region shown in the background (for example, a paper box of skin color) cannot be easily distinguished from the hand region using only the skin color model. There are disadvantages. Therefore, a sensor for distance estimation can be used to accurately distinguish this. The sensor may comprise a depth sensing camera. For example, a depth camera may use Microsoft's Kinect sensor. Depth cameras can calculate the distance information of the face, background, and hands by acquiring the rays that are returned by illuminating the laser or infrared light on the face, background, and hands, respectively. However, the Kinect sensor provides not only the distance but also the position of the joint. Since the frame processing speed decreases due to the joint position detection calculation time, it is not suitable for real-time processing. Can be used.

Step 4: Detect Hand Area ( S40 )

If a person makes a gesture, the hand area can be defined as shown in the equation below using distance information on the assumption that the hand moves in front of the face.

(1)

(One)

Here, Dface represents the distance between the sensor and the face, and through the experiment, the hand region can be extracted using a t value of 0.1.

In order to extract the center point of the hand, the detected hand region is binarized and then projected to Px and Py on the two-dimensional axis, respectively, and the center points x 'and y' of the hand can be obtained as shown in the following equation.

(2)

(2)

(3)

(3)

Where w and h are the horizontal and vertical dimensions of the image, and N is the sum of the coordinate axes projected onto the x and y axes.

Step 5: Gesture section detection step ( S50 )

The moving section setting for gesture recognition is a very important part, and how to set this section greatly changes the recognition rate. It is defined that the hand movement difference between frames increases and decreases during natural gesture operation. The movement difference of the hand is calculated with the center coordinates of the hand area of the previous frame and the current frame. The start and end of the gesture may be determined using the obtained coordinate difference value, and may be used as recognition information of the gesture.

For example, when a hand movement occurs, a difference value between the hand center coordinates of the previous frame and the hand center coordinates of the current frame is inserted into a queue of size N. If the difference in coordinates gathered in the queue increases and becomes more than a certain amount of variance, it is considered to be the beginning of the gesture. At this time, the coordinate difference list between the start and end of the gesture is defined as βx and βy as follows, and gesture recognition is performed.

(4)

(4)

(5)

(5)

Here, n is the number of frames of the section in which the hand moved.

Step 6: Gesture Recognition Step ( S60 )

Recognition up, down, left, and right uses the difference value of the coordinates of the center of the hand. The ratio of the variance of βx and βy of the horizontal and vertical axes of the hand movement is calculated and determined. Equation (6) below is used for discriminating the top, bottom, and (7) left and right.

(6)

(6)

(7)

(7)

Here, sigma 2 (βx) and sigma 2 (βy) represent the variances on each axis, and θ₁, θ 2 represent the threshold of the dispersion ratio. If the above equations (6) and (7) are satisfied, the sign of the sum of the coordinates of each set of αx and βy is +, and it is recognized as down (6), left (7), and-if up (6), right (7) Recognize.

4A to 4D are graphs showing difference values of x and y coordinates for each gesture. Blue is x, and red is y. However, in FIG. 4A and FIG. 4B, the change in the difference in the y coordinate is severe, and in FIG. 4C and FIG. 4D, the change in the difference in the x coordinate is severe.

In addition, the circle recognition recognizes when the number of frames of the gesture section is n, and when n is a predetermined value or more. In the case of circle recognition, the coordinate difference cannot be used, so the hand center coordinate value is used as it is. If the radius information is obtained as follows and the equation (10) is satisfied, it is recognized as a circle.

(8)

(8)

(9)

(9)

(10)

(10)

Here, x and y represent each coordinate value.

Hereinafter, the experimental results of the recognition rate of the face detection based hand gesture will be described.

The experiment was performed by installing OpenCV 2.2 on Intel Core2 Duo 3.0GHz, Memory 3G, Kinect, and Visual C ++ 2010 on Windows 7. A total of five experiments were conducted with 10 gestures of 5 gestures each, and the recognition rate was 97%. Table 1 below shows the number and recognition rate of gesture recognition for each experimenter.

left Right side top bottom won Experimenter 1 10 10 9 10 10 Experimenter 2 10 9 10 10 9 Experimenter 3 9 10 10 10 10 Experimenter 4 10 10 10 9 9 Experimenter 5 10 9 10 10 10 Recognition rate 98% 96% 98% 98% 96%

There seems to be no significant difference between the experimenters here. For comparison, we experimented with Microsoft LifeCam Cinema under the same conditions using a 2D camera without using distance information. 5 is a graph comparing recognition rates of 2D and 3D cameras for each gesture. Here, the method using the 2D camera showed a lower recognition rate than the method using the 3D camera except for the above gesture, and the overall recognition rate was 94%, which is about 3% lower than the method using the 3D camera.

As described above, it can be seen that the present invention has a technical concept that a skin color is modeled in real time through face detection and combined with distance information to detect a hand region. Many other modifications will be possible to those skilled in the art, within the scope of the basic technical idea of the present invention.

110: detection unit 120: recognition unit
130:

Claims (10)

Detecting a face at the point of hand gesture;
Detecting a hand based on the face detection; And
Recognizing the hand gesture,
Detecting the hand,
The skin color model is generated in real time through the face detection at the time of the hand gesture, and the hand region is extracted using the skin color model.
Before the step of detecting the hand, further comprising obtaining distance information using a depth camera,
Detecting the hand,
And extracting the hand region by combining the skin color model and the distance information. delete delete The method of claim 1,
Wherein the step of recognizing the hand gesture comprises:
A face detection-based hand gesture recognition method using a difference value between a hand center coordinate of a previous frame and a hand center coordinate of a current frame. Face detection step;
Skin color modeling through face detection;
Calculating distance information with the sensor; And
Detecting a hand region using the skin color model and the distance information,
The hand region detection step,
Face detection based hand gesture recognition method, characterized in that for extracting the hand region by distinguishing the hand and the face or background through the distance information with the sensor. The method of claim 5, wherein
The face detection step,
A face detection based hand gesture recognition method using Har-like features. The method of claim 5, wherein
The skin color modeling step,
And generating a skin color model by converting the color into a YCbCr color model. The method of claim 5, wherein
Distance information calculation step with the sensor,
The distance information

Calculated by Equation 2, wherein Dface is a distance between a sensor and the face, and the t value is 0.1. delete The method of claim 5, wherein
Further comprising a gesture interval detection step,
The gesture section detecting step,
And calculating the center coordinates of the hand region of the previous frame and the current frame, and determining the start and end of the gesture using the obtained coordinate difference value.

Images