KR101085536B1 - Method for Designing Interface using Gesture recognition - Google Patents

Abstract

Receiving a motion image of the subject; Detecting skin color from the input image; Detecting a target area based on the detected skin color; Detecting a face area from the target area; Detecting a hand region by removing the detected face region from the detected target region; And configuring an object selection and movement or count interface by detecting and tracking fingertip coordinates based on the detected hand region.

Description

Method for Designing Interface using Gesture recognition

The present invention relates to a method for configuring an interface, and more particularly, to a method for configuring an interface for object selection and movement by detecting and tracking finger coordinates using image-based gesture recognition.

As the average age of human beings increases due to the development of science and medicine, the aging phenomenon is prominent. The elderly population in Korea was 7.5% in 2000, but the rapid increase in the proportion of the elderly accounted for 10.7% of the total population as of 2009, and 20% in 2026. As the number of the elderly increases, senile diseases also increase. As a result, the most representative dementia of senile disease and the necessity of rehabilitation treatment for the elderly are rapidly increasing, and studies for this are being actively conducted.

Dementia prevention and rehabilitation treatment does not simply end with the treatment of drugs, but the process of inducing treatment effects by repeating long-term regular exercise and specific purpose activities.

Recently, boredom caused by long-term and repetitive treatments and artificial rehabilitation treatments have been developed for rehabilitation and computer-based rehabilitation and dementia prevention games. have.

However, the keyboard, which is a computer interface for gaming, is inconvenient for older seniors to use because of its small size and many keys. Accordingly, a function button interface made of a simple button is often used. However, since this simple button interface is made for only a few games, there is a limit to the type of game that can be selected. By simply operating the function button interface, it can be used for mental rehabilitation treatment, but requires physical movement. The disadvantage is that it is difficult to use for rehabilitation therapy.

Accordingly, an object of the present invention is to detect a hand region through gesture recognition, to detect and track the end point of a finger, and to provide a gesture interface such as the number of fingers.

The above object is to receive a motion image of the subject; Detecting skin color from the input image; Detecting a target area based on the detected skin color; Detecting a face area from the target area; Detecting a hand region by removing the detected face region from the detected target region; And configuring an object selection, movement, or count interface by detecting and tracking finger coordinates based on the detected hand region.

Preferably, the detecting of the skin color comprises: a sub-step of converting an RGB color model of the input image into an HSI color model; A substep of setting a predetermined region of the converted color model, obtaining a histogram of hue and saturation values of pixels included in the color model, and generating a histogram probability distribution image; A substep of binarizing the probability distribution image to obtain a binarized image; Performing a morphology operation on the binarized image to remove noise; And a substep of performing labeling on the image from which the noise is removed.

Also, preferably, the detection of the face region may be performed by applying a Haar-like algorithm.

In the above description, the center of the palm excluding the finger region may be determined as the center coordinate of the hand region, the distance from the center coordinate of the hand region to the outline of the hand may be calculated, and the coordinate at the longest distance may be set as the fingertip coordinates.

Preferably, the object selection interface may select a corresponding object when the y value moves up and down within a predetermined frame while the x value of the finger coordinate does not change significantly.

The counting interface may generate a circle having a radius from a center coordinate of the hand region, detect a portion overlapping the skin color region of the circumference of the circle, calculate the number of the detected portion, and calculate the number of fingers. You can count.

According to the above configuration, it is possible to increase the efficiency of rehabilitation treatment by causing interest in the rehabilitation treatment subjects by using it for physical rehabilitation treatment requiring movement.

1 is a flowchart showing a hand region detection process applied to the present invention.
FIG. 2 (a) shows an image after conversion into an HSI color model, and FIG. 2 (b) shows a color histogram.
FIG. 3 (a) shows an original image, FIG. 3 (b) is an image created by histogram probability distribution, FIG. 3 (c) is a binarized image, and FIG. 3 (d) is a result image where morphology calculation is performed. to be.
4 shows face detection with prototype applied.
5 shows a Haar-like feature prototype.
Fig. 6 (a) shows the SAT, and Fig. 6 (b) shows the sum of the pixels in the rectangle.
FIG. 7 illustrates a result of detecting an accurate hand region using face region detection and wrist region detection.
8 is a diagram illustrating a tracking interface of fingertip coordinates.
9 is a diagram illustrating a finger counting interface.

Hereinafter, with reference to the accompanying drawings will be described an embodiment of the present invention.

In general, gesture recognition technology methods can be classified into two types, a sensor-based gesture recognition method and an image-based gesture recognition method.

Sensor based gesture recognition Mechanical methods, acoustic methods, inertial methods, magnetic methods and optical methods are used to obtain the position and direction of movement, and magnetic methods and optical methods are mainly used.

Magnetic method is a method of measuring the motion data using the change of the magnetic field. This is very advantageous for real-time application because it outputs the flow of three-dimensional data in real time, but there is a disadvantage in that a lot of cables are connected to attach a receiver to a human body to limit the operation. For example, the sensor-based method is mostly used when a user wears a glove, attaches a sensor to the glove, and recognizes a hand motion using a signal input from the glove, or a remote controller of a game console wil, which is popular today. Use the method of hand mounting the equipment. However, since this method requires the user to wear gloves or mark the hands, there are many problems in establishing a natural interface since the movement of the finger is limited since the equipment is always mounted on the hand.

Image-based gesture recognition is a method of acquiring gesture information using a video camera and computer vision technology without attaching a device to an actor's body. There are two ways to use the image itself or to extract geometric features from the camera's input image. The use of the image itself is suitable for real-time recognition due to the small amount of calculation, but lacks generality. Holding it. The use of geometric features is mainly used when interpreting static gestures and poses. The use of the angles of joints allows precise expression, but it is not suitable for real-time recognition because it requires a large amount of computation.

As a method of using 3D geometric information, there is a method of using an image without equipment, and using a 3D geometric information of a hand extracted by using a stereo image. The three-dimensional geometric method has the disadvantage of requiring high performance hardware because of the large amount of computation required to extract three-dimensional information using stereo images and to build a complex three-dimensional motion model of the hand, which is difficult to adapt to applications requiring real-time processing. .

On the other hand, the method of using the two-dimensional pattern information is mainly used to extract the hand region by analyzing the color distribution of the human skin color. Compared to the 3D method, it can be used in applications requiring processing speed in real time. However, it is affected by light changes and complex backgrounds, and thus it is not recognized the unintentional hand motion or finds the hand area. In addition, since the method using the 2D pattern information uses the image projected in 2D, the shape deformation may occur due to the movement of the finger, occlusion, or rotation of the hand. Therefore, there is a disadvantage that it is difficult to trust the characteristic information on the acquired contour or region.

In this aspect, in the present invention, the method of using 3D modeling to detect and recognize in real time by the movement of a hand uses a method of using 2D pattern information having a small amount of computation because it is difficult to recognize a real time gesture due to a large amount of computation. .

The method of using two-dimensional pattern information is a method of detecting a human skin color, finding a hand region in the detected region, and analyzing a pattern. First, in order to effectively find the skin color region in the image, the skin color region is detected by converting the RGB color model into a HSI curl model. After binarizing the detected skin color region, noise is removed through morphology calculation, and skin color regions such as face and hand are detected by labeling. In the detected area, the face and hand area are classified by finding the face area using the Haar-like feature. Accurate hand regions can be found by removing face regions from the classified face and hand regions.

1 is a flowchart showing a hand region detection process applied to the present invention.

Video input

An image is input through the camera (step S11).

In general, the image coming through the camera uses the RGB color model. This RGB color model is difficult to extract the desired color. Therefore, in the present invention, the RGB color model is converted into an HSI color model of Hue, Saturation, and Intensity to be robust to lighting changes.

Skin color detection

Skin color is detected using histogram matching on the input image (step S21).

2 (a) shows an image after conversion into an HSI color model. A rectangle area of size 70 × 70 is set in the portion corresponding to the palm of the image, and a histogram of the color (H) and the saturation (S) values is obtained for the pixels included in the area as shown in FIG.

After the histogram is obtained, each pixel value of the image composed of Hue values is mapped to the histogram as shown in Equation 1 to obtain a probability value P of each pixel value. The P value corresponds to the height of the histogram for each color value in the histogram.

A saturation value is also obtained through the same process, and FIG. 3 (b) is an image created by probability distribution of histograms in FIG. 3 (a) which is an original image. If each pixel value is greater than zero in the image created by the probability distribution, binarization is performed with all values being 255 and all remaining values being 0.

Also, small lumps were removed through the Open operation of Morphology and small holes were removed using the Close operation. FIG. 3C is a binary image, and FIG. 3D is a result image of a morphology operation.

Next, labeling is performed to group adjacent pixels into objects in the image from which noise is removed through binarization and morphology. When the user looks directly at the camera and uses it, assuming that the area determined by skin color is the largest area, the small area except for the largest area and the second largest area is removed and the remaining area is the skin color of the face and hands. It is detected as an area.

Face area detection

When the skin color is detected through the above step S12, the face area and the hand area are detected together. In order to find the correct hand region, a task of detecting and removing the face region by classifying the face and the hand region is necessary (step S13). For this purpose, since the hand region is large in fluidity and is not fixed in shape, it is easier to find a face region in which eyes, nose, and mouth are fixed.

Preferably, the present invention uses the Haar-like feature as a method for detecting the facial region in the face region and the hand region recognized as the skin color.

The haar-like feature is one of the face area detection methods, which has the advantage of being simple and very fast. By using these Haar-like features, it is possible to reduce complex computations and effectively apply to real-time object detection requiring high detection performance.

4 shows face detection with prototype applied.

In addition, in the present invention, the Haar-like feature prototypes shown in FIG. 5 are used as effective features for detecting a face in an image.

When face region is detected by using Haar-like feature coefficients, it shows robust detection performance even on fluid face characteristics.

Preferably, an integral image was used to quickly perform pixel sum calculation of Haar-like squares in the image. The integral image is similar to the summed area table (SAT) used in image processing. The SAT speeds up computation by minimizing redundant operations.

The SAT of FIG. 6 (a) is the sum of all pixel values of a rectangular region including the coordinate region (x, y) from the origin (0,0) of the input image, and is calculated as in Equation 2 and Equation 3.

Shown as a 6 (b) the coordinates of the top left corner (x, y) is the width and the respective ω and h in height, and the square rotation angle α r = (x, y, ω, h, α) in a rectangle The sum of the pixel values of the area r is expressed as RecSum (r). The value of RecSum (r) can be effectively calculated using the SAT. The value of RecSum (r) for the rectangular region r = ( x, y, ω, h, 0) with the rotation angle α = 0 is calculated by equation (4).

Hand area detection

When the hand region from which the face region is removed through step S13 is detected, the skin color detection region also varies according to the length of the user's clothes in the detected hand region. In the case of wearing a long arm to the wrist, the area of the hand excluding the arm is extracted, but in the case of wearing a short arm, the hand area including the arm area is recognized. Therefore, only the unique hand region from which the wrist region has been removed must be detected for gesture recognition (step S14).

Finding the wrist area requires a non-moving area with little movement. The finger region of the hand region is the region with the most movement, but the palms and wrists are regions of very low mobility. In particular, the width of the wrists and arms is almost unchanged. So you can find your wrist using the width of your arms and palms, regardless of the shape of your hand. In addition, the wrist line can be detected using a feature that narrows gradually from the lower width of the arm to the upper width and then rapidly increases from the wrist portion. FIG. 7 illustrates a result of detecting an accurate hand region using face region detection and wrist region detection.

Interface configuration

An interface using gesture recognition is constructed using the above-described hand region detection method (step S15).

First of all, in order to make the function of the mouse, we implement the object selection interface that moves the cursor and selects objects and the object movement interface that can move the selected object by detecting and tracking finger coordinates. It also configures a count interface that can count.

To create an object selection interface, fingertip coordinates are required, and to detect fingertip coordinates, the center coordinates of the hand are needed. The fingers of the entire area of the hand are not fixed in shape but very flexible. In contrast, since the palm occupies a larger area and is fixed than the finger, the center of the palm excluding the finger region may be defined as the center coordinate of the hand region.

8 is a diagram illustrating a tracking interface of fingertip coordinates.

As shown in FIG. 8 (a), the distance from the center coordinate of the hand to the outline of the hand is calculated and the coordinate at the longest distance is set as the fingertip coordinate as shown in FIG. 8 (b).

Since the detected fingertip coordinates are often shaken by lighting or movement of the hand, a 5 × 5 region of up, down, left, and right pixels is set so that the center point in the region is detected as the position coordinate of the fingertip.

In this state, when the x value of the detected fingertip coordinate does not move significantly, the object may be implemented when the y value moves up and down within a predetermined frame.

9 is a diagram illustrating a finger counting interface.

Create a circle with a certain distance radius from the center coordinates of the hand, and then detect the portion of the circumference of the circle overlapping the skin color region, and calculate the number of the detected portion as shown in Figure 9 (a) to count the number of fingers Can be implemented. However, if a finger is attached as shown in FIG. 9 (b), an error is recognized as one, so that the number of fingers is accurate even if two or more fingers are attached to the circumference of the circle, assuming that 0.15 times the width of the wrist is one finger. Can be counted.

Application to rehabilitation games

The method of detecting the hand region and the gesture interface are applied to the game currently being used in the medical rehabilitation center. Rehabilitation games currently used in medical rehabilitation centers: Attention and Concentration, Memory, Reaction Ability, Logical Reasoning and Planning, Visual Motor Coordination (Visuo-) Motor Coordination) and 2 to 4 games for each treatment.

In addition, the game is aimed at children with mental retardation or the elderly for the prevention of dementia. The game is made for the game more simply than a complicated keyboard. It uses an input device consisting of arrows and selection buttons.

In relation to the picture matching game (AUFM), which is used to treat attention and concentration, we made a game that applied an interface specialized for gesture recognition instead of the existing interface. A picture-matching game is a game where one picture is shown and the same is selected from several pictures in different positions.

Gesture recognition interface was implemented in quad-core 2.4GHz and 2GB of RAM. The camera used a VIJE Talk Cam camera of CCD, and the resolution was maintained at 640 * 480, and OpenCV software was used in Visual C ++.

As shown in FIG. 9 (a), a game that finds the same picture among the six pictures at the top of the picture at the bottom part of the screen is made and moved to find the coordinates of the fingertips from the image received through the camera. Use a mouse-like interface that bends your fingers in the correct position and expands them to select a picture.

In addition, as shown in Figure 9 (b) to figure out how many pictures, such as the picture on the bottom, by selecting a picture to produce a game of moving to the same shape of the picture on the top.

At first, figure out how many fingers match the figure. In addition, by using the interface that counts the number of fingers, the figure is selected when the fingers are spread out on the picture to be moved at the bottom, that is, the number of fingers becomes five and then the fingers are held again and the number of fingers becomes zero. do. At this time, if the end point of the hand is moved on the screen, the figure moves accordingly. If you stretch your fingers again to reach five fingers, the moving picture will stop.

As a result of using the gesture recognition interface according to the present invention, not only the operation of the treatment subjects is easy and concise, the efficiency is increased, but the treatment subjects have a lot of interest, and the efficiency is increased.

Although the above has been described with reference to the preferred embodiment of the present invention, various changes and modifications can be made at the level of those skilled in the art, and such changes and modifications should be interpreted as belonging to the present invention without departing from the scope of the present invention.

S11: video input step
S12: skin color detection step
S13: face area detection step
S14: Hand Area Detection Step
S15: Interface Configuration Steps

Claims (6)

Receiving a motion image of the subject;
Detecting skin color from the input image;
Detecting a target area based on the detected skin color;
Detecting a face area from the target area;
Detecting a hand region by removing the detected face region from the detected target region; And
And detecting and tracking fingertip coordinates based on the detected hand region to construct an object selection and movement or count interface. The method according to claim 1,
Detecting the skin color,
Converting an RGB color model of the input image into an HSI color model;
A substep of setting a predetermined region of the converted color model, obtaining a histogram of hue and saturation values of pixels included in the color model, and generating a histogram probability distribution image;
A substep of binarizing the probability distribution image to obtain a binarized image;
Performing a morphology operation on the binarized image to remove noise; And
And a sub-step of performing labeling on the image from which the noise is removed. The method according to claim 1,
The face region detection method is performed by applying a Haar-like algorithm. The method according to claim 1,
A gesture characterized in that the center of the palm excluding the finger region is set as the center coordinate of the hand region, the distance from the center coordinate of the hand region to the outline of the hand is set, and the coordinate at the longest distance is set as the fingertip coordinates. How to configure interface using recognition. The method of claim 4,
The object selection interface is an interface configuration method using a gesture recognition, characterized in that for selecting the object when the y value is moved up and down within a predetermined frame in the state that the x value of the finger coordinates do not change significantly. The method of claim 4,
The counting interface may generate a circle having a radius from a center coordinate of the hand region, detect a portion overlapping the skin color region of the circumference of the circle, calculate the number of the detected portion, and count the number of fingers. Interface configuration method using a gesture recognition characterized in that.

Images