KR100457928B1 - Hand signal recognition method by subgroup based classification - Google Patents

Abstract

The present invention relates to a method for recognizing hand signals according to hand posture by small group based classification.

In the hand signal recognition method according to the present invention, for a plurality of hand posture models, models having a high degree of image matching are grouped into small groups, and the first feature value and each small group which common hand posture models constituting any small group have in common. A first feature value extraction step of extracting a first feature value of the query hand image when a query hand image is input while storing a representative hand pose model image of the second hand value model and a second feature value of each hand pose model in a database; An image comparison step of comparing the query hand image with the representative hand posture model image of each small group; A small group recognition step of recognizing a small group to which the query hand image belongs by using a first feature value of the query hand image and an image comparison result value of the image comparing step; A second feature value extraction step of extracting a second feature value of the query hand image; Comparing the second feature value of the hand pose model and the second feature value of the hand pose models belonging to the small group to recognize the hand pose model that matches the query hand image and to recognize the hand signal applied to the hand pose model. A hand posture recognition step.

Description

Hand signal recognition method by small group based classification and recording medium recording the same {Hand signal recognition method by subgroup based classification}

The present invention relates to a method for recognizing hand signals, and more particularly, to a method for recognizing hand signals according to hand posture by small group based classification. The present invention also relates to a computer-readable recording medium having recorded thereon a program for performing the hand signal recognition method by the small group based classification.

As the utilization of computers in everyday life and the interest in virtual reality have increased, research on new user interface devices has been conducted accordingly. In the process of evolving from a text-based computer environment to a graphical user interface environment, in the past, as the mouse became the main interface device in addition to the existing keyboard, a virtual reality technology was developed to simulate the real world in three-dimensional space. If appropriate, a new interface device will be required.

Due to the necessity of the development of such a new interface device, various researches for recognizing the shape of the hand or the movement trajectory and controlling the system using the same are being conducted. To date, hand signal recognition systems using contact sensors and some camera systems have been announced. However, no research has been done on the hand posture recognition method considering the scalability, and no method for recognizing hand signals in 3D space has been studied.

The object of the present invention is to classify and store various hand postures into small groups of similar hand postures, and when a hand image of a query is input, the characteristics of the hand image of the query are extracted and classified into small groups. By recognizing, it is to provide a method for recognizing a hand signal according to a hand posture taken by a user.

Another object of the present invention is to obtain hand posture and hand trajectory information in each direction using two cameras, and recognize hand signals according to the hand posture and hand trajectory taken by the user using the obtained hand posture and hand trajectory information. It is to provide a way to.

1 is a block diagram of a hand signal recognition system according to an embodiment of the present invention;

2 is a diagram illustrating a detailed configuration of an image acquisition unit and a user's hand image acquired by the image acquisition unit according to the present invention;

3 is a detailed block diagram of a hand region extracting unit according to an embodiment of the present invention;

FIG. 4 shows 32 representative hand pose models,

FIG. 5 is a diagram illustrating a small group classifying the hand pose models of FIG. 4; FIG.

6 is a detailed block diagram of a hand posture recognition unit according to an embodiment of the present invention;

7 is a view showing a skeleton method used in the present invention,

8 is a diagram illustrating a normalization method of hand images;

9 is a view illustrating a length ratio of long and short axes of a hand image;

FIG. 10 is a diagram illustrating the convexity of the hand image.

※ Explanation of code about main part of drawing ※

110: image acquisition unit 120: hand region extraction unit

130: hand posture recognition unit 140: hand trajectory recognition unit

150: hand signal recognition unit 160: hand signal database

In the hand signal recognition method according to the present invention for achieving the above object, for a large number of hand posture models, models having a high degree of image matching are grouped into small groups, and hand posture models constituting arbitrary small groups have in common. Extracting the first feature value of the query hand image when the query hand image is input while the first feature value, the representative hand pose model image of each subgroup, and the second feature value of each hand pose model are stored in a database; 1 feature value extraction step;

An image comparison step of comparing the query hand image with the representative hand posture model image of each small group;

A small group recognition step of recognizing a small group to which the query hand image belongs by using a first feature value of the query hand image and an image comparison result value of the image comparing step;

A second feature value extraction step of extracting a second feature value of the query hand image;

Comparing the second feature value of the hand pose model and the second feature value of the hand pose models belonging to the small group to recognize the hand pose model that matches the query hand image and to recognize the hand signal applied to the hand pose model. Characterized by including a hand posture recognition step.

In addition, the hand signal recognition method according to the small group based classification according to the present invention, for a large number of hand posture models, models having high image matching group into small groups, and have a common hand posture model that constitutes any small group The first feature value, the representative hand pose model image of each subgroup, and the second feature value of each hand pose model are stored in the hand pose database, and one meaning is given to the combination of the hand pose and the hand trajectory and stored in the hand signal database. In the state, the image acquisition step of continuously photographing the user's hand in at least two directions;

Extracting a first feature value of the query hand image acquired in each direction, and comparing the query hand image with a representative hand posture model image of each small group, and comparing the first feature value of the query hand image with the image. A small group recognition step of recognizing a small group to which the query hand image belongs using a value;

Extracts a second feature value of the query hand image and compares the second feature value of the query hand image with a second feature value of the hand pose models belonging to the small group to recognize a hand pose model that matches the query hand image. An individual hand posture recognition step;

A hand trajectory recognition step of recognizing a trajectory of the continuous question hand image photographed in the image acquisition step;

And a hand signal recognition step of recognizing the hand signal of the user by applying a combination of the hand posture recognized in the individual hand posture recognition step and the hand trajectory recognized in the forward trajectory recognition step to the hand signal database.

According to the present invention, there is provided a computer-readable recording medium having recorded thereon a program for executing the hand signal recognition method by the small group based classification as described above.

Hereinafter, with reference to the accompanying drawings, a "hand signal recognition method by small group based classification and a recording medium recording the same" according to an embodiment of the present invention will be described in more detail.

1 is a block diagram of a hand signal recognition system according to an embodiment of the present invention. It is composed of at least two cameras, the image acquisition unit 110 for photographing the user's hand from the front and side, respectively, and the hand region extraction unit 120 for extracting the hand region from the image data obtained by the image acquisition unit 110 And a hand posture recognition unit 130 for recognizing hand posture in the extracted hand region, a hand trajectory recognition unit 140 for recognizing hand traces of the extracted hand region, and a hand posture recognition unit 130 and the hand. The hand signal recognition unit 150 which recognizes the meaning of the hand signal taken by the user by applying the hand posture and hand trace information respectively obtained by the trajectory recognition unit 140 to the hand signal database 160, and the meaning of the hand posture and hand trace information It has a hand signal database 160 that stores the.

The image acquisition unit 110 photographs the user's hand from the front and the side, respectively, and provides the hand region extraction unit 120 while the user receives the hand signal. The hand region extractor 120 extracts a hand region from the front and side images obtained by the image acquirer 110. A process of extracting the hand region from the image will be described later. The hand posture recognition unit 130 recognizes the hand posture of the hand image obtained by the hand region extraction unit 120, where the hand posture includes hand shape and hand direction information. This hand posture is recognized in two steps, which will be described later. The hand trajectory recognition unit 140 recognizes the trajectory of the center of gravity of the hand image in the image obtained by the hand region extractor 120. In this embodiment, the trajectory of the center of gravity of the hand image is referred to as the hand trajectory. The hand signal recognizing unit 150 applies the hand posture and hand trajectory information recognized by the hand posture recognition unit 130 and the hand trajectory recognition unit 140 to find out the meaning of the hand signal taken by the user.

2 illustrates a detailed configuration diagram of an image acquisition unit and a user's hand image acquired by the image acquisition unit according to the present invention. 2A is a detailed configuration diagram of the image acquisition unit, and includes a first camera 111 photographing the front surface of the user's hand 100 and a second camera 112 photographing the side of the user's hand 100. do. That is, the two cameras are arranged at right angles to each other. Assuming that the user moves the left and right hand to point the front with the index finger, the first camera captures the image as shown in Figure 2 (b) and the second camera captures the image as shown in Figure 2 (c). Accordingly, the hand trace moving from side to side and the hand posture like a fist are acquired from the image photographed by the first camera, and the hand posture information with the index finger removed from the image captured by the second camera. At this time, since the user moves the hand from side to side, the second camera photographing the side of the hand does not obtain hand trajectory information.

3 is a detailed block diagram of a hand region extracting unit according to an exemplary embodiment of the present invention. It includes an HSI conversion unit 121, a color histogram matching unit 122, a skin color model histogram 123, and a hand region candidate determination unit 124. The hand region information determined by the hand region candidate determiner 124 is used to determine the hand region of the next frame. The hand region is determined using the skin region information and the hand region information of the previous frame. The skin region information is obtained by the HSI converter 121 and the color histogram matching unit 122.

The HSI converter 121 converts an RGB image into an HSI (Hue, Saturation, Intensity) color space by applying the following equation (1). In addition, only H (color) and S (saturation) except for I (brightness) are provided to the color histogram matching unit 122 in order to minimize the effects of changes in illuminance.

The color histogram matching unit 122 is provided with a skin color model histogram 123. The skin color model histogram 123 selectively updates its color information according to the color distribution of the input image. In addition, the color histogram matching unit 122 extracts a skin color region from an input HS image by using a Gaussian Mixture Model. The hand region candidate determiner 124 extracts the hand region of the current frame by using the skin color region information of the current frame obtained by the color histogram matching unit 122 and the hand region information extracted from the previous frame. The hand region of this current frame is provided to the hand region candidate determiner 124 when extracting the hand region of the next frame. The hand image of the region extracted by the hand region candidate determiner 124 is provided to the hand posture recognizer 130 and the hand trajectory recognizer 140, respectively. Hereinafter, the hand image of the region extracted by the hand region candidate determiner 124 is called a query hand image. The hand posture recognition unit 130 recognizes the hand posture of the query hand image, and the hand trajectory recognition unit 140 extracts the hand trace of the query hand image which is continuously input.

In the present invention, 32 representative hand pose models as shown in FIG. 4 are classified into 12 small groups as shown in FIG. Each hand pose model is grouped into small groups of relatively high hand matching models, each of which includes at least one hand pose model and as many as four hand pose models. Representative hand posture images representing the hand poses constituting each subgroup are stored in a database. The method of obtaining the representative hand posture for each small group is the same as the method for normalizing the hand image of the query described later.

6 is a detailed block diagram of a hand posture recognition unit according to an exemplary embodiment of the present invention. The hand posture recognition unit 130 extracts the feature information of the query hand image and extracts the small group to which the query hand image belongs. An individual hand pose classification unit 132 extracts a hand pose model that matches the query hand image. The hand pose recognition unit 130 includes a database 131b which stores 12 small group representative hand pose images, a database 131f which stores 12 small group feature values (number of fingers), and a 32 hand pose model. The database 132d which stores the characteristic value (direction angle (theta) of the long axis, length ratio E of the long axis, and convexity C)) is provided. Here, the method of calculating each feature value of the hand posture model, that is, the number of finger feature values, the direction angle of the major axis, the length ratio of the major axis and the convexity, is the same as the method of obtaining the respective feature values of the query hand image described later. Here, the specific method for obtaining the feature value of the hand posture model is omitted.

The hand posture small group classification unit 131 obtains a direction angle (θ) of the long axis of the query hand image, and reverses the query hand image by R (-θ) by the corresponding direction angle to obtain a normalized query hand image (131a). ), A block 131c for image comparison of the 12 small group representative hand postures and the normalized query hand image stored in the 12 small group representative hand pose image databases 131b, and a skeleton image by performing a skeleton operation on the query hand image. A block 131d for outputting, a block 131e for calculating the number of fingers from the skeleton image, an image comparison value output from the image comparison block 131c, and finger number information output from the finger counting block 131e A block 131g is applied to the feature set database 131f of twelve small groups to classify the small group to which the query hand image belongs.

On the other hand, the individual hand pose classification unit 132 is a block for calculating the length ratio (E, Elongation) of the long / short axis of the query hand image, and a block for calculating the convexity (C, Convexity) of the query hand image 132b, a block 132c for selecting a required feature value from the length ratio E of the long / short axis of the query hand image, the convex information C of the query hand image, and the direction angle? And block 132e for classifying the individual hand pose models that match the query hand image by applying them to the feature set database 132d of 32 hand poses.

When the query hand image is input, the present invention first extracts the small group to which the query hand image belongs, and recognizes the hand posture in a two-step structure in which an individual hand pose model is extracted in the small group. This two-step structure of the hand posture recognition method calculates the amount of calculation because it calculates many features from the beginning and does not calculate the matching with all the hand posture models. Can be reduced. In addition, even when adding a new hand posture model, the hand posture recognition section can be easily extended because the hand posture model to be added is added to the most similar small group without modifying the entire hand posture recognition section.

Hereinafter, the functions of each component of the hand posture small group classification unit and the individual hand posture classification unit will be described in more detail.

First, a method of normalizing a query hand image is described in block 131a. This is the same as the method for obtaining the representative hand posture of the hand posture model that constitutes the small group. 8 is a diagram illustrating a normalization method of a query hand image. If the direction angle (θ) of the long axis (L) is obtained through blob analysis of the query hand image (see (a) of FIG. 8), and if the query hand image is reversed by the corresponding direction angle, (R (-θ)), a normalized quality hand image as shown in FIG. 8 (b) is obtained. As described above, the normalization method of the quality hand image can be used as a method of obtaining a representative hand posture for each subgroup. If one hand posture is normalized for each subgroup, a representative hand posture is obtained. Posture images are stored in a database.

here, , , I (x, y) represents the brightness value of the query posture image with respect to the point (x, y).

Next, the image comparison block 131c compares the normalized query hand image and the representative hand pose image of each small group by applying the equation (3).

here, Is the representative hand image of the i th small group, Is a normalized query hand image input by the user, Is the sum of all pixel values of the image Ix, n _T is the total size of the representative hand image (the number of horizontal pixels × the number of pixels), Video And video Exclusive NOR operation of, Video And video Exclusive OR operation of. The image comparison result T _i of the image quality of the hand image of the vagina and the representative hand pose of the i-th subgroup is normalized to [0,1] ∈R. At this time, the query hand image is likely to belong to the small group corresponding to the largest value among the image comparison results with the representative hand image of each small group for the normalized query hand image.

Next, as shown in FIG. 7, the skeleton calculating block 131d outputs a skeleton image representing a query hand image as a graph (composed of one pixel thickness), which is a connection form of a plurality of pixels. This skeleton method is a kind of morphology (Morphology) operation widely used in image processing, and is used to grasp structural features of an image. The finger count calculation block 131e calculates and outputs the number of fingers from this skeleton image. The hand posture small group classification unit 131g classifies the small group to which the corresponding query hand image belongs among the 12 predefined small groups by using the result of comparing the number of fingers of the query hand image and the image comparison result of the representative hand posture.

After the small groups for the vaginal hand image are classified, the individual hand pose classification unit 132 performs the direction angle (θ) of the long axis of the vaginal hand image, the length ratio (E) of the long / short axis of the vaginal hand image, and the vaginal hand. Using the feature values such as the convexity (C) of the image, a hand pose model matching the corresponding hand image is extracted.

The direction angle θ of the long axis of the query hand image may be obtained through Equation 2 as described above.

Block 132a calculates the length ratio E of the long and short axes of the query hand image using Equation 4 below.

Here, the subscript BOX means the smallest rectangle surrounding the corresponding hand image as shown in FIG. 9. This E value is also normalized to [0,1] ∈R. The closer this E value is to 1, the closer to the circle or square (see Fig. 9 (b)), and the closer to 0, the closer to the straight line (see Fig. 9 (a)).

Block 132b calculates the convexity C of the query posture image using Equation 5 below.

Here, p denotes the total perimeter of the contour surrounding the hand image obtained by blob analysis of the hand image, and A represents the area of the hand image. This C value is normalized to [0,1] ∈R according to the shape of the hand image.

As shown in FIG. 10, the convexity value of the hand pose image is a measure of the roughness of the blob surface. As the C value approaches 1, the hand image becomes closer to a smooth circle, and closer to 0. The more irregular the surface.

Block 132c selects the feature values needed to extract the individual hand poses from the small group. That is, one or two feature values used in the corresponding small group are selected from the three feature values (θ, E, C) of the query hand image for extracting individual hand postures. Table 1 shows the feature value types required for the individual hand posture extraction for each subgroup.

Small group Hand Posture Feature Values Used Number of Hand Posts in Small Group (One) E, C One (2) E, C One (3) θ, E 4 (4) θ 2 (5) θ, E 3 (6) θ 2 (7) θ 2 (8) E, C One (9) θ, E 4 10 θ 4 (11) θ 4 (12) θ, E 4

The hand trajectory recognition unit 140 recognizes a trajectory of the hand image by applying a hidden Markov model that is effective for data modeling of a space-time system.

The hand signal recognition unit 150 determines the meaning of the hand signal taken by the user by applying the recognition results of the hand posture recognition unit 130 and the hand trajectory recognition unit 140 to the hand signal database 160. At this time, if a number of hand posture or a plurality of hand traces can be taken to one hand signal, the number of hand signals that can be recognized can be extended. That is, when three hand postures are recognized per camera, for example, the first hand posture-middle hand posture-last hand posture and one hand trajectory, all eight pieces of information (four pieces of information per one camera) are obtained. The eight pieces of information and one hand signal are stored in the hand signal database, and when the user takes a hand posture and hand trajectory corresponding to the eight pieces of information, the hand signal of the user is recognized.

While the invention has been described above based on the preferred embodiments thereof, these embodiments are intended to illustrate rather than limit the invention. It will be apparent to those skilled in the art that various changes, modifications, or adjustments to the above embodiments can be made without departing from the spirit of the invention. Therefore, the protection scope of the present invention will be limited only by the appended claims, and should be construed as including all such changes, modifications or adjustments.

As described above, the present invention can be utilized as a human-friendly human-machine interface that recognizes a hand signal of a user used in daily life and delivers a command to a machine. In particular, when the use of hand signals is effective compared to other communication means such as peripheral devices, voices or sentences inside the vehicle, when the transmission of voice signals is impossible, the invention proposed in the present invention is effectively applied to communication with foreigners. Can be.

Claims (8)

For a large number of hand posture models, models with high image matching are grouped into small groups, and the first feature value common to the hand posture models constituting any small group and the representative hand posture model image of each subgroup and each hand A first feature value extraction step of extracting a first feature value of the query hand image when the query hand image is input while the second feature value of the posture model is stored in a database; An image comparison step of comparing the query hand image with the representative hand posture model image of each small group; A small group recognition step of recognizing a small group to which the query hand image belongs by using a first feature value of the query hand image and an image comparison result value of the image comparing step; A second feature value extraction step of extracting a second feature value of the query hand image; Comparing the second feature value of the hand pose model and the second feature value of the hand pose models belonging to the small group to recognize the hand pose model that matches the query hand image and to recognize the hand signal applied to the hand pose model. Hand signal recognition method according to the small group based classification characterized in that it comprises a hand posture recognition step. The method of claim 1, wherein the extracting of the first feature value comprises performing a skeleton operation on the query hand image to extract finger number information. The method of claim 1, wherein the comparing of the images comprises normalizing the query hand image, and applying the normalized query hand image and the representative hand pose model image of each small group to the following equations to obtain an image comparison result value. Hand signal recognition method according to the small group based classification, characterized in that it comprises the step of obtaining. [Equation] here, Is the representative hand image of the i th small group, Is a normalized query hand image input by the user, Is the sum of all pixel values of the image Ix, n _T is the total size of the representative hand image (the number of horizontal pixels × the number of pixels), Video And video Exclusive NOR operation of, Video And video Exclusive OR operation of, T _i is the result of comparing the image of the query hand image and the representative hand posture of the i-th small group, and is normalized to [0,1] ∈R. The method of claim 3, wherein the normalizing of the query hand image comprises: The method for recognizing a hand signal based on a small group based classification, wherein the query hand image is applied to the following equation to obtain a direction angle of a long axis, and the query hand image is rotated by the direction angle. [Equation] here, , , I (x, y) means the brightness value of the query hand image with respect to the point (x, y). The method of claim 1, wherein the extracting of the second feature value comprises: applying the query hand image to Equation 1 below to obtain a length ratio E of the long axis and the short axis of the query hand image, and applying the equation to the equation 2. A hand signal recognition method using small group-based classification, wherein the degree of convexity (C) of a quality hand image is obtained. [Equation 1] Here, the subscript BOX means the smallest rectangle surrounding the vaginal hand image. [Formula 2] Here, p is the total length (Perimeter) of the contour surrounding the query hand image, A is the area (Area) of the query hand image. On your computer, For a large number of hand posture models, models with high image matching are grouped into small groups, and the first feature value common to the hand posture models constituting any small group and the representative hand posture model image of each subgroup and each hand A first feature value extraction step of extracting a first feature value of the query hand image when the query hand image is input while the second feature value of the posture model is stored in a database; An image comparison step of comparing the query hand image with the representative hand posture model image of each small group; A small group recognition step of recognizing a small group to which the query hand image belongs by using a first feature value of the query hand image and an image comparison result value of the image comparing step; A second feature value extraction step of extracting a second feature value of the query hand image; Comparing the second feature value of the hand pose model and the second feature value of the hand pose models belonging to the small group to recognize the hand pose model that matches the query hand image and to recognize the hand signal applied to the hand pose model. A computer-readable recording medium having recorded thereon a program for executing a hand signal recognition method including a hand posture recognition step. For a large number of hand posture models, models with high image matching are grouped into small groups, and the first feature value common to the hand posture models constituting any small group and the representative hand posture model image of each subgroup and each hand Image of continuously photographing the user's hand in at least two directions while storing the second feature value of the posture model in the hand pose database and giving one meaning to the combination of the hand pose and the hand trace in the hand signal database. Obtaining step; Extracting a first feature value of the query hand image acquired in each direction, and comparing the query hand image with a representative hand posture model image of each small group, and comparing the first feature value of the query hand image with the image. A small group recognition step of recognizing a small group to which the query hand image belongs using a value; Extracts a second feature value of the query hand image and compares the second feature value of the query hand image with a second feature value of the hand pose models belonging to the small group to recognize a hand pose model that matches the query hand image. An individual hand posture recognition step; A hand trajectory recognition step of recognizing a trajectory of the continuous question hand image photographed in the image acquisition step; And a hand signal recognition step of recognizing the hand signal of the user by applying a combination of the hand posture recognized in the individual hand posture recognition step and the hand trajectory recognized in the forward trajectory recognition step to the hand signal database. Hand signal recognition method by the. On your computer, For a large number of hand posture models, models with high image matching are grouped into small groups, and the first feature value common to the hand posture models constituting any small group and the representative hand posture model image of each subgroup and each hand Image of continuously photographing the user's hand in at least two directions while storing the second feature value of the posture model in the hand pose database and storing the hand pose and the hand trajectory in a hand signal database. Obtaining step; Extracting a first feature value of the query hand image acquired in each direction, and comparing the query hand image with a representative hand posture model image of each small group, and comparing the first feature value of the query hand image with the image. A small group recognition step of recognizing a small group to which the query hand image belongs using a value; Extracts a second feature value of the query hand image and compares the second feature value of the query hand image with a second feature value of the hand pose models belonging to the small group to recognize a hand pose model that matches the query hand image. An individual hand posture recognition step; A hand trajectory recognition step of recognizing a trajectory of the continuous question hand image photographed in the image acquisition step; To perform a hand signal recognition method including a hand signal recognition step of recognizing a hand signal of the user by applying a combination of the hand posture recognized in the individual hand posture recognition step and the hand trajectory recognized in the forward trajectory recognition step to the hand signal database. Computer-readable recording medium that records the program.