JP6202157B2 - Hand gesture identification method and apparatus - Google Patents

Description

  The present invention relates to the field of image processing, and more specifically to static hand gesture identification.

  Hand gestures are already widely used in various human-computer interaction systems such as interactive projection, interactive display, and hand-controlled TV. Hand gestures are an intuitive, natural, friendly, and human-interactive dialogue system provided by people. In general, there are two types of hand gestures: dynamic gestures and static gestures. A dynamic hand gesture is a sequence of movements of various hand forms, or a trajectory of changes in hand position, while a static hand gesture represents a hand form, that is, the same similar form held by a hand for a short period of time. .

  Unlike dynamic gestures, static gestures can be classified only by hand shape information. Template mapping is a method widely used in static gesture identification, and creates a static template library of gestures, and maps the identification waiting gestures by searching for the closest one from the template library. Usually, one hand gesture may correspond to multiple templates, and the hand gesture identification result mainly depends on the template and distance calculation rules, so the template mapping method is simple and intuitive. Although there is insufficient elasticity, it is not robust enough.

  In addition, as for some simple hand gestures, saliency rule judgment is also a hand gesture identification method that is relatively commonly used in this field. The method distinguishes different hand gestures mainly using distinctive features or rules. For example, some simple hand gestures can be identified by the number of fingers and their orientation. However, the features usually used by these rules are not only descriptive but only by piecewise features, and cannot fully reflect the features of hand gestures, and the method is completely dependent on the rules, This method is also not robust enough and has a relatively high possibility of transmitting an erroneous message in actual use.

  However, in some public situations, for example in public places such as museums, libraries, exhibitions, etc., taking into account that the actual operating users of the system are all untrained participants, It is desirable to be mostly simple and intuitive, as natural as possible, and at the same time to identify these gestures robustly and to minimize false messages. The conventional method of easily identifying a static gesture using template mapping or rule judgment cannot increase the accuracy of identification in a situation where erroneous messages are reduced.

  In view of the above problems, it is desired to provide a hand gesture identification method and apparatus that can increase the accuracy of hand gesture identification and reduce the possibility of erroneous determination.

  According to one aspect of the present invention, an input image of a target scene including a hand is obtained, and an image of a hand is extracted from the input image; the image of the hand is classified by using a classifier, and thereby identified. Obtaining a classification result of the waiting hand shake; extracting a rule feature from the image of the hand; and applying the rule feature to the classification result of the waiting hand gesture to obtain a hand gesture identification result. Including hand gesture identification method.

  Classifying the image of the hand using the classifier, and thereby obtaining the classification waiting hand shake classification result, extracting the classification feature from the image of the hand, and applying the classification feature to the classifier Thus, it can include obtaining a classification result of the identification waiting hand shake.

  The classification features extracted from the hand image may include at least one of a hand contour convex hull feature, a hand contour Fourier descriptor feature, and a hand contour invariant matrix feature. The rule features extracted from the image of the hand may include at least one of the number of fingers, the length of the finger, the orientation pointed by the finger, the relative position between the fingers, and the depression angle between the two fingers. .

  The classifier may be a classifier obtained by optimizing a classification strategy using rule characteristics, and the classification strategy includes a number of classifications and a classification designation strategy.

  The following iterative process, that is, the process of training a classifier having a predetermined classification strategy using a training sample collection, the process of applying the sample collection evaluation to the classifier that trains the evaluation of the sample collection, thereby obtaining the sample classification result, Applying a predetermined rule feature to the sample classification result, thereby obtaining a sample hand gesture identification result, and repeating the process of optimizing the classifier's classification strategy based on the sample hand gesture identification result Strategy can be optimized.

  The classifier may be a decision tree classifier, random forest classifier, perceptron classifier, neural network classifier, or SVM classifier. Alternatively, the classifier may use a template mapping classification technique.

  Alternatively, the input image can be selected to include a continuous K-frame input image, and if there is a matching M-frame gesture identification result in the K-frame input image, The identification result is determined as the hand gesture identification result in the input image of the K frame, and K and M are predetermined positive integers and K> M.

  According to another aspect of the present invention, an acquisition unit that acquires an input image of a target scene including a hand and extracts an image of the hand from the input image; classifies the image of the hand using a classifier; A classification unit that obtains a classification result of an identification waiting hand shake; a feature extraction unit that extracts a rule feature from the image of the hand; and an application of the rule feature to the classification result of the identification waiting hand shake, thereby obtaining a gesture identification result A hand gesture identification device including an identification unit to obtain is provided.

  According to the present invention, a classifier is used to classify hand images, thereby eliminating invalid hand gestures to obtain identification waiting hand shake classification results, and subsequently applying rule features to the classification waiting hand shake classification results to perform hand gestures. Complete final identification. The method is a combination of weak classifiers and strong rule decision technology, and classifiers are not used directly for the final identification of various hand gestures, and the majority of invalid hand gestures are eliminated by the filtering function of the classifiers. Therefore, a weak classifier built on the basis of the overall features, and at the same time, valid hand gestures can be reliably passed through the classifier with minimal leakage. After that, the strong rule is applied to the classification waiting hand shake classification result of the classifier, and various kinds of various hand gestures are further classified according to the distinguishing feature. As a result, the distinction rule feature and the description feature of the entire classifier complement each other, improving the accuracy of hand gesture identification and reducing the possibility of misjudgment, and realizing robust recognition of static hand gestures Can do.

5 is a flowchart of a hand gesture identification method according to an embodiment of the present invention. It is a figure which shows the example of an input image, and the binarized silhouette image of the hand extracted from it. 6 is a flowchart of a method for optimizing a classifier. It is a figure which shows a confusion matrix. It shows five types of effective hand gestures and corresponding binarized silhouette images. It is a figure which shows the gesture identification process of said 5 types of gesture. It is a figure which shows the area | region division of the image of a hand. It is a figure which shows a gesture identification rule in the format of a table | surface. It is a figure which shows the data flow in an example of hand gesture identification. It is a functional block diagram of the hand gesture identification device by one Example of this invention. It is a figure which shows the hardware constitutions of the gesture identification system by one Example of this invention.

  In order that those skilled in the art may better understand the present invention, the present invention will be described in more detail below in conjunction with the accompanying drawings and specific implementation methods.

The following order:
1. 1. Outline of the inventive idea Example
2.1 Hand gesture identification method
2.2 Optimization of classifier
2.3 Application example of hand gesture identification method
2.4 Hand gesture identification device
2.5 Hand gesture identification system The description will be given in the order of summarization.

<1. Overview of Invention Thought>
As described above, when performing hand gesture identification, it is desired to increase the accuracy of the identification and at the same time reduce the possibility of erroneous messages, that is, avoid the identification of invalid hand gestures as valid as much as possible.

  In hand gesture identification, machine learning and classification techniques can be viewed as a kind of probabilistic extension to template mapping, which focuses primarily on descriptive features, but uses rules in hand gesture identification methods that discriminate using rules. The feature is usually a piecewise feature. As is well known, descriptive features, such as Fourier descriptors, are features that fully describe a gesture, usually avoiding the possibility of invalid gestures being mistakenly identified as valid, Some specific gestures usually do not have any categorization, resulting in inaccurate identification results, and categorical features can be better distinguished from hand gestures, but these features are usually all Because it is a local feature, it is highly contingent and prone to misjudgment.

  In view of this, the present invention uses a classifier to classify the input image based on the classification features (descriptive features), eliminate invalid hand gestures, acquire identification waiting hand gestures, and continue to identify waiting hand shaking classification results Applying the distinctive feature of rules to complete the identification of hand gestures. As a result, the classifier according to the present invention does not bear the final hand gesture identification task, the elimination of invalid hand gestures is its own task, and the classification result of the ruler is added to the classification result of the classifier from which invalid hand gestures are excluded. In this way, the final hand gesture identification is performed. In this way, the accuracy of the hand gesture identification result can be guaranteed, and at the same time, the number of erroneous messages can be reduced.

<2. Example>
<2.1 gesture identification method>
FIG. 1 shows an overall flowchart of a gesture identification method according to an embodiment of the present invention. As shown in FIG. 1, the hand gesture identification method 100 according to this embodiment acquires an input image of a target scene including a hand, and extracts a hand image from the input image Step S110; Step S120 for classifying hand images and thereby obtaining a classification result of identification waiting hand gestures; Step S130 for extracting rule features from the hand images; and Step S140 may be included to apply and thereby obtain a hand gesture identification result.

  Specifically, in step S110, a target scene including a hand is photographed by a camera, a photographing machine, etc., thereby obtaining an image of the target scene, performing image processing on the image, and thereby locally covering the hand Images can be extracted. For example, the hand region is determined by tracking the movement of the hand in a predetermined number of continuous frame images captured by the photographing machine, thereby extracting a local image of the hand region. It is easy for those skilled in the art to extract a predetermined region of interest from the input image, and a usable method is not limited to the tracking method.

  As an example, in order to make it easier to extract hand image features in subsequent steps, the hand part is separated from the local image and the binarized silhouette of the hand is obtained, for example, skin color model, threshold division based on depth Alternatively, the binarized silhouette of the hand can be acquired by a method such as region extension analysis.

  For example, FIG. 2 shows an example of an input image and a binarized silhouette image of a hand extracted from the input image. In the binarized image, for example, any element that may be affected by a background or a part that is not a hand has already been removed, so that it is useful for accurately extracting the features of the hand itself from the elements. Of course, the binarized silhouette described here is merely an example of a hand image to which the present invention can be applied, and the present invention can use any other suitable image.

  After extracting the hand image, in step S120, the classifier is used to classify the hand image, thereby obtaining the classification result of the identification waiting hand shake. The classifier used here is a classifier having an optimized classification strategy that has been trained in advance.

  A classification strategy can typically include two aspects: the number of classifications as well as the specified model of the classifier. For example, taking the identification of three valid gestures ABC and one invalid gesture D as an example, if the number of classifications is 3, there are at least three different combinations of classification designations, {AB, C, D }, {AC, B, D} and {A, BC, D}. If the number of classifications is 2, the designated model is usually {ABC, D}. In the present invention, the output of the input hand image represented through the classification of a possible hand gesture to be identified (because it is not necessarily a valid hand gesture and may not necessarily be a hand gesture). The classified result is merely an intermediate result, in other words, the output of a so-called implication layer, and the possible gesture to be identified is classified as a valid gesture (ie, an identification waiting gesture) or an invalid gesture. If it is classified or there are many types of valid gestures, it simply represents what type of valid gestures it may belong to. When the final result of hand gesture identification is reached, it is necessary to further determine and acquire it according to rules.

  The classification waiting gesture described here refers to the classification result of a gesture that is classified as valid by the classifier but needs to be further determined according to the rules, that is, a gesture other than that classified as invalid by the classifier, For convenience, this is called identification waiting gestures, but those skilled in the art will not necessarily identify these identification waiting gestures as valid gestures in the end, but may still include invalid gestures. It must be clearly understood that it needs to be judged and confirmed. In other words, the identification waiting gesture is also a pseudo identification waiting gesture.

  Similar to the number of classifications, the classification specification model is not independent, and depends on the classification characteristics obtained when applying classification characteristics to the classification results, and the cohesiveness of classification features used when classifiers execute classification. Various classification strategies affect the agglomeration of classification features and affect the final discrimination result, and this good / bad evaluation is also considered with the aid of complementarity to the rule classification Since there is a need, the design and optimization of the classifier's classification strategy is an iterative process with reference to the effect of using rules.

  The specific training and optimization process of the classifier will be described in detail later, and here we apply the classifier assuming that a classifier with an optimized classification strategy has already been acquired. The hand images acquired in step S110 are classified, and thereby, the classification waiting hand classification result is acquired.

  Specifically, when classifying a hand image by applying the classifier, the classification feature is extracted from the hand image, and the classification feature is applied to the classifier. Can be obtained.

  Classification features are features used by the classifier when classifying hand images, describing all gestures that may need to be identified as fully as possible, and valid gestures are mistakenly classified as invalid gestures. In order to avoid being excluded, some global descriptive features can now be used as classification features. For example, applicable classification features may include at least one or more of hand contour convex hull features, hand contour Fourier descriptor features, hand contour invariant matrix features, and the like.

  After extracting the classification feature from the hand image extracted in step S110, applying the extracted classification feature to the classifier having the above-described optimized classification strategy, the classification result of the waiting for identification hand shake can be acquired.

  As a result of classification by the classifier, as described above, if a hand gesture that needs to be identified included in the image of the hand is classified as a set of valid hand gestures, then it is judged by a rule, To which hand gesture of the set of valid hand gestures corresponds. Of course, as described above, the waiting gesture for identification here means that it is not necessarily identified as one of the groups classified into the effective gesture, and the classifier used here Is a weak classifier, which is intended to prevent valid gestures from being mistakenly excluded as much as possible, but there is still a possibility that invalid gestures or gestures are classified as valid gestures. Become.

  For example, when the classifier as described above uses {ABC, D} as a specified model and classifies the classification target into two types, the classification result of the waiting hand gesture is classified into one type according to the classification characteristics of the extracted hand image. It can be an effective hand gesture ABC.

  In other words, after classifier classification, some hand images, including invalid or non-invalid gestures, can be eliminated. For those possible gestures classified by the classifier as waiting for identification, further rules need to determine whether the identification is indeed a valid gesture and which valid gestures. For this reason, in step S130, a rule feature is extracted from the image of the hand.

  A rule feature is a feature corresponding to a rule used when making a rule decision. As mentioned above, several strong rules with clearly distinctiveness can be used so that a distinction can be made between one type of hand gesture. In order to complement the overall classification feature described above, the rule feature may be a local feature. For example, the rule features that can be used can include one or several of the number of fingers in the hand image, finger length, finger pointing direction, depression angle between two fingers, etc. .

  After extracting the rule feature of the hand image, in step S140, the extracted rule feature can be applied to the identification waiting hand shake classification result obtained in step S130, and the final result of hand gesture identification can be obtained. In other words, if the rule feature extracted from the image of the hand is matched with any one of the types of identification waiting gestures obtained by classification in step S120, it is included in the image of the hand. The possible hand gesture is identified as the hand gesture. On the other hand, if the rule feature extracted from the hand image does not match any one of the hand gestures waiting to be identified, the possible hand gesture included in the hand image is invalidated. Identify.

  Thus, according to the hand gesture identification method 100 of the present embodiment, the classifier is used to classify the hand images, thereby eliminating invalid hand gestures and obtaining the identification waiting hand shaking classification result, and subsequently continuing the identification waiting hand shaking classification. Apply rule features to the result to complete the final identification of hand gestures. This method is a combination of weak classifiers and strong rule judgment technology, and classifiers are not used directly for final identification of various hand gestures, but are weak classifiers built on the overall feature basis. Invalid hand gestures are eliminated by the filtering function of the classifier, and at the same time valid hand gestures ensure that the classifier can pass with minimal leakage. After that, the strong rule is applied to the classification waiting hand shake classification result of the classifier, and further, various kinds of hand gestures are classified. As a result, the advantages of the rule feature and the classifier feature complement each other, thereby improving the accuracy of hand gesture identification and at the same time reducing the possibility of misjudgment and realizing robust recognition of static hand gestures.

<2.2 Optimization of classifier>
In order to obtain the best classifier suitable for specific hand gesture classification, the classifier is trained by machine learning technology, and the classification strategy is adjusted by evaluating the classification result. Can be acquired.

  When designing a classifier, the classification features and the rules used during subsequent identification must be established prior to the classifier, which is based on these characteristics and rules. Because. Therefore, a classifier having a predetermined classification strategy can be designed based on a priori knowledge of classification features and rule features of those skilled in the art, and an optimized design of the classifier can be completed by iterative evaluation. .

  Hereinafter, a method of an optimization example of the classifier will be described with reference to FIG. FIG. 3 shows a flowchart of an example method for optimizing the classifier. As shown in FIG. 3, the optimization method is applied to the following steps, ie, training a classifier having a predetermined classification strategy using a training sample collection S310; A step S320 for acquiring a sample classification result thereby; a step S330 for applying a predetermined rule feature to the sample classification result, thereby acquiring a sample hand gesture identification result; and a classifier based on the sample hand gesture identification result. Step S340 may be included to optimize the classification strategy.

  In step S310, a sample collection having a positive sample and a negative sample is prepared in advance. For example, a sample collection containing 100 samples is prepared in advance, 80 of which are used for training samples, and the remaining 20 samples are used for the necessary test samples afterwards. However, one skilled in the art can select a sample as needed.

  A classifier having a predetermined classification strategy is trained by a training sample for machine learning technology. Thereafter, in step S320, the test sample collection is applied to the classifier trained in step S310, thereby obtaining a sample classification result. As mentioned above, various classification strategies affect the agglomeration of classification features and further affect the final discrimination result, and the evaluation of the discrimination result with the help of complementarity to the rule classification. Therefore, in step S330, the predetermined rule feature is applied to the sample classification result obtained in step S320, and the test sample is subjected to hand gesture identification. Thus, in step S340, the classification strategy of the classifier can be optimized based on the sample hand gesture identification result.

  Specifically, it is possible to evaluate whether or not the hand shake identification result of the test sample is accurate, and determine whether or not the classification strategy of the classifier needs to be adjusted based on the evaluation result. The evaluation can be performed based on a confusion matrix of test sample hand gesture identification results. The confusion matrix records the expected sample identification results for each test sample as well as the actual identification results.

  FIG. 4 shows an example of a confusion matrix. For example, as shown in Figure 4, the input invalid gesture sample has a 95.1% probability of being identified as an invalid gesture, a 1.8% probability of being identified as a thumb upward gesture, and is identified as a thumb downward gesture. The probability is 3.0%.

  At the time of evaluation, for example, if the exact identification rate of each hand gesture is all greater than or equal to the first predetermined threshold (eg 90%) and the omission rate is all less than the second predetermined threshold (eg 10%), If it reaches the predetermined threshold value (for example, 5%) or less, it is considered that such a gesture identification result is satisfactory, and at this time, adjustment of the classification strategy is not required. Otherwise, if it is evaluated and the hand gesture identification result does not satisfy the person, the classification strategy of the classifier is adjusted, that is, the number of classifications and / or the specified model of the classification is adjusted, and the adjusted classification strategy The above-described steps S310 to S340 are continuously executed on the classifier having the above, and the classification strategy can be optimized until a hand gesture identification result that satisfies the obtained classification strategy can be obtained.

  In other words, in the above example, the classifier optimization may be an iterative process of “adjustment-training-evaluation”. In practice, however, if the person skilled in the art fully understands the classification features and rule features that he or she wants to use and is familiar with the merits, demerits, and complementarity of each of them, the classification strategy of the classifier can be designed directly, The classifier can be optimized without the need for an iterative process.

  Here, various classifiers well-known in the field such as a decision tree classifier, random forest classifier, perceptron classifier, neural network classifier, or SVM classifier can be used. The classifier described here is a broad classifier, and may include some other classifying means, for example, a classifying means for classifying by a template mapping technique.

  The design of this classifier is not performed independently, but by referring to the effect of use of the rule, the classification result can be output with an accurate identification result after the rule is strengthened.

  This classifier optimization method makes it possible to obtain a classifier with an optimized classification strategy, which is not designed directly for the final identification of various hand gestures, but instead of the overall feature basis. A weak classifier built on top of machine learning, which eliminates the majority of invalid gestures by classifier filtering, and at the same time ensures that valid gestures pass through the classifier with minimal leakage. To.

<2.3 Application example of hand gesture identification method>
Hereinafter, specific examples of identifying five types of effective gestures will be described with reference to FIGS. 5 to 9 by applying the gesture identification method of the present invention.

  FIGS. 5A to 5B show examples of five effective hand gestures and corresponding binarized silhouette images. Fig. 5 (a) shows five examples of identifiable hand gestures: left to right, left pointing (GA), right pointing (GB), thumb upward (GC), thumb downward ( GD) and palm open hand gesture (GE), left pointing hand gesture (GA) and right pointing hand gesture (GB) can be used for direction indication, thumb upright hand gesture (GC) and thumb down hand gesture (GD) It can be used to express an individual's point of view with “Nice” or “Bad”, and palm open hand gestures (GE) can understand whether it is stopped or prohibited. Figure 5 (b) is extracted from the input image. An example of a binarized silhouette image corresponding to each kind of gesture is shown.

  For convenience of explanation, (GF) represents all other input images not belonging to the above five types of hand gestures. In other words, the above five types of gestures are understood as effective gestures, and (GF) is input as an invalid gesture.

  FIG. 6 shows a conceptual diagram of the gesture identification process for the above five types of gestures.

  As shown in FIG. 6, in step S00, an image of a target scene including a hand is input. In step S01, a binarized silhouette image of the hand is extracted. In step S02, the classification feature of the binarized silhouette image of the hand is extracted.

  In this example, the number of fingers and the direction of the fingers have a significant function and important significance in distinguishing these five types of hand gestures, and therefore can be used as strong rules. Given that all of these rules are local features, several global features can therefore be selected to be the descriptive features of the classification. Specifically, the following two types of features, a hand contour convex hull feature and a hand contour Fourier descriptor feature can be extracted from the binarized silhouette image of the hand image.

  For hand contour convex hull features, two feature quantities are calculated: the ratio of the length of the convex line to the length of the hand contour line (denoted as r1) and the ratio of the convex area to the hand contour area (denoted as r2). be able to.

  By performing a Fourier transform on the hand contour, a hand contour Fourier descriptor can be obtained, which includes coefficients describing the hand contour shape with various probabilities, but here we extract some low-frequency coefficients. The contour shape of the hand can be represented. For example, ten low frequency coefficients f0, f1, f2, f3, f4, f5, f6, f7, f8, and f9 can be used.

  These two types of classification features can constitute one feature vector and are written as F {r1, r2, r3, r4, r5, r6, r7, r8, r9}. Based on a priori knowledge, the feature vector F {r1, r2, r3, r4, r5, r6, r7, r8, r9} has feature-invariant properties for rotation, translation and scaling, and has a palm shape Reflecting the overall characteristics of the above, it can be seen that a complementary relationship with the above strong rules is formed.

  In step S03, the image of the hand is applied to a classifier trained in advance, thereby obtaining a classification result of the identification waiting hand shake. Here, the above feature vector F {r1, r2, r3, r4, r5, r6, r7, r8, r9} has invariant characteristics with respect to rotation, translation, and enlargement / reduction conversion. It has very similar characteristics to (GA), (GB), (GC), (GD). Based on this a priori knowledge, when designing a classifier, the classifier can be trained directly by specifying the classification designation strategy directly. Therefore, the number to be classified here may be 3, and the designated model of classification may be {CA, CB, CC}, and {(GA) (GB) (GC) (GD), (GE), (GF)} respectively. Correspond. Divide hand gestures (GA), (GB), (GC), and (GD) into one type of CA, and more effectively use the cohesiveness when classifying features. Regardless of the number of classifications, finally most invalid input images CC are first removed by the classifier, and the remaining two hand gestures of CA and CB are judged by further rules and further identified. The Here, it can also be seen that the classifier increases the effect of the weak classifier divided into two. For this reason, in another embodiment, the number of classifications is set to 2, that is, the above five types of effective gestures can be combined into one type, and ineffective gestures can be combined into one type.

  In step S04, rule features are extracted from the image of the hand. Here, regular features can be obtained by using the number of fingers and the relative direction of the fingers. In order to detect the number of fingers in the hand image, the position of the fingertip can be detected first. The detection of the fingertip position is calculated based on the curvature of the hand contour.

  First, it looks for possible candidate positions for all pseudo fingertips, but these candidate positions are usually at the vertices of the convex. FIG. 6 shows an example of the fingertip candidate position point P0.

  After that, select candidate position point p0 near the fingertip on the contour of the hand and two points p1 and p2 located on both sides of the candidate position point p0, respectively, and based on the positional relationship between the two points Calculate the curvature of each candidate position point and express it as the following formulas (1)-(3).

v1 = P1-P0 (1)
v2 = P2-P0 (2)
Cur ＝ acos (v1 ・ v2 / (| v1 | * | v2 |)) (3)
In the formula, v1 represents a vector from the p1 point to the p0 point, v2 represents a vector from the p2 point to the p0 point, and P0, P1, and P2 are position coordinate vectors of the p0, p1, and p2 points, respectively. Cur represents the curvature value at the location of the candidate position point p0.

  By setting one curvature threshold value Ct in advance, it can be determined whether or not the candidate position point of the hand is the fingertip. If Cur <Ct, Cur's corresponding candidate position point p0 is taken at the fingertip position, otherwise it is not taken.

  Since there is relatively large noise in the finger contour, the above-mentioned 10 low-frequency Fourier descriptors are subjected to inverse Fourier transform to reconstruct the finger contour, and the above fingertip position calculation is performed on the reconstructed finger contour. This is because the reconstructed contour is smoother than the original contour.

  Thereby, the number of fingers can be determined based on the detected position of the fingertip. Thereafter, in step S05, the relative direction of the finger is detected.

  When only one finger is not detected, the relative direction of the finger can be determined by taking out the center of gravity of the hand. Specifically, the two-dimensional space of the entire hand image is divided into four regions around the center of gravity of the hand.

  FIG. 7 shows an example of region segmentation of the hand image. As shown in FIG. 7, these four regions are labeled A, B, C, and D, respectively. By detecting in which region the fingertip is located, the relative direction of the finger can be determined, and in step S06, the number of fingers and the relative direction of the finger can be applied to identify the hand gesture as a regular feature.

  FIG. 8 presents an example of hand gesture identification rules in the form of a table. As shown in FIG. 8, when only one finger is detected, if the position of the fingertip is located in the area A, it is determined that the thumb is upwardly waving. In other cases, if the fingertip is located in the region B, C, or D, it is determined that the thumb is downward, left pointing, and right pointing respectively. Of course, if the number of fingers greater than or equal to four is detected, it is not necessary to detect the relative direction of the fingers any more and it is determined as a prohibited gesture.

  FIG. 9 shows the data flow in this hand gesture identification example. As shown in FIG. 9, the input hand image may be classified into an identification waiting gesture (G01) or an invalid gesture (G02) via a classifier, and the classification waiting gesture (G01) Further rule judgment is performed, and thereby, it is identified as a valid gesture (G011, which may be A, B, C, D or E) or an invalid gesture (G012).

  Of course, the above is only an example, and the hand gestures that can be identified in a specific application are not limited to the above five types, and any other possible hand gesture can be used, and the classification features and rule features to be used at the same time are also specifically identified. It can be changed by waiting for you.

  In addition, in actual application, the target scene video is usually captured and static hand gesture identification is performed. Therefore, when applying the hand gesture identification method according to the present invention, voting is performed based on a predetermined number of frames in the captured video. Hand gesture recognition can be performed by applying the principle. Specifically, for example, for consecutive K-frame input images, if the M-frame gesture identification result matches the K-frame input image, the M-frame gesture identification result is input to the K-frame input image. It is determined that the result is the hand gesture identification result in the image, and K and M are predetermined positive integers and K> M.

  For example, in this example for identifying the above five types of hand gestures, when the video capture speed is 30 frames per second, if the hand gesture identification method of the embodiment of the present invention matches the hand gesture identification results in the continuous 10 frames of the input image. If the result of identifying images of 8 frames or more matches, the “8/10” criterion can be used, in which the hand gesture identification result is considered to be the hand gesture identification result in the input image of the continuous 10 frames. At this time, the error message rate is close to 0, and the gesture identification accuracy is greater than 99%. Of course, this voting rule is only an example, and those skilled in the art can see and adjust the specific application situation.

<2.4 Hand gesture identification device>
Hereinafter, a hand gesture identification apparatus according to another embodiment of the present invention will be described with reference to FIG. FIG. 10 shows a functional block diagram of the hand gesture identification device. As shown in FIG. 10, the hand gesture identification device 1000 acquires an input image of a target scene including a hand and extracts an image of the hand from the input image; an image of the hand using a classifier A classification unit 1020 for obtaining a classification result of a waiting gesture for identification; a feature extraction unit 1030 for extracting a rule feature from the image of the hand; and applying the rule feature to a classification result of the identification waiting gesture Thus, an identification unit 1040 that acquires a hand gesture identification result can be included.

  The feature extraction unit 1030 can extract a classification feature from the image of the hand, and the classification unit 1020 applies the classification feature extracted by the feature extraction unit 1030 to the classifier. A classification result can be acquired.

  The classification feature extracted by the feature extraction unit 1030 may include at least one of a hand contour convex hull feature, a hand contour Fourier descriptor feature, and a hand contour invariant matrix feature. The rule feature extracted by the feature extraction unit 1030 may include at least one of the number of fingers, the length of the finger, the orientation pointed by the finger, the relative position between the fingers, and the depression angle between the two fingers. .

  The hand gesture identification device can also include an optimization unit (not shown) used to acquire the classifier by optimizing the classification strategy using the rule feature, and the classification strategy includes the number of classifications. As well as a designation strategy for classification. The optimization unit is applied to the following iterative process, that is, a process for training a classifier having a predetermined classification strategy using a training sample collection, and a classifier for training evaluation of a sample collection. The process of obtaining the classification result, applying a predetermined rule feature to the classification result of the sample, thereby obtaining the sample hand gesture identification result, and based on the hand gesture identification result of the sample, the classification strategy of the classifier is optimized Can be arranged to perform the process.

  The classifier may be a decision tree classifier, random forest classifier, perceptron classifier, neural network classifier, or SVM classifier. Alternatively, the classifier may use a template mapping classification technique.

  Optionally, the input image can be selected to include a continuous K-frame input image, and if there is a matching M-frame gesture identification result in the K-frame input image, the identification unit Confirms the gesture identification result of the M frame as the gesture identification result in the input image of the K frame, where K and M are predetermined positive integers and K> M.

  The specific operation processes of the acquisition unit 1010, the classification unit 1020, the feature extraction unit 1030, and the identification unit 1040 of the hand gesture identification apparatus 1000 can be referred to the description in steps S110 to S140 in the above method 100. Yes, I will not repeat it here.

  The hand gesture identification device 1000 according to this embodiment of the present invention classifies hand images using a classifier, thereby eliminating invalid hand gestures, obtaining classification waiting hand gesture classification results, and continuing to identify hand gestures. Apply the rule features to the classification result to complete the final identification of the hand gesture. This method is a combination of weak classifiers and strong rule judgment technology, and classifiers are not used directly for final identification of various hand gestures, but are weak classifiers built on the overall feature basis. Invalid hand gestures are eliminated by the filtering function of the classifier, and at the same time valid hand gestures ensure that the classifier can pass with minimal leakage. After that, the strong rule is applied to the classification waiting hand shake classification result of the classifier, and further, various kinds of hand gestures are classified. As a result, the merits of the rule feature and the classifier feature complement each other, and the accuracy of hand gesture identification is improved, and at the same time, the possibility of misjudgment is reduced and the robust recognition of the static hand gesture can be realized.

<2.5 Hand gesture identification system>
Next, with reference to FIG. 11, a hardware arrangement for realizing a hand gesture identification system according to an embodiment of the present invention will be described. As shown in FIG. 11, the hand gesture identification system 1100 is used to input an image or video for hand gesture identification from the outside. For example, a keyboard, a mouse, and a communication network and a remote input device connected thereto are used. Included input device 1110; used for implementation of the hand gesture identification method according to the above-described embodiment of the present invention, or implementation of the hand-shaking identification device according to the above-described embodiment of the present invention. A processing device 1120 that includes a chip, etc., can be connected to the Internet network (not shown), and obtains necessary data from the network according to the necessity of the processing process; externally identified in the above hand gesture identification result, that is, the input image Used to output specific hand gestures or invalid hand gesture alarms, for example, An output device 1130 including a ray, a printer, and a communication network and a remote output device to which the communication network is connected; and images, data, acquired results, commands, and intermediates related to the above-described processing processes in a volatile or non-volatile system It includes a memory device 1140 that is used to store data and includes various volatile or non-volatile memories such as random access memory (RAM), read only memory (ROM), hard disk, or semiconductor memory.

  Of course, for simplification, FIG. 11 shows only a part of the modules related to the present invention in the system. For example, modules such as a bus and an input / output interface are omitted. In addition, depending on the specific application situation, the system 1100 may include any other suitable module.

<3. Summary>
According to the present invention, an input image of a target scene including a hand is acquired, an image of a hand is extracted from the input image, and a hand image is classified using a classifier, thereby acquiring an identification waiting hand shake classification result Then, a rule feature is extracted from the image of the hand, and the rule feature is applied to the classification waiting hand shake classification result, thereby providing a gesture identification method, apparatus, and system for acquiring the gesture identification. This is a fusion of weak classifiers and strong rule decision technology, and classifiers are not used directly for the final identification of various hand gestures, and the majority of invalid hand gestures are eliminated by the filtering function of the classifiers. Therefore, a weak classifier built on the basis of the overall features, and at the same time, valid hand gestures can be reliably passed through the classifier with minimal leakage. After that, the strong rule is applied to the classification waiting hand shake classification result of the classifier, and further, various kinds of hand gestures are classified. As a result, the merits of the rule feature and the classifier feature complement each other, and the accuracy of the hand gesture identification is enhanced, and at the same time, the possibility of erroneous determination is lowered, and the robust hand gesture recognition can be realized.

  The hand gesture identification method and apparatus according to the embodiments of the present invention have been described in detail with reference to the accompanying drawings. The block diagrams of the devices, devices, and systems mentioned in this disclosure are only examples and are not intended to require or imply that they must be connected, arranged and arranged in the manner shown in the block diagrams. Absent. Those skilled in the art will know that these devices, devices, and systems can be connected, arranged and arranged in any manner. For example, the words “including”, “including”, “having”, etc. are words having an open meaning, which means “including but not limited to” and can be used interchangeably. The words “or” and “and” as used herein refer to the words “and / or” and can be used interchangeably as long as they are not clearly different from the preceding and following sentences. As used herein, “for example” refers to the phrase “for example but not limited to” and can be used interchangeably.

  The step flow charts in the present disclosure and the above method description are merely examples, and are not intended to imply or imply that the steps of each embodiment must be performed in the order presented. As will be appreciated by those skilled in the art, the order of the steps of the above embodiments can be performed in any order. For example, the words “after”, “then”, “follow”, etc. are not intended to limit the order of the steps, and these words are only used to let the reader read through the explanation of these methods Only. In addition, the use of, for example, the article “a”, “a” or “the” should not be construed as limiting any citation of the singular element to the singular.

  The foregoing description of the proposed public aspects is intended to enable any person skilled in the art to make or use the present invention. Various modifications to these aspects are also readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects, but do not depart from the scope of the invention. For this reason, the present invention is not intended to be limited to the aspects shown here, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (12)

Obtaining an input image of a target scene including a hand and extracting an image of the hand from the input image;
Classifying the image of the hand using a classifier, thereby obtaining a classification result of the identification waiting hand shake; and
Extracting rule features from the hand image;
Applying the rules characterized classification result of the identification waiting hand gestures, thereby viewing including acquiring a gesture identification result, and
The rule feature is a gesture identification method , which is a feature corresponding to a rule used when making a rule decision . Classifying the image of the hand using the classifier, thereby obtaining the identification waiting hand shake classification result,
Extracting classification features from the hand image;
Applying said classification features to the classifier, thereby viewing including acquiring a classification result of the identification waiting hand waving, a,
2. The gesture identification method according to claim 1, wherein the classification feature is a feature used when the classifier classifies the hand image .   3. The method according to claim 2, wherein the classification feature includes at least one of a hand contour convex hull feature, a hand contour Fourier descriptor feature, and a hand contour invariant matrix feature.   The rule feature includes at least one of the number of fingers, the length of the finger, the orientation pointed by the finger, the relative position between the fingers, and the depression angle between the two fingers. The hand gesture identification method according to any one of the items.   The gesture according to claim 1, wherein the classifier is a classifier obtained by optimizing a classification strategy using rule features, and the classification strategy includes a number of classifications and a designation strategy for classification. Identification method. The optimization of the classification strategy involves the following iterative process:
Using a training sample collection to train a classifier having a predetermined classification strategy;
Applying the sample collection evaluation to a classifier that trains, thereby obtaining sample classification results,
Applying predetermined rule features to the sample classification results, thereby obtaining sample hand gesture identification results;
6. The gesture identification method according to claim 5, comprising a step of optimizing a classification strategy of the classifier based on the sample gesture identification result.   The hand gesture identification method according to claim 1, wherein the classifier is a decision tree classifier, a random forest classifier, a perceptron classifier, a neural network classifier, or an SVM classifier.   The hand gesture identification method according to claim 1, wherein the classifier uses a template mapping classification technique.   The input image includes continuous K-frame input images, and if there is a matching M-frame gesture identification result in the K-frame input image, the M-frame gesture identification result is converted to the K-frame identification result. 2. The hand movement identification method according to claim 1, wherein the hand movement identification result in the input image is determined, K and M are predetermined positive integers, and K> M. An acquisition unit for acquiring an input image of a target scene including a hand and extracting an image of the hand from the input image;
Classifying the image of the hand using a classifier, thereby obtaining a classification result of identification waiting hand shake,
A feature extraction unit for extracting regular features from the hand image;
The application of the rules, wherein the classification result of the identification waiting hand gestures, thereby viewing including an identification section, a to get the gesture identification result,
The rule feature is a gesture identification device that is a feature corresponding to a rule used when making a rule decision .   A program for causing a computer to execute the hand gesture identification method according to any one of claims 1 to 9.   A computer-readable storage medium storing the program according to claim 11.

Images