TWI539329B - Gesture recognition system - Google Patents

Description

Gesture recognition system

The present invention relates to a gesture recognition system, and more particularly to a gesture recognition system executable in a complex scene.

The Natural User Interface (NUI) is a user interface that is invisible and does not require manual controls such as keyboards and mice. The interaction between people and machines can be achieved by static or dynamic gestures. Microsoft's somatosensory device Kinect is a vision-based gesture recognition system that uses static gestures or dynamic gestures to interact between the user and the computer.

The traditional vision-based gesture recognition system is easy to get the wrong judgment result when the object is recognized due to the surrounding light and the background object. After the self-identified object (here referred to as the hand) has acquired the feature, the training data is used to perform the classification to identify the gesture. The shortcomings of the traditional classification method are huge training materials or misjudgment due to unknown features.

In view of the above, it is therefore necessary to propose a novel gesture recognition system for recognizing static gestures and/or dynamic gestures more accurately and quickly.

In view of the above, one of the features of the embodiments of the present invention is to provide a robust gesture recognition system that can be correctly executed in a complex scene and can reduce the complexity of gesture classification.

According to an embodiment of the invention, the gesture recognition system includes a candidate point detection unit, a static gesture recognition unit, a multi-hand tracking unit, and a dynamic gesture recognition unit. The candidate point detecting unit receives the input image to generate a candidate point. The static gesture recognition unit recognizes the static gesture according to the candidate points. The multi-hand tracking unit tracks multiple hands by continuously inputting pairs between images. The dynamic gesture recognition unit obtains a momentum cumulative value according to the tracking path of the multi-hand tracking unit, thereby identifying the dynamic gesture.

The first figure shows a block diagram of a gesture recognition system 100 in accordance with an embodiment of the present invention. In this embodiment, the gesture recognition system 100 mainly includes a candidate node detection unit 11, a static gesture recognition unit 12, a multi-hand tracking unit 13, and a dynamic gesture recognition unit 14. Details of these units will be described in detail below. Gesture recognition system 100 can be executed by a processor, such as a digital image processor.

The second figure shows a flow chart of the steps performed by the candidate point detecting unit 11 of the first figure. In step 111 (ie, interactive feature extraction), features are captured according to color, depth, and momentum to generate a color reliable map, a depth reliable map, and a momentum reliable map.

The color reliability map is generated based on the skin color of the input image. In a color-reliable map, pixels that are closer to skin tone are assigned a higher value.

The depth reliable map is generated based on the depth of the hand of the input image. In the depth-reliable map, pixels located in the depth range of the hand are assigned a higher value. In one embodiment, the face is first identified by a face recognition technique, and then the depth of the face is determined relative to the depth of the recognized face.

Momentum reliable maps are generated based on the momentum of a series of input images. In the momentum reliable map, pixels with larger momentum are assigned to higher values, and the momentum can be measured using the sum of absolute differences (SAD) between the two input images.

In step 112 (ie, natural user scene analysis), the weights of the captured colors, depths, and momentum are determined relative to the operational state. The operational state can be an initial statement, momentum, or whether the hand is close to the face. The table shows some weights: Table 1         <TABLE style="BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; BORDER-COLLAPSE: collapse; BORDER-TOP: medium none; BORDER-RIGHT: medium none; mso-border-alt: double windowtext 1.5pt; Mso-yfti-tbllook: 1184; mso-padding-alt: 0cm 5.4pt 0cm 5.4pt; mso-border-insideh: 1.5pt double windowtext; mso-border-insidev: 1.5pt double windowtext"_0003"MsoTableGrid" border=" 1" cellSpacing="0" cellPadding="0"><TBODY><tr><td> Operation Status</td><td> Weight</td></tr><tr><td> Initial Statement </ Td><td> Momentum</td><td> Hand close to face</td><td> Color</td><td> Depth</td><td> Momentum</td></tr><tr ><td> No</td><td> Strong</td><td> No</td><td> 0.286 </td><td> 0.286 </td><td> 0.429 </td>< /tr><tr><td> No</td><td> Strong</td><td> Yes </td><td> 0.25 </td><td> 0.375 </td><td> 0.375 </td></tr><tr><td> No</td><td> Weak</td><td> No</td><td> 0.5 </td><td> 0.5 </td ><td> 0 </td></tr><tr><td> No</td><td> Weak</td><td> Yes </td><td> 0.4 </td><td > 0.6 </td><td> 0 </td></tr><tr><td> yes</td><td> strong</td><td> no </td><td> 0 </td><td> 0.4 </td><td> 0.6 </td></tr><tr><td> yes</td><td> weak</ Td><td> ignores</td><td> 0 </td><td> 1 </td><td> 0 </td></tr></TBODY></TABLE>

Finally, in step 113 (ie, the hybrid reliable map generation), the weights of the color, depth, and momentum provided in step 112 are used to combine the color reliability map, the depth reliability map, and the momentum reliability map, thereby generating a hybrid reliable map. This can detect candidate points.

The third figure shows a flow chart of the steps performed by the static gesture recognition unit 12 of the first figure. In step 121 (ie, dynamic palm cutting), the detecting hand (from the candidate point detecting unit 11) is cut into palms and arms, wherein the palms are used later and the arms are discarded.

In step 122 (ie, high-accuracy finger detection), a distance curve is generated by recording the distance between the center of the cutting palm and the edge of the cutting palm. The fourth graph illustrates a distance curve with five peaks indicating that five extended fingers are recognized.

In step 123 (ie, hierarchical gesture recognition), a plurality of gestures are classified to facilitate subsequent procedures. The fifth figure illustrates the classification of gestures obtained based on the number of recognized fingers. When using a hierarchical approach to recognize gestures, first decide the number of fingers to expand. The number of merged fingers can be known by calculating the width of the finger. Next, the void and its width determine the number of fingers that are bent between the unfolded fingers.

In the multi-hand tracking unit 13 of the first figure, the multi-hands are tracked by matching between successive frames, as illustrated in the sixth figure. There is a tracking path between matching tracking used. If the tracking hand is unmatched because the object leaves, the tracking path is deleted. If the tracking hand is mismatched due to object obscuration, the extrapolation technique can be used to generate the expected tracking hand. If the tracking hand is mismatched due to the entry of the object, a new gesture must be identified and a new path followed. When the mismatched tracking hand is found, it is fed back to the candidate point detecting unit 11 (first picture) for discarding the corresponding candidate point.

In the dynamic gesture recognition unit 14 of the first figure, the supervised tracking path is used to obtain a momentum cumulative value in each axial direction of the three-dimensional space, thereby identifying the dynamic gesture. The recognized dynamic gestures are fed to the natural user interface to perform predefined tasks.

Figure 7A shows the natural user interface for single hand drawing on the captured image. As illustrated in FIG. 7B, after using the static gesture No. 1 (not shown in the seventh B diagram), the user can continuously use the dynamic gesture composed of the static gesture No. 2 to draw a line, and the user can use the number 3 during the period. Or static gesture No. 4 to change the color.

The above description is only the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention; all other equivalent changes or modifications which are not departing from the spirit of the invention should be included in the following Within the scope of the patent application.

100 gesture recognition system 11 candidate point detection unit 111 interactive feature capture 112 natural user scene analysis 113 hybrid reliable map generation 12 static gesture recognition unit 121 dynamic palm cutting 122 high accuracy finger detection 123 hierarchical gesture recognition 13 Hand tracking unit 14 dynamic gesture recognition unit

The first figure shows a block diagram of a gesture recognition system in accordance with an embodiment of the present invention. The second figure shows a flow chart of the steps performed by the candidate point detecting unit of the first figure. The third figure shows a flow chart of the steps performed by the static gesture recognition unit of the first figure. The fourth figure illustrates the distance curve. The fifth figure illustrates the classification of gestures obtained based on the number of recognized fingers. The sixth figure illustrates the tracking of multiple hands by matching between successive frames. Figure 7A shows the natural user interface for single hand drawing on the captured image. Figure 7B shows the static gesture using Figure 7A to form a dynamic gesture.

100 gesture recognition system 11 candidate point detection unit 12 static gesture recognition unit 13 multi-hand tracking unit 14 dynamic gesture recognition unit

Claims (15)

A gesture recognition system includes: a candidate point detecting unit that receives an input image to generate a candidate point; a static gesture recognition unit that recognizes a static gesture according to the candidate point; and a multi-hand tracking unit by continuously inputting Pairing between images to track multiple hands; and a dynamic gesture recognition unit, according to the tracking path of the multi-hand tracking unit to obtain a momentum cumulative value, thereby identifying a dynamic gesture; wherein the candidate point detecting unit performs the following steps: according to the color , depth and momentum to capture features, thereby generating a color reliable map, a depth reliable map and a momentum reliable map; determining the weight of the color, depth and momentum relative to the operating state; and using the weights to combine the color reliably The map, the depth reliable map, and the momentum reliable map, thereby generating a hybrid reliable map to provide the candidate point. The gesture recognition system of claim 1, wherein the color reliability map is generated based on a skin color of the input image. The gesture recognition system of claim 1, wherein the depth reliable map is generated based on a depth of a hand of the input image. According to the gesture recognition system of claim 3, wherein the pixel in the depth range of the hand is assigned a higher value in the depth reliable map. The gesture recognition system of claim 1, wherein the momentum reliability map is generated based on momentum of a series of input images. The gesture recognition system of claim 5, wherein the momentum in the momentum reliability map is measured using a sum of absolute differences (SAD) between the two input images. The gesture recognition system of claim 1, wherein the operational state comprises an initial statement, momentum, whether the hand is close to the face, or a combination thereof. The gesture recognition system according to claim 1, wherein the static gesture recognition unit performs the steps of: cutting a palm from a corresponding hand of the candidate point; by recording between the center of the palm and the edge of the palm The distance is used to generate a distance curve to identify a static gesture; and to classify a plurality of static gestures. The gesture recognition system of claim 8, wherein the static gestures are classified according to the number of recognized deployed fingers. According to the gesture recognition system described in claim 1, in the multi-hand tracking unit, if the tracking hand is mismatched due to the object leaving, the corresponding tracking path is deleted. According to the gesture recognition system of claim 1, in the multi-hand tracking unit, if the tracking hand is mismatched due to object obscuration, a heterogeneous technique is used to generate an expected tracking hand. According to the gesture recognition system described in claim 1, in the multi-hand tracking unit, if the tracking hand is mismatched due to the entry of the object, a new tracking path is generated. According to the gesture recognition system described in claim 1, when the mismatched tracking hand is found, the multi-hand tracking unit is fed back to the candidate point detecting unit. The gesture recognition system of claim 1, wherein the recognized dynamic gesture is fed to a natural user interface to perform a predefined work. The gesture recognition system of claim 14, wherein a user uses the recognized dynamic gesture to draw a line according to the natural user interface.