KR101299031B1 - Handgesture recongnition apparatus and method for thereof - Google Patents

Abstract

Disclosed is a hand gesture recognition apparatus. The apparatus for recognizing a hand gesture includes an input unit for receiving a video photographing a user's movement, a search unit for searching for and tracking the position of the user's hand using the input video, and a gesture sequence using the searched and tracked hand position. A feature extractor for extracting the data; a storage unit for storing a plurality of sequence data; a pattern matching unit for calculating a statistical distance between each of the extracted sequence data and the plurality of stored sequence data; A recognition unit for recognizing the gesture as a gesture of the searched and tracked hand position, and an output unit for outputting the recognized gesture.

Description

HANDGESTURE RECONGNITION APPARATUS AND METHOD FOR THEREOF

The present invention relates to a hand gesture recognition apparatus and a method thereof, and more particularly, to a hand gesture recognition capable of actively responding to a change in the number of gestures or the number of users, and for easily analyzing time series data having different lengths. An apparatus and a method thereof are provided.

Gesture recognition is attracting attention as a method for efficient and natural interaction and information exchange between a user and a device along with the development of smart IT devices. In particular, the advantage of vision-based gesture recognition is that robot control, computer games can be performed at no additional cost by utilizing an image acquisition device built into the device without additional equipment such as a keyboard or a mouse as in conventional computer use. , Smart TV, smart phones, etc., many researches for utilizing this are in progress.

Existing researches for gesture recognition include Self-Organizing Map (SOM), Functional Data Analysis (FDA), Dynamic Time Warping (DTW), Hidden Markov Model (HMM) Various methods have been tried, such as model and neural network. They extracted the features expressing the gesture well and analyzed the characteristics of each motion to construct case-based reasoning or rule-based system.

However, the conventional method is sensitive to modified data and has a difficulty in requiring an additional learning process every time a new operation is added.

In addition, in the case of the traditional statistical or machine learning method using data at the feature extraction and classification stages, sufficient performance is needed when designing the model so that stable performance can be expected. In addition, as mentioned above, there is a problem that requires a separate learning process in order to add a new user or action.

Accordingly, an object of the present invention is to provide an apparatus and method for recognizing a hand gesture that can actively respond to a change in the number of gestures or the number of users, and can easily analyze time series data having different lengths.

As described above, an apparatus for recognizing a hand gesture according to the present invention includes an input unit for receiving a video photographing a user's movement, and a search unit for searching for and tracking the position of a user's hand using the received video. A feature extractor for extracting a gesture sequence using the searched and tracked hand position, a storage for storing a plurality of sequence data, a pattern for calculating a statistical distance between the extracted sequence data and each of the stored plurality of sequence data A matching unit, a recognition unit for recognizing a gesture corresponding to the sequence data having the lowest statistical distance as the gesture of the searched and tracked hand position, and an output unit for outputting the recognized gesture.

In this case, the storage unit preferably stores a plurality of sequence data classified through a nearest neighbor classifier (K-Nearest Neighbor Classifier).

On the other hand, the pattern matching unit, it is preferable to calculate the statistical distance using the following equation.

Wherein X is the extracted sequence data, Y is stored sequence data, d (X, Y) is a statistical distance between X and Y, and corr (X, Y) is a correlation between X and Y The coefficient, cov (X, Y), is the covariance between X and Y, and std (X, Y) is the standard deviation between X and Y.

The storage unit preferably stores the plurality of sequence data as a vector value.

On the other hand, it is preferable that the feature extractor extracts the gesture sequence as a vector value.

On the other hand, the hand gesture recognition method according to the present embodiment, the step of receiving a video captured by the user's movement, using the received video to search for the location of the user's hand, tracking, the search and tracked Extracting a gesture sequence using a hand position, calculating a statistical distance between the extracted sequence data and each of the plurality of prestored sequence data, and searching for a gesture corresponding to the sequence data having the lowest statistical distance. And recognizing it as a gesture of the tracked hand position, and outputting the recognized gesture.

In this case, the prestored plurality of sequence data is preferably a plurality of sequence data classified through a nearest neighbor classifier (K-Nearest Neighbor Classifier).

On the other hand, the step of calculating the statistical distance, it is preferable to calculate the statistical distance using the following equation.

Wherein X is the extracted sequence data, Y is stored sequence data, d (X, Y) is a statistical distance between X and Y, and corr (X, Y) is a correlation between X and Y The coefficient, cov (X, Y), is the covariance between X and Y, and std (X, Y) is the standard deviation between X and Y.

On the other hand, the plurality of pre-stored sequence data preferably has a vector value.

In the extracting step, the gesture sequence may be extracted as a vector value.

As described above, the hand gesture recognition apparatus and the hand gesture recognition method according to the present embodiment can actively respond to the change in the number of gestures or the number of users by using a K-Nearest Neighbor Classifier, and perform dynamic time matching (DTW). ), The analysis of time series data of different lengths can be easily performed. In addition, by using statistical correlation information on time series data sorted using dynamic time matching (DTW), the similarity between two time series data is measured and used in the recognition process, thereby enabling more stable and robust performance recognition.

1 is a block diagram showing the configuration of a hand gesture recognition apparatus according to an embodiment of the present invention;
2 is a view for explaining the operation of the pattern matching unit of FIG.
3 is a view showing an example of data used in an experiment of a hand gesture recognition apparatus according to the present embodiment, and
4 is a diagram illustrating an example of an image used in an experiment of a hand gesture recognizing apparatus according to the present embodiment;
5 and 6 are diagrams showing experimental results of the hand gesture recognition apparatus according to the present embodiment;
7 is a flowchart illustrating a hand gesture recognition method according to an embodiment of the present invention.

Hereinafter, a temporal example of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating a configuration of an apparatus for recognizing a hand gesture according to an exemplary embodiment.

Referring to FIG. 1, the hand gesture recognizing apparatus 100 may include an input unit 110, an output unit 120, a storage unit 130, a search unit 140, a feature extractor 150, and a pattern matching unit 160. ), The recognition unit 170 and the controller 180. The hand gesture recognizing apparatus 100 may be a device that performs only hand gesture recognition, and may be a smart TV or a game device that uses hand gesture recognition as user input information. .

The input unit 110 receives a video captured by the user's movement. Specifically, the input unit 110 can receive a moving image captured by an external imaging device. Meanwhile, in the present exemplary embodiment, only the process of receiving and processing a video captured by an external image capturing apparatus is described, but the input unit 110 may be implemented as an image capturing device to directly generate a video by capturing a user.

The output unit 120 outputs the recognized gesture. In detail, the outputter 120 may output a gesture recognized by the recognizer 170 to be described later to the user. For example, the output unit 120 may be implemented as a display device such as a CRT or an LCD, and may display a recognized gesture. In addition, the output unit 120 may be implemented as a communication interface capable of communicating with an external device, and may be implemented in a form of displaying or using a gesture recognized by the external device by transmitting information of the recognized gesture.

The storage unit 130 may store the input moving image. In detail, the storage unit 130 may temporarily store a video received through the input unit 110.

일 수 있다. 이와 같이 최근접 이웃 분류기로 분류된 시퀀스 데이터를 저장하는바, 새로운 사용자 또는 제스처가 추가되더라도, 학습 데이터를 등록하는 과정만으로 학습이 수행될 수 있게 된다. The storage unit 130 stores a plurality of sequence data. Such a plurality of sequence data is a plurality of sequence data classified through a nearest neighbor classifier (K-Nearest Neighbor Classifier). And each sequence data is m × L? Vector of dimensions?

Lt; / RTI > As such, the sequence data classified by the nearest neighbor classifier is stored. Even though a new user or gesture is added, learning can be performed only by registering the learning data.

The storage unit 130 may include a storage medium and an external storage medium in the hand gesture recognition apparatus 100, for example, a removable disk including a USB memory, a storage medium connected to a host, and a web server through a network. And the like.

The searcher 140 searches for and tracks the position of the user's hand using the received video. In detail, the searcher 140 may search for the position of the hand in the input video and track the change in the searched position of the hand in chronological order.

일 수 있다.The feature extractor 150 extracts a gesture sequence using the searched and tracked hand position. In detail, the feature extractor 150 extracts one gesture sequence to be used for gesture recognition using the hand position searched and tracked by the searcher 140. This gesture sequence is m × L? Vector of dimensions?

Lt; / RTI >

The pattern matching unit 160 calculates a statistical distance between the extracted sequence data and each of the stored plurality of sequence data. A detailed operation of the pattern matching unit 160 will be described later with reference to FIG. 2. As such, the pattern matching unit 160 does not perform additional operations for feature extraction by using the information on the gesture sequence extracted by the feature extraction unit 150 as it is.

The recognition unit 170 recognizes the gesture corresponding to the sequence data having the lowest statistical distance as the gesture of the searched and tracked hand position. The recognition unit 170 may compare the calculated statistical distance with a preset value. For example, when a hand gesture that is completely different from the hand gesture corresponding to the stored sequence data is input, the calculated statistical distance and the preset value may be set to prevent the input hand gesture from being recognized as one of the stored hand gestures. In comparison, when all of the calculated statistical distances are larger than the predetermined value, it may be recognized that there is no matched hand gesture.

The controller 180 controls each component in the hand gesture recognition apparatus 100. Specifically, when a video is input through the input unit 110, the controller 180 controls the searcher 140 and the feature extractor 150 to extract the gesture sequence from the input video, and stores the extracted gesture sequence. The pattern matching unit 160 may be controlled to calculate a statistical distance between gesture sequences previously stored in the unit 130. When the statistical distance is calculated, the controller 180 recognizes that the gesture corresponding to the gesture sequence closest to the calculated statistical distance is included in the input video, and controls the output unit 120 to output the recognized gesture. Can be.

Meanwhile, the controller 180 may control the output unit 120 to output no recognized hand gesture when the user's hand position is not extracted from the input image or when only a statistical distance of more than a predetermined value is calculated. Can be.

Therefore, the hand gesture recognizing apparatus 100 according to the present embodiment measures statistical similarity between two time series data by additionally using statistical correlation information on time series data arranged by using dynamic time matching (DTW) to use in the recognition process. This allows more stable and robust performance recognition. In addition, by using the nearest neighbor classifier, it is possible to actively respond to the change in the number of gestures or the number of users, and it is also possible to easily analyze time series data having different lengths using the dynamic time matching (DTW) method.

FIG. 2 is a diagram for describing an operation of the pattern matching unit of FIG. 1.

Referring to FIG. 2, the pattern matching method according to the present embodiment has two steps. In the first step, we first use dynamic time matching (DTW) to align two samples of different lengths. Here, dynamic time matching is a method that can calculate an optimal path for aligning two sequence data and can also be used to calculate the distance between time series data. However, in gesture recognition, the distance calculation through this dynamic time matching is not suitable due to various factors between users or between gestures. As an alternative to this, in this embodiment, a statistical measure is used to calculate the distance between two aligned sequences, which is expressed by Equation 1 below.

Where X is the extracted sequence data, Y is stored sequence data, d (X, Y) is the statistical distance between X and Y, corr (X, Y) is the correlation coefficient between X and Y, cov (X, Y) Is the covariance between X and Y, and std (X, Y) is the standard deviation between X and Y.

?라 하고, 시각 t에서 m개의 값을 가진다고 하자(즉, _ ). 그렇다면, 하나의 제스처 시퀀스 데이터는 m×L? 차원의 벡터 ?

로 표현할 수 있게 된다. 즉 수식 (1)을 통해 벡터화된 데이터들 사이의 통계적 거리를 계산할 수 있게 된다. In order to use Equation 1 for gesture recognition, a process of converting a gesture sequence into a vector form is required. To do this, we first use sequence data of length L

Suppose we have m values at time t (that is, _). If so, one gesture sequence data is m × L? Vector of dimensions?

Can be expressed as In other words, it is possible to calculate the statistical distance between the vectorized data through the equation (1).

3 to 6 are diagrams showing experimental results of the hand gesture recognition apparatus according to the present embodiment.

Specifically, to determine the suitability of the proposed method, a comparison experiment with the conventional distance calculation method using dynamic time matching (DTW) was performed. Since the classifier used in the proposed method is influenced by the number of learning data, the recognition rate is observed by changing the number of learning data to find the number of learning data suitable for the actual application. Next, in actual applications, the number of gestures may vary, so the change in recognition rate according to the number of gestures is also observed. Each experiment was repeated 50 times the process of selecting the learning and test data through random sampling and observing the recognition rate, and summarized the average recognition rate at this time.

3 is a summary of the data used in the experiment.

Referring to FIG. 3, the data of FIG. 3 is obtained by obtaining five gestures from four users in two situations in which the distances from the imaging apparatus are different, and the five gestures are respectively (A) from left to right and (B) from top to bottom. , (C) clockwise rotation, (D) counterclockwise rotation, and (E) push 'gestures. This data is a hand search, tracking, and gesture segmentation process from video sequence to preprocessing. It consists of 167 data. One gesture data is a time series in which coordinates in three-dimensional space are sequenced over time. Data.

4 is a sample image extracted from a video input during an experiment.

Referring to FIG. 4B, the red dot on the figure indicates the center of the searched hand.

5 shows a change in the recognition performance according to the conversion of the number of training data. Here, the horizontal axis represents the number of learning data for each gesture, and the vertical axis represents the average recognition rate. In addition, "DTW + CORR" represents the performance of the recognition method according to the present embodiment, "DTW" refers to the existing method used for comparison.

Referring to FIG. 5, it can be seen that the proposed method always shows better performance than the existing method regardless of the number of training data. In addition, we can observe that the proposed method outperforms the existing method when the number of training data is small. This suggests that the proposed method is more suitable when the number of learning data is small. The proposed method showed stable recognition performance when the number of training data is 5 or more. On the other hand, two types of dynamic time matching (DTW) algorithms were used in the experiments, which were represented as 3-step and 5-step, respectively. there was.

6 is a diagram illustrating a change in recognition performance according to the number of gestures.

Referring to FIG. 6, it can be seen that an increase in the number of gestures causes a decrease in recognition performance. This is because the complexity of the problem increases as the number of gestures increases. Nevertheless, it can be seen that the degree of reduction of the recognition device according to the present embodiment is smaller than that of the conventional method. While the recognition method continuously decreases with the increase of the number of gestures, the proposed method can reduce the proposed method up to five. Increasing the number of gestures was also observed to maintain about 90% recognition performance.

For reference, although it was seen in the previous experiment, the proposed gesture recognition system has the advantage that even if a new gesture is added, the new gesture can be recognized only by registering new learning data without complicated learning process like SVM or HMM.

7 is a flowchart illustrating a hand gesture recognition method according to an embodiment of the present invention.

First, a video captured by a user's movement is received (S710). In detail, an external image capturing apparatus may receive a video captured by a user. Meanwhile, in the present exemplary embodiment, only the process of receiving and processing a moving image captured by an external image capturing apparatus has been described, but such a step may be implemented by directly capturing a user's movement.

Then, the position of the user's hand is searched and tracked using the input video (S720). Specifically, the position of the hand may be searched in the input video, and the change of the position of the searched hand may be tracked in chronological order.

일 수 있다.Then, the gesture sequence is extracted using the searched and tracked hand position (S730). Specifically, one gesture sequence to be used for gesture recognition may be extracted using the searched and tracked hand position. This gesture sequence is m × L? Vector of dimensions?

Lt; / RTI >

In operation S740, a statistical distance between the extracted sequence data and each of the plurality of prestored sequence data is calculated. The operation of calculating the statistical distance has been described above with reference to FIG. 2, and thus redundant description thereof will be omitted.

The gesture corresponding to the sequence data having the lowest statistical distance is recognized as the gesture of the searched and tracked hand position (S750). In this case, the calculated statistical distance may be compared with a predetermined value. For example, when a hand gesture that is completely different from the hand gesture corresponding to the stored sequence data is input, the calculated statistical distance and the preset value may be set to prevent the input hand gesture from being recognized as one of the stored hand gestures. In comparison, when all of the calculated statistical distances are larger than the predetermined value, it may be recognized that there is no matched hand gesture.

In operation S760, the recognized gesture is output. In detail, a gesture recognized by a display device such as a CRT, an LCD, or the like may be output, or a gesture recognized by an external device may be displayed or transmitted to the external device.

Therefore, the hand gesture recognition method according to the present embodiment is more stable by measuring the similarity between two time series data by additionally using statistical correlation information on time series data arranged using dynamic time matching (DTW). And robust performance recognition is possible. In addition, by using the nearest neighbor classifier, it is possible to actively respond to the change in the number of gestures or the number of users, and it is also possible to easily analyze time series data having different lengths using the dynamic time matching (DTW) method. The hand gesture recognition method as shown in FIG. 7 may be executed on the hand gesture recognition apparatus having the configuration of FIG. 1 or may be executed on the hand gesture recognition apparatus having another configuration.

In addition, the hand gesture recognition method as described above may be implemented with at least one execution program for executing the hand gesture recognition method as described above, and the execution program may be stored in a computer-readable recording medium.

Thus, each block of the present invention may be embodied as computer readable code on a computer readable recording medium. The computer readable recording medium may be a device capable of storing data that can be read by a computer system.

Although the above has been illustrated and described with respect to preferred embodiments of the present invention, the present invention is not limited to the above-described specific embodiments, and the present invention belongs to the present invention without departing from the gist of the present invention as claimed in the claims. Anyone of ordinary skill in the art can make various modifications, and such changes are within the scope of the claims.

100: hand gesture recognition device 110: input unit
120: output unit 130: storage unit
140: search unit 150: feature extraction unit
160: pattern matching unit 170: recognition unit
180:

Claims (10)

In the hand gesture recognition device,
An input unit configured to receive a video photographing a user's movement;
A search unit for searching for and tracking the position of a user's hand using the received video;
A feature extractor which extracts gesture sequence data using the searched and tracked hand position;
A storage unit which stores a plurality of sequence data stored in the nearest neighbor classifier (K-Nearest Classifier);
A pattern matching unit configured to calculate a statistical distance between the extracted gesture sequence data and each of the stored plurality of sequence data;
A recognition unit recognizing a gesture corresponding to the sequence data having the lowest statistical distance as the gesture of the searched and tracked hand position; And
And an output unit configured to output the recognized gesture.
Wherein the pattern matching unit comprises:
An apparatus for recognizing a hand gesture, wherein the statistical distance is calculated using the following equation:

Wherein X is the extracted gesture sequence data, Y is stored sequence data, d (X, Y) is a statistical distance between X and Y, and corr (X, Y) is between X and Y The correlation coefficient, cov (X, Y) is the covariance between X and Y, and std (X) is the standard deviation of X. Std (Y) is the standard deviation of Y. The method of claim 1,
Wherein the pattern matching unit comprises:
And performing a sort on each of the extracted gesture sequence data and the stored plurality of sequence data, and calculating a statistical distance between the sorted gesture sequence data and each of the sorted sequence data. . The method of claim 2,
Wherein the pattern matching unit comprises:
And performing alignment on each of the extracted gesture sequence data and the stored plurality of sequence data using dynamic time matching (DTW). The method of claim 1,
Wherein,
Hand gesture recognition apparatus, characterized in that for storing the plurality of mood-like sequence data as a vector value. The method of claim 1,
The feature extraction unit may extract,
And extracting the gesture sequence data as a vector value. In the hand gesture recognition method,
Receiving a video photographing the movement of the user;
Searching for and tracking the position of a user's hand using the received video;
Extracting gesture sequence data using the searched and tracked hand position;
Calculating a statistical distance between the extracted gesture sequence data and each of a plurality of previously stored sequence data stored through the nearest neighbor classifier (K-Nearest Neighbor Classifier);
Recognizing a gesture corresponding to the sequence data having the lowest statistical distance as the gesture of the searched and tracked hand position; And
Outputting the recognized gesture;
Wherein the calculating the statistical distance comprises:
Hand gesture recognition method characterized in that to calculate the statistical distance using the following equation.

X is the extracted gesture sequence data, Y is stored sequence data, d (X, Y) is a statistical distance between X and Y, and corr (X, Y) is between X and Y The correlation coefficient, cov (X, Y) is the covariance between X and Y, and std (X) is the standard deviation of X. Std (Y) is the standard deviation of Y. The method according to claim 6,
Wherein the calculating the statistical distance comprises:
Hand gesture recognition, wherein the extracted gesture sequence data and the previously stored plurality of sequence data are aligned, and a statistical distance between the aligned gesture sequence data and each of the aligned plurality of sequence data is calculated. Way. The method of claim 7, wherein
Wherein the calculating the statistical distance comprises:
And aligning each of the extracted gesture sequence data and the plurality of previously stored sequence data using dynamic time matching (DTW). The method according to claim 6,
The pre-stored plurality of sequence data,
Hand gesture recognition method characterized in that it has a vector value. The method according to claim 6,
Wherein the extracting comprises:
And extracting the gesture sequence data as a vector value.

Images