KR101511146B1 - Smart 3d gesture recognition apparatus and method - Google Patents

Abstract

The present invention relates to a smart three-dimensional gesture recognition device and a method thereof. The invention consists of the followings: a video acquisition system to acquire video data made up of multiple continuous frames; a hand tracking system to acknowledge and track the location of a user′s hand using both color-based and depth-based tracking of video; a gesture decision system to analyze the location of the hand identified as a result of the hand tracking system to decide whether the user makes an intentional hand gesture; a characteristics extraction system to take out the characteristics of hand gesture based on settings; a user profile storage system where multiple templates of hand gestures are stored; a recognition classification system to recognize the patterns of a hand gesture based on the extracted characteristics from the extraction system, produce user′s behavior profile based on the patterns, recognize the gesture by searching for a gesture template corresponding to a recognized hand gesture out of the multiple templates stored in the user profile storage system and by applying the recognized pattern to the gesture template found as a result; and an instruction conversion system to convert the recognized gesture into an instruction corresponding thereto.

Description

[0001] SMART 3D GESTURE RECOGNITION APPARATUS AND METHOD [0002]

The present invention relates to a gesture recognition apparatus and method, and more particularly, to a non-wearable recognition-based smart three-dimensional gesture recognition apparatus and method.

The gesture recognition device is mainly a device for detecting and recognizing the movement of the user's hand or finger, that is, a gesture, and generating a command corresponding to the recognized gesture, and the application field is gradually enlarged It is going.

The gesture recognition device is largely classified into a wearable type and a non-wearable type according to a method of recognizing a user's gesture. The wearable gesture recognition apparatus performs an operation in a state in which the user wears the gesture recognition apparatus and recognizes the user's operation by using a plurality of sensors such as an acceleration sensor, a geomagnetism sensor, a gravity sensor, and a gyro sensor. On the other hand, the non-wearing gesture recognition apparatus is an image-based gesture recognition apparatus which detects a user's operation and recognizes a user's command by using an image acquisition means such as a camera.

Unlike the wearable gesture recognition device, the non-wearable gesture recognition device has a variety of applicable fields because of convenience that the user does not need to wear the gesture recognition device. However, the gesture recognition device does not reflect the difference depending on the behavior characteristics of each user There is a problem that the recognition rate depends on the behavior characteristics of the user. In addition, since the conventional non-wearable gesture recognition device does not directly contact the user's body, it is configured to provide only two-dimensional visual feedback on the gesture recognition, which limits the user's inconvenience and fatigue.

Korean Patent Laid-Open No. 10-2012-0029738 entitled " Method and interface for recognizing a user dynamic engine gesture, and an electric device using the same "(published on Mar. 23, 2012) A hand region is detected so that a hand gesture can be easily recognized even when there is a change in illumination or a noise occurs. However, the above-mentioned technique does not take into consideration the behavior characteristics of each user, and thus there is a limit to improve the accuracy of the gesture recognition.

An object of the present invention is to provide a smart three-dimensional gesture recognition device capable of improving the accuracy of gesture recognition in consideration of the behavior characteristics of individual users.

Another object of the present invention is to provide a smart three-dimensional gesture recognition method for achieving the above object.

According to an aspect of the present invention, there is provided an apparatus for recognizing a gesture, comprising: an image acquiring unit acquiring an image composed of a plurality of consecutive frames; A hand tracking unit for detecting and tracking a position of a user's designated hand by simultaneously using a color-based method and a depth-based method in the image; A gesture determining unit for analyzing the position information of the hand tracked by the hand tracking unit and determining whether the user intends the gesture using the analyzed positional change of the hand; A feature extraction unit for extracting a feature of the hand position change in a preset manner; A user profile storage unit storing a plurality of gesture templates; Recognizing a pattern of the gesture from the extracted features, generating the recognized pattern as a user behavior profile, searching a gesture template corresponding to the user behavior profile among the plurality of gesture templates stored in the user profile storage unit A recognition classifier recognizing the gesture by applying the recognized pattern to a searched gesture template; And a command conversion unit for converting the recognized gesture into a corresponding user command; .

Wherein the hand tracer detects a hand using a local binary pattern (LBP) for a color vector of an Lab color space in the color-based manner, And tracks the detected hand of the user according to the CAMSHIFT algorithm.

Wherein the gesture determination unit calculates a motion gradient for a positional change of the hand being tracked and if the calculated motion gradient is equal to or greater than a predetermined upper limit value or equal to or lower than a predetermined lower limit value, .

And the feature extracting unit extracts the speed of the hand position change with the feature.

The recognition and classification unit recognizes the pattern of the gesture using dynamic time warping (DTW).

If the gesture template corresponding to the user behavior profile is not found, the recognition and classification unit may filter the user behavior profile with a Kalman filter to generate a new gesture template, and store the generated gesture template in the user profile storage unit .

Wherein the gesture recognition apparatus determines that the user's hand is not detected in the hand detection unit or that the gesture is not intended by the user in the gesture determination unit and when the recognition classification unit recognizes the gesture A feedback control unit for generating feedback so that the user can recognize at least one of visual, auditory, and tactile senses when at least one occurs; And further comprising:

According to another aspect of the present invention, there is provided a gesture recognition method including an image acquisition unit, a hand tracking unit, a gesture determination unit, a feature extraction unit, a user profile storage unit, a recognition classification unit, A gesture recognition method of a device, comprising: acquiring an image composed of a plurality of consecutive frames; Detecting and tracking a position of a user's designated hand by simultaneously using a color-based method and a depth-based method in the image; Analyzing position information of the hand in which the gesture judging unit is traced, and determining whether the user intends the gesture using the analyzed hand position change; Extracting a feature of the hand position change in a predetermined manner; Recognizing a pattern of the gesture from the feature extracted by the recognition and classification unit; Wherein the recognizing and classifying unit generates the recognized pattern as a user behavior profile, searches a gesture template corresponding to the user behavior profile among the plurality of gesture templates stored in the user profile storage unit, Recognizing the gesture by applying the gesture template; And converting the recognized gesture into a corresponding user command; .

Accordingly, the smart three-dimensional gesture recognition apparatus and method according to the present invention maximizes gesture recognition accuracy by determining whether a gesture is a gesture by analyzing a gesture intention of a user at the time of gesture recognition of a user, can do. Therefore, it can be widely applied to various applications such as realistic media, games, augmented reality, and telepresence that require natural sensation interaction. In addition, it can be used for interaction with 3D image contents, which can contribute to the development of 3D image display industry. In addition, by providing multiple sensory feedback, the user can easily recognize the gesture recognition state abnormality, etc., thereby improving the usability of the gesture recognition device.

FIG. 1 shows a configuration of a smart three-dimensional gesture recognition apparatus according to an embodiment of the present invention.
2 shows an application example in which the depth information is used as a feature to detect a user's hand.
FIG. 3 is a diagram showing a difference between DTW matching in consideration of time matching and general matching in consideration of time variability.
FIG. 4 shows an embodiment of recognizing a user gesture pattern using a DTW matching technique considering a time-variancy of a recognition classification part.
FIG. 5 shows an example of a comparison between a user adaptive template analyzing a user profile and a template not considering characteristics of a user.
FIG. 6 is a graph showing a result of experiment of changing the recognition rate of the gesture according to the feedback method.
7 illustrates a smart three-dimensional gesture recognition method according to an embodiment of the present invention.

In order to fully understand the present invention, operational advantages of the present invention, and objects achieved by the practice of the present invention, reference should be made to the accompanying drawings and the accompanying drawings which illustrate preferred embodiments of the present invention.

Hereinafter, the present invention will be described in detail with reference to the preferred embodiments of the present invention with reference to the accompanying drawings. However, the present invention can be implemented in various different forms, and is not limited to the embodiments described. In order to clearly describe the present invention, parts that are not related to the description are omitted, and the same reference numerals in the drawings denote the same members.

Throughout the specification, when an element is referred to as "including" an element, it does not exclude other elements unless specifically stated to the contrary. The terms "part", "unit", "module", "block", and the like described in the specification mean units for processing at least one function or operation, And a combination of software.

FIG. 1 shows a configuration of a smart three-dimensional gesture recognition apparatus according to an embodiment of the present invention.

Referring to FIG. 1, the smart three-dimensional gesture recognition apparatus 100 of the present invention includes an image acquisition unit 110, a hand tracking unit 120, a gesture determination unit 130, a feedback control unit 140, 150, a recognition classifying unit 160, a user profile storing unit 170, and an instruction converting unit 180.

The priority image acquiring unit 110 acquires an image composed of a plurality of consecutive frames. In the present invention, the image acquiring unit 110 is configured to acquire not only existing color images but also depth images.

Most of the non-contact gesture recognition devices use image-based gesture recognition techniques. Image-based gesture recognition techniques are classified into color-based recognition methods and depth-based recognition methods.

The color-based recognition method transforms the acquired image into a color space such as RGB, HSV, HIS, YUV, YIQ, and YcbCr to find and recognize the hand. The color-based recognition method generally uses a method of detecting a skin color of a user, but it is vulnerable to a change in ambient illumination in an image, and thus has a disadvantage of low gesture recognition rate.

On the other hand, the depth-based recognition method has a merit that it is robust against ambient illumination change in the image, but it has a disadvantage that the image that can be acquired is simple compared with the color-based image. Also, since the sensor that can acquire the depth image to use the depth - based recognition technique is expensive and the resolution is low, the utilization rate is not high. Recently, structured light, which adds a unique pattern to light originating from a light source, has been developed for object recognition, and a sensor capable of acquiring color image and depth image in real time at a low price has been developed, There have been a lot of studies on gesture recognition methods.

The image acquisition unit 110 acquires the color image and the depth image together using the structured light, and transmits the acquired image to the hand tracking unit 120. [ At this time, it is assumed that the color image is an RGB image, but it may be another color image based image.

The hand tracking unit 120 detects and traces the user's hand or finger from the image transmitted from the image obtaining unit 110. [ The hand tracking unit 120 may detect a hand or a finger in various ways. For example, a skin color detection method using a Bayesian model and a depth image restricting method may be used to detect a hand or a finger .

If the conditional probability function class is P (c | skin) and P (c | nonskin), respectively, for the color vector c of the color space, 1 < / RTI >

Where c is the color vector of the color space and P (c | skin) and P (c | nonskin) are the conditional probability function classes for skin and skin, respectively, and θ is a threshold value.

In Equation (1), the threshold value (?) Can be obtained according to Equation (2).

Where f (skin) and p (nonskin) are prior probabilities for skin and non-skin, respectively, and λ _fd and λ _fr are the false detection and false detection costs, ).

However, the above-described hand detection method using the skin color detection method has a problem in that, when a color similar to the skin color of the face, leg, or human exists in the image, the hand to be detected can be treated as noise. Therefore, in the present invention, a semi-naive classifier is used to robustly detect a user's hand.

If the feature F is extracted from the image when the user's _hand is defined as C _hand and the user's hand probability is defined as the conditional probability P (C _hand | F) . If the number of features to be used to track the user's hand is m, then the feature is F _{i = 1, 2, ... m} , where m different characteristics must be independent in order to use the semi-Naive classifier.

The features for detecting the user's hand can be skin color vectors, motion vectors, texture information, location vectors, and the like. However, since the present invention seeks a natural user interface (NUI) and robust detection of the user's hand, it is necessary to use the RGB It uses texture information of local binary pattern (LBP) with different color vectors, depth information, and radius in color space. Various color features such as RGB, HSV, YCbCr, and LUV can be used. However, in the present invention, a color vector of a Lab color space is used to extract robust features of a hand. LBP is a robust, fast and robust texture descriptor that can be expressed as LBP as an intensity value g _c corresponding to a particular pixel (x _c , y _c ) have.

(Where P denotes the number of surrounding sampling points with respect to the center pixel, and R denotes the distance between the surrounding pixel and the center pixel).

Other texture descriptors, such as census transforms and modified census transforms, can be used to detect texture features, but the present invention allows the user to track the user's hand using the most common and popularized LBPs. To prove that it is possible, LBP is used.

Although the user's hand can be tracked through the process, the depth information obtained through the RGB-D camera can be used as a feature to trace the user's hand more robustly.

In order to use the classifiers and first need to machine learning, and displays the location of the hand in the image, the user's hand _{taken, P (F i = 1,} 2, ... m | C hand) and P (F _{i = 1, 2 , ... m} | C _nonhand ), respectively. After finishing the learning process, the candidate group of the user's hand can be detected by Equation (4).

In the present invention, i is 2, and F is a combination of three or more independent features because it can be considered only two cases, not a hand and a hand. Here, the probability that the detected feature is the hand loss of the user can be obtained by Equation (5).

The likelihood of not being the user's hand can also be obtained in the same manner as in Equation (5), and a probability that the user's hand is located in the image is obtained by selecting a larger value among the two probability values. In this way, candidates of the user's hand can be obtained.

2 shows an application example in which the depth information is used as a feature to detect a user's hand.

As shown in FIG. 2, in a method of detecting a user's hand using depth information as a feature, a hand detecting unit 120 is learned using a machine learning method for recognizing and displaying the position of a hand in a plurality of images , The hand detection unit 120 detects the hand using the conditional probability with respect to the input image.

In addition, the hand tracking unit 120 uses a hand and finger tracking algorithm to efficiently perform computation and real-time operation so as to provide feedback quickly when providing feedback to a user. Various algorithms can be used as the tracking algorithm, but here we use the CAMSHIFT algorithm as an example.

The CAMSHIFT algorithm

1. setting a region of interest (ROI) of a probability distribution image for the entire image;

2. selecting an initial position of a mean shift search window; The selected location here is the target distribution being tracked.

3. calculating a color probability distribution of a centered region in an average movement search window;

4. Repeating the averaging algorithm to find the center point of the probability image, storing the 0th moment (distribution area) and the center point location;

5. For a subsequent frame, setting a search window centered on the mean position searched in step 4 and having a function of the 0 < th >

6. Repeat step 3 to repeat;

.

The 0th, 1st and 2nd moments in the above CAMSHIFT algorithm can be calculated as Equation (6).

(Where M ₀₀ is the 0th moment, M ₁₀ and M ₀₁ are the first moments, M ₂₀ and M ₀₂ are the second moments, and P (x, y) is the color probability distribution Lt; / RTI >

The size of the search window updated every frame through the CAMSHIFT algorithm is calculated by Equation (7).

(Where width and height are the width and height of the search window, respectively).

In the case of an RGB-D camera capable of simultaneously acquiring an RGB image and a depth image, there is a problem in that the position of the user's hand or finger, which is detected and tracked due to noise existing in the depth image, is not accurate. Such noise may lead to deterioration of the recognition rate at the time of recognizing the gesture, so there is a need to remove noise. In the present invention, the prediction filter is applied to the hand to be tracked, thereby removing noise and improving the gesture recognition rate. Since the prediction filter is a known technique, it is not described in detail here.

If the hand tracking unit 120 tracks a hand or a finger in the image, the gesture determining unit 130 determines whether the position of the user's hand or finger tracked in each frame is a motion gradient technique or a moment of inertia technique Analyze. Although the operation of the user included in the acquired image is intentionally performed so that the gesture recognition apparatus recognizes the gesture, there are cases where the unintended operation of the user appears similar to the gesture. Accordingly, the gesture judging unit 130 discriminates and excludes unintentional gestures of the user, thereby raising the gesture recognition rate.

In the present invention, the position information of a hand or a finger is analyzed using a motion gradient to determine a gesture intention of a user.

The motion gradient technique calculates the motion gradient of the hand or finger position tracked in each frame by the hand tracking unit 120 as shown in equation (8).

(Where ∇f represents the motion gradient of the hand or finger).

Gesture when the judgment unit 130 when the gradient of the hand or finger to be tracked in the t-th frame is called G _T, the gradient (G _T) is a predetermined upper limit value (G _max) or more or less than the lower limit value (G _min), the user and The gesture is judged to be an intended gesture, while in other cases, the gesture is not intended by the user. That is, the motion gradient caused by a general operation in which the user is not conscious of the gesture judges that the gesture is unintentional.

If the gesture determination unit 130 determines that the user's action is a gesture, the feature extraction unit 150 extracts a feature to be used for gesture recognition using the position of the hand. The features for gesture recognition can be hand position, direction, speed, etc. However, by using dual velocity, features can be extracted quickly. Equation (9) is a formula for extracting features using velocity.

(Where F is a feature in the t-th frame and P _t (X, Y, Z) is the position of the hand detected and tracked in the t-th frame).

When the feature extraction unit 150 extracts the feature of the gesture, the recognition and classification unit 160 recognizes the gesture by recognizing the pattern of the gesture from the extracted feature. At this time, the recognition classifying unit 160 compares characteristics of the gesture extracted by the feature extracting unit 150 with a plurality of gesture templates stored in the user profile storage unit 170, extracts the most similar gesture templates, and recognizes the user gesture User-adaptive gesture recognition can be performed to improve gesture recognition performance.

The gestures recognized through the motion of the hand or the finger can be largely divided into a static pattern and a dynamic pattern, and the conventional general pattern recognition algorithm has a limitation that it can not correctly recognize a dynamic pattern.

However, the dynamic pattern can be thought of as a sequence of static patterns added with a variable of time. Accordingly, the present invention uses a pattern recognition algorithm that considers time-varying dynamic patterns as a dynamic gesture in which a continuous change pattern of a hand or a finger position becomes one operation. Various methods have been proposed for the pattern recognition algorithm considering time-varying, but the most widely used method is Hidden Markov Models (HMM) and Dynamic Time Warping (DTW). The recognition classifier 160 of the present invention can also recognize the gesture using various pattern recognition algorithms, but it is assumed here that DTW is used as an example. DTW has the advantages of real-time implementation and high accuracy.

FIG. 3 is a diagram showing a difference between DTW matching in consideration of time matching and general matching in consideration of time variability.

In FIG. 3, (a) represents Euclidean matching that does not take time-variance into account, and (b) represents DTW matching that takes time-variance into account. The distance or difference between the two matched signals as shown in Figure 3 (a) can be measured by Euclidean distance between signal sequences in the time axis, but in this case local compression or expansion of the signal is considered I can not. On the other hand, in the case of using the DTW matching shown in (b), the distance measurement between the two signals measures the difference through a calibration path having a minimum distance between sequences of different lengths, so that the change due to the compression and expansion of the signal is reflected .

DTW is an algorithm that computes the optimal matching of two patterns by distorting the time length of two sequential data and calculates the distance between two data in the matching. DTW can be implemented in software through dynamic programming. Two time series feature vectors of length m and n are A = a ₁ , a ₂ , ... a _m , B = b ₁ , b ₂ , ... b _n , the distance between two points a _i and b _i (d (a _i , b _i )) can be calculated as the Euclidean distance. However, in the DTW, the warping path (W) defines the mapping between A and B, which must satisfy three conditions: boundary condition, continuity, and monotonicity. A path that minimizes the warping cost among at least one calibration path W satisfying the condition is searched.

A calibration path that minimizes the calibration cost in the DTW can be obtained by Equation (10).

(Where W = w ₁ , w ₂ , ... w _k , and max (m, n) ≤ K ≤ m + n-1.

a k-th correction path (w _k) when said mapping of the (i, j), cumulative distances (D (i, j)) to use the Euclidean distance (d (i, j)) (i, j) is Is defined as Equation (11).

The cumulative distance D (i, j) according to Equation (11) can be easily implemented using dynamic programming as described above.

FIG. 4 shows an embodiment of recognizing a user gesture pattern using a DTW matching technique considering a time-variancy of a recognition classification part.

4A is a diagram showing a mapping of a correction path wk between (i, j) and two time series feature vectors (Time Series A, B). As described above, the DTW algorithm calculates the similarity between the reference feature pattern and the pattern of the user's hand gesture feature acquired in real time using dynamic programming.

 The similarity can be obtained in the form of a cost matrix as shown in (b), and a length of a feature vector of a reference gesture set in advance for discriminating a user gesture is M, a gesture test feature vector Is N, the size of the cost matrix is M N. By this method, it is possible to compare feature vectors having different feature lengths, so that it is possible to calculate the similarity from a nonlinear correspondence.

The user profile storage unit 170 stores a plurality of gesture templates generated in advance from the user behavior profile for a plurality of users and stores user behavior generated by the recognition classifier 160 among the plurality of stored gesture templates The gesture template corresponding to the profile is searched and transmitted to the recognition classifier 160.

When the gesture template corresponding to the user's behavior profile is retrieved from the user profile storage unit 170, the recognition classifying unit 160 recognizes the gesture according to the retrieved gesture template. In other words, user adaptive gesture recognition is performed to enhance the recognition performance of the gesture by reflecting the user's behavior profile in the gesture template.

The conventional recognition classifying unit 160 includes only one reference gesture feature vector for each of a plurality of gestures to be recognized and compares the user's gesture feature vector with a reference gesture feature vector with a reference gesture feature vector to recognize a user gesture Respectively. However, in the present invention, the recognition classifier 160 compares the gesture template obtained by analyzing the user's behavior profile with the gesture feature vector of the user, and determines the most similar gesture template to be obtained. Thus, the user adaptive gesture Recognition is performed to improve gesture recognition performance.

As described above, the user adaptive gesture recognition for reflecting the user's behavior profile in the gesture template is performed because the patterns of the gestures are different from user to user. Even if a plurality of users take the same gesture, the patterns of the gestures do not coincide with each other. For example, even though a large number of users take the same gesture, some people perform gestures quickly, while others perform slowly. Also, the movement of the hand or finger performing the gesture is different for each user. Therefore, the recognition classifier 160 analyzes the user-specific gesture difference, that is, the behavior pattern for each user, and performs the user-adaptive gesture recognition in which the analyzed behavior pattern is applied to the corresponding gesture template among the plurality of preset gesture templates , The gesture recognition rate can be greatly increased.

However, if the gesture template corresponding to the user behavior profile generated by the recognition classifier 160 is not retrieved, the user profile storage unit 170 may store the generated user profile as a new gesture template.

At this time, the recognition classifier 160 does not store the user behavior profile as it is as a gesture template, but analyzes it using an estimation technique such as a Kalman filter or a particle filter to generate a gesture template .

In particular, the Kalman filter is a recursive filter for tracking the state of a linear dynamic system including noise, and is highly efficient when applied to the present invention. The Kalman filter operates on a discrete-time linear dynamic system, assuming a Markov chain in which the state vector at each time is only related to the vectors of the previous time. If the state vector at a certain time k is defined as x _k, and defined as the user input u _k at that time, the Kalman filter can be assumed to be the same relationship as equation (12).

(Here, F _k is the state transition matrix based on a previous state at that time, B _k is the state transition matrix by a user input, and w _k is a covariance matrix of a multivariate normal distribution having a Q _k (w _k ~ N (0, Q _k )).

Then, the state vector X _k and the actually obtained vector Z _k when the vector is measured have the relationship as shown in Equation (13).

(Where H _k is a matrix related to the measurement at that time, and v _k is a noise parameter inferred from a multivariate normal distribution (v _k ~ N (0, R _k )) with a covariance matrix R _k .

The Kalman filter operates recursively. That is, the Kalman filter estimates the current value through the estimated value at the immediately preceding time, and also does not use the measured value or the estimated value immediately outside the immediately preceding time. Each estimation calculation consists of two steps and calculates the expected state when the user input is applied to the state estimated at the previous time. This step is referred to as a prediction step, and there is an update step for calculating an accurate state based on the predicted state and the actually measured state, which have been calculated previously. The computation of the prediction step is a priori, and the a-priori state prediction and the a-priori covariance prediction follow equations (14) and (15), respectively.

는 k 시점의 측정값을 기초로한 k시점의 상태 추정값을 나타낸다.)(here,

Represents the state estimation value at time k based on the measured value at time k.)

는 k 시점의 측정값을 기초로한 k 시점의 상태 공분산 행렬을 나타낸다.)(here,

Represents the state covariance matrix at time k based on the measured value at time k.)

)는 수학식 16으로 계산된다.In the correction step, the previously obtained value is corrected inductively by using the error between the predicted value obtained in the prediction step and the actual measured value. The error between the predicted value obtained in the prediction step and the actual measurement value

) Is calculated by the following equation (16).

And the optimal Kalman gain K _k can be obtained using Equation (17).

Using the equations (18) and (19), the Kalman filter can be implemented by the inductive state correction formula and the inductive covariance correction formula.

FIG. 5 shows an example of a comparison between a user adaptive template analyzing a user profile and a template not considering characteristics of a user.

FIG. 5A is a graph showing the analysis of the user behavior profile, FIG. 5B is a graph showing the relationship between the gesture template generated by analyzing the user behavior profile of FIG. 5A and the existing gesture template generally used, FIG.

In (b), the red line is the gesture template generated by analyzing the user behavior profile, and the blue line indicates the existing gesture template. As shown in FIG. 4 (b), the gesture template corresponding to the user behavior profile appears in a different form from the existing gesture template, and if the gesture is recognized using the gesture template corresponding to the user behavior profile, To recognize the gesture.

Referring again to FIG. 1, the command conversion unit 180 converts the gesture recognized by the recognition and classification unit 160 into a corresponding command, and outputs the command to a device connected to the outside. The gesture recognition device is basically a device for recognizing a command input by a user as a gesture. Therefore, it is configured not to be used as an independent device but to transmit a user command corresponding to a recognized gesture as a device which receives a user command and performs an operation corresponding to the user command. The command conversion unit 180 serves as an interface unit that converts the recognized user's gesture into a corresponding user command and transmits the corresponding user command. In some cases, a gesture recognition device is not separately provided, but may be included in an apparatus that performs an operation corresponding to a user command.

The feedback control unit 140 may be configured to determine whether the hand tracking unit 120 can detect the user's hand or finger in the image or fail the gesture recognition such as when the gesture determination unit 130 can not determine the gesture, If so, provide feedback to the user. The feedback control unit 140 of the present invention is configured such that the visual image, the auditory sense, the tactile sense, and the tactile sense recognition apparatus are constructed so that the conventional image-based gesture recognition apparatuses provide only visual feedback of the dimension, So that the user can immediately judge the gesture recognition result.

This multi-sensory feedback allows the user to instantly determine the gesture recognition result, which also affects the gesture behavior of the user.

FIG. 6 is a graph showing a result of experiment of changing the recognition rate of the gesture according to the feedback method.

The experiment shown in Fig. 6 shows a change in the gesture recognition rate for each of cases in which no feedback is provided, when visual feedback is provided, when auditory feedback is provided, and when tactile feedback is provided, , It can be seen that when the feedback is provided in various ways, a change occurs in a user's operation characteristic and a meaningful change occurs in an actual gesture recognition rate. Especially, it can be confirmed that the influence of each different feedback differs depending on the gesture operation. Accordingly, in order to improve the gesture recognition rate, the present invention provides a multi-sensory feedback of visual, auditory, and tactile signals, thereby greatly improving the gesture recognition rate.

The feedback control unit 140 may be configured to provide multi-sensory feedback directly to the user, but it is common to transmit a feedback command to an external device so as to provide multi-sensory feedback through a generally connected external device.

7 illustrates a smart three-dimensional gesture recognition method according to an embodiment of the present invention.

Referring to FIG. 1, the smart three-dimensional gesture recognition method of FIG. 7 will be described. First, the image acquisition unit 110 acquires an image composed of a plurality of frames (S11). The hand tracking unit 120 receives the image acquired by the image acquiring unit 110 and analyzes whether the user's hand or finger is sensed by using the color-based method and the depth-based method in combination with the transmitted image ). If it is determined that the user's hand or finger is not detected in the image, the multi-sensory feedback is provided to the user through the feedback control unit 140 to recognize that the user has failed to recognize the gesture (S13).

However, if a hand or a finger is detected, the hand tracking unit 120 tracks the position of the hand or the finger (S14). The gesture judging unit 130 analyzes the position information of the user's hand or finger tracked in each frame to determine whether the gesture is intended by the user (S15). Since the gesture is an action according to the user's body motion, in some cases, the user can perform an action similar to the gesture even though the user does not intend to apply the gesture. The gesture determination unit 130 determines a threshold value in advance and determines whether or not the gesture is intended by the user by comparing the tracked positional change of the hand or the finger with the threshold value.

If it is determined that the user is not the intended gesture, the multi-sensory feedback can be provided to the user through the feedback control unit 140 (S13). However, unlike the case where the user's hand or finger is not detected in the image, the feedback may be configured not to provide feedback as to whether the user is the intended gesture.

On the other hand, if it is determined that the user is the intended gesture, the feature extraction unit 150 extracts a feature to be used for gesture recognition from a change in the position of the hand or finger being tracked (S16). The recognition classifying unit 150 recognizes the gesture pattern using the extracted feature (S17). Here, the recognition classifying unit 150 recognizes a pattern using a pattern recognition algorithm that considers time-variant dynamic patterns.

The recognition classifier 150 generates a recognized pattern as a user behavior profile when the pattern is recognized (S18). Then, a gesture template corresponding to the generated user behavior profile is searched among a plurality of gesture templates stored in the user profile storage unit 170 (S19). Then, it is determined whether a corresponding gesture template exists in the user profile storage unit 170 (S20). If the gesture template does not exist, the gesture template is generated by analyzing the generated user behavior profile, and the generated gesture template is stored in the user profile storage unit 170 (S21). However, if there is a corresponding gesture template, the gesture template is recognized and the gesture is recognized by applying the recognized pattern to the gesture template (S22). Here, recognizing the gesture by pre-setting the gesture template and applying the recognized pattern to the gesture template is intended to increase the recognition rate of the gesture recognition device by reducing the recognition rate deterioration due to the difference of the gesture per user. The instruction conversion unit 180 converts the recognized gesture into a corresponding user command and transmits the corresponding user command to the connected external device. At this time, the feedback control unit 140 notifies that the gesture recognition is successful, and the feedback control unit 140 provides the user with multiple sensory feedback to recognize that the user's gesture has been recognized.

As a result, the gesture recognition apparatus and method according to the present invention can detect and track the user's hands and fingers in combination with the color-based method and the depth-based method, thereby improving the detection and tracking performance and discriminating whether the user's action is intended for the gesture The gesture recognition rate can be greatly increased. In addition, a gesture template is provided to perform gesture recognition in consideration of gesture difference per user, and by providing multiple sensory feedback, not only the gesture recognition rate can be further increased, but also user convenience is improved.

In the above description, the user's hand or finger is detected and tracked as an example for gesture recognition. However, the user may be configured to detect and track another body part of the user. That is, any part of the body part of the user that is easy for a user to generate a gesture and can be detected by the gesture recognition device can be detected for gesture recognition.

The method according to the present invention can be implemented as a computer-readable code on a computer-readable recording medium. A computer-readable recording medium includes all kinds of recording apparatuses in which data that can be read by a computer system is stored. Examples of the recording medium include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like, and a carrier wave (for example, transmission via the Internet). The computer-readable recording medium may also be distributed over a networked computer system so that computer readable code can be stored and executed in a distributed manner.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art.

Accordingly, the true scope of the present invention should be determined by the technical idea of the appended claims.

Claims (14)

An image acquiring unit acquiring an image composed of a plurality of consecutive frames;
A hand tracking unit for detecting and tracking a position of a user's designated hand by simultaneously using a color-based method and a depth-based method in the image;
The motion analyzer analyzes the position information of the hand tracked by the hand tracking unit, calculates a motion gradient for the hand positional change analyzed, and determines whether the calculated motion gradient is equal to or greater than a predetermined upper limit value, A gesture judging unit for judging that the gesture of the user is intended by the user;
A feature extraction unit for extracting a feature of the hand position change in a preset manner;
A user profile storage unit storing a plurality of gesture templates;
Recognizing a pattern of the gesture from the extracted features, generating the recognized pattern as a user behavior profile, searching a gesture template corresponding to the user behavior profile among the plurality of gesture templates stored in the user profile storage unit A recognition classifier recognizing the gesture by applying the recognized pattern to a searched gesture template; And
A command conversion unit for converting the recognized gesture into a corresponding user command; And a gesture recognition device. The apparatus of claim 1, wherein the image obtaining unit
And acquiring the color image and the depth image in combination using an RGB-D camera. The hand tracking system according to claim 1,
A hand is detected using a local binary pattern (LBP) for the color vector of the Lab color space in the color-based method, and a hand detected by the machine learning in the depth-based method is detected using the conditional probability And tracks the detected hand of the user according to the CAMSHIFT algorithm. delete The apparatus of claim 1, wherein the feature extraction unit
Wherein the gesture recognition device extracts the speed of the hand position change with the feature. The apparatus of claim 1, wherein the recognition classifier
And recognizes a pattern of the gesture using a dynamic time warping (DTW). The apparatus of claim 1, wherein the recognition classifier
If the gesture template corresponding to the user behavior profile is not searched, generating a new gesture template by filtering the user behavior profile with a Kalman filter, and storing the generated gesture template in a user profile storage unit, Device. The apparatus according to claim 1, wherein the gesture recognition device
If at least one of the following cases occurs: the user's hand is not detected by the hand detection unit, the gesture determination unit determines that the gesture is not intended by the user, and the recognition classification unit recognizes the gesture A feedback control unit for generating feedback so that the user can recognize at least one of visual, auditory, and tactile senses; The gesture recognition apparatus comprising: A gesture recognition method of a gesture recognition apparatus including an image acquisition unit, a hand tracking unit, a gesture determination unit, a feature extraction unit, a user profile storage unit, a recognition classification unit, and an instruction conversion unit,
Acquiring an image composed of a plurality of consecutive frames;
Detecting and tracking a position of a user's designated hand by simultaneously using a color-based method and a depth-based method in the image;
Analyzing position information of the hand in which the gesture judging unit is traced, and determining whether the user intends the gesture using the analyzed hand position change;
Extracting a feature of the hand position change in a predetermined manner;
Recognizing a pattern of the gesture from the feature extracted by the recognition and classification unit;
Wherein the recognition classifier generates the recognized pattern as a user behavior profile, searches a gesture template corresponding to the user behavior profile among the plurality of gesture templates stored in the user profile storage unit, and stores the recognized pattern as a retrieved gesture Recognizing the gesture by applying it to a template; And
Converting the recognized gesture into a corresponding user command; Lt; / RTI >
The step of determining whether the gesture was intended
Calculating a motion gradient for a positional change of the hand being tracked; And
Determining that the gesture is intended by the user if the calculated motion gradient is greater than or equal to a predetermined upper limit value; The gesture recognition method comprising: 10. The method of claim 9, wherein sensing and tracking the position of the hand
Detecting a hand using a local binary pattern (LBP) for the color vector of the Lab color space in the color-based manner;
Detecting a hand sensed by machine learning in the depth-based manner using a conditional probability; And
Tracking the detected hand of the user according to the CAMSHIFT algorithm; The gesture recognition method comprising: delete 10. The method of claim 9, wherein generating the user behavior profile comprises:
Wherein a pattern of the gesture is recognized using a dynamic time warping (DTW) method. 10. The method of claim 9, wherein recognizing the gesture comprises:
Generating the recognized pattern as the user behavior profile;
Retrieving the gesture template corresponding to the user behavior profile;
Recognizing the gesture by applying the recognized pattern to the searched gesture template;
Filtering the user behavior profile with a Kalman filter to generate a new gesture template if the gesture template corresponding to the user behavior profile is not retrieved; And
Storing the generated gesture template in the user profile storage; The gesture recognition method comprising: 10. The method of claim 9, wherein the gesture recognition method comprises:
Wherein the gesture recognition apparatus further comprises a feedback control section,
When the feedback control unit determines that the hand of the user is not detected in the hand detection unit or that the gesture is not intended by the user in the gesture determination unit and when the recognition classification unit recognizes the gesture Generating feedback so that the user can recognize at least one of visual, auditory, and tactile, if one occurs; The gesture recognition method further comprising:

Images