KR101543150B1 - Apparatus for gesture recognition and method thereof - Google Patents

Abstract

The present invention relates to a motion recognition apparatus resistant to distance and rotation, which is capable of recognizing the motion of a user, and to a motion recognition method thereof. According to an embodiment of the present invention, the motion recognition method comprises: a data acquisition step of acquiring joint information and depth information of a user; a motion template confirmation step of confirming whether a motion template exists in a data storage unit; a matching step of generating an input motion template from the joint information and the depth information acquired in the data acquisition step and matches the input motion template with a plurality of motion templates stored in the data storage unit; and a motion estimation step of estimating a motion of the user by outputting a motion having the most similar motion template with the input motion template.

Description

[0001] Apparatus for gesture recognition and method [0002]

The present invention relates to a behavior recognition apparatus and a behavior recognition method thereof.

There are two methods of recognizing human behavior in the image-based method: a method using a classifier based on a machine learning algorithm and a method using a behavior image-based matching algorithm. In the former case, it has the advantage of recognizing the action robustly even in a disturbed environment, but it has a drawback that it is possible to recognize only the predefined action because the training process for recognizing the action is long. Therefore, there is a need for a method that uses an image-based matching algorithm but recognizes the behavior of a person with a simple behavior registration.

Korean Patent Publication No. 10-2013-0117553 (Oct. 28, 2013)

The present invention provides a behavior recognition apparatus and a behavior recognition method capable of recognizing user's behavior robust to distance and rotation.

A behavior recognition method of a behavior recognition apparatus according to the present invention includes: a data acquisition step of acquiring joint information and depth information of a user; A behavior template checking step of checking whether a behavior template exists in the data storage unit; A matching step of creating an input action template from the joint information and depth information of the user acquired in the data acquiring step and matching the input action template with a plurality of behavior templates stored in the data storage unit; And a behavior estimation step of outputting a behavior having a behavior template most similar to the input behavior template and estimating a behavior of the user.

In the data acquiring step, the joint information and the depth information of the user can be acquired using the depth camera.

In the behavior template checking step, the behavior template may be formed as an image by connecting the positions of the respective joints with straight lines around the reference point of the user's body.

If the behavior template does not exist in the behavior template confirmation step, the behavior recognition apparatus can switch to the registration mode and register the behavior template.

In the matching step, an input behavior template is matched with a plurality of stored behavior templates to calculate respective similar scores, and a behavior template having a maximum score among the plurality of behavior templates can be found.

In the matching step, the similarity of the plurality of behavior templates is compared to calculate a minimum acceptable score, so that the score of the behavior template having the maximum score is compared with the minimum acceptable score.

In the behavior estimation step, if the score of the behavior template having the maximum score is larger than the minimum allowable score, the action represented by the behavior template having the maximum score may be output to estimate the behavior of the user.

And after the matching step, filtering the estimated result using the sliding window filter to prevent the user's behavior estimation from being misjudged.

The sliding window filter can output the most estimated result within a certain range based on the result estimated in the past including the estimated result within a predetermined time.

In addition, the behavior recognition apparatus according to the present invention includes: a data acquisition unit for acquiring joint information and depth information of a user; A data storage unit for storing a plurality of behavior templates according to the behavior of the user; A behavior template generation unit that generates an input behavior template from the joint information and depth information of the user acquired by the data acquisition unit; And an estimator for matching the input behavior template with a plurality of behavior templates stored in the data storage unit and outputting a behavior having a behavior template most similar to the input behavior template to estimate a behavior of the user .

The action template may be formed as an image by connecting the positions of the respective joints with a straight line around the reference point of the user's body.

Wherein the estimating unit calculates a similarity score by matching an input action template with a plurality of stored behavior templates, finds a behavior template having a maximum score among the plurality of behavior templates, and compares the similarities of the plurality of behavior templates with each other, Compares the score of the behavioral template having the maximum score with the minimum acceptable score, and if the score of the behavioral template with the maximum score is greater than the minimum acceptable score, The behavior can be output to estimate the user's behavior.

The behavior recognition apparatus and the behavior recognition method according to an embodiment of the present invention can increase the behavior recognition rate by estimating the behavior of the user by using the joint information and the depth information of the arms and legs having relatively many movements.

In addition, the behavior recognition apparatus and the behavior recognition method according to an embodiment of the present invention generate a behavior template using the joint information and the depth information of the user, so that the behavior of the user can be recognized robustly with distance and rotation.

1 is a block diagram illustrating a behavior recognition apparatus according to an embodiment of the present invention.
2 is a diagram showing a behavior template generated by the behavior template generating unit.
3 is a flowchart illustrating a behavior recognition method of a behavior recognition apparatus according to an embodiment of the present invention.
Fig. 4 is a flowchart specifically showing the steps of Fig. 3. Fig.
5A to 5D are diagrams showing experimental results of the behavior recognition method of the behavior recognition apparatus according to the present invention.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, so that those skilled in the art can easily carry out the present invention.

1 is a block diagram illustrating a behavior recognition apparatus according to an embodiment of the present invention. 2 is a diagram showing a behavior template generated by the behavior template generating unit.

1, a behavior recognition apparatus 100 according to an exemplary embodiment of the present invention includes a data acquisition unit 110, a data storage unit 120, a behavior template generation unit 130, a behavior behavior estimation unit 140, . The behavior recognition apparatus 100 is a device that receives user's joint information and depth information and recognizes a user's behavior.

The data acquisition unit 110 acquires joint information and depth information of the user. The data acquisition unit 110 may acquire joint information and depth information of the user using the joint tracking information provided by the depth camera. This depth camera can acquire RCB color image, depth image and joint tracking information in real time. As described above, the behavior recognition apparatus 100 according to the present invention can increase the behavior recognition rate by weighting the joint information of the arms and legs, which are relatively moved in the human body.

The data storage unit 120 stores the information acquired by the data acquisition unit 110. Specifically, the data storage unit 120 stores the joint information and the depth information of the user acquired by the data acquisition unit 110. In addition, the data storage unit 120 stores a gesture template. The behavior template is basic information for recognizing the behavior of the user. Various behavior templates are stored in the data storage unit 120 to recognize various behaviors of the user. In the data storage unit 120, a plurality of action templates generated according to a predetermined period may be grouped into a set and defined as a set of behavior templates.

The behavior template generating unit 130 receives the joint information and the depth information of the user, and generates the behavior template. In addition, the behavior template generating unit 130 may generate an action template to be stored in the data storage unit 120, or may generate an input action template by receiving information of a currently input user. Here, the action template is to make the joint information of the user at a constant time (t) into an image having a constant size. Specifically, the behavior template connects the position of each joint with a straight line having a specific scalar value based on the moment point of the user, and creates an image of the motion template. However, according to the position and the viewpoint of the depth camera of the human joint information, the coordinate values projected to the 2D image are different. Accordingly, the behavior template generating unit 130 generates a behavior template robust against translation and rotation using depth information. That is, the behavior template generating unit 130 performs a transformation operation to generate a robust behavior template and a rotation operation to generate a rotation-resistant behavior template. First, the behavior template generating unit 130 performs a transformation operation to obtain a scale of a human body ratio according to a distance, using the joint information and the depth information of the inputted user. In order to perform the rotation operation, the behavior template generation unit 130 calculates the normal vector using the center point and the depth information of both shoulder points in the joint information of the user. Here, when the normal vector is calculated, the scale of the depth information is much larger than the position value of the joint. Therefore, the normalization operation is performed through the conversion operation described above and then externally calculated. Then, the behavior template generating unit 130 calculates and corrects the rotation difference between the normal vector of the template registered as the action and the normal vector calculated according to the input information. Here, the rotation calculation of the joint position is performed after the conversion operation. Since the position of the joint is expressed based on the center point, the position in the XY plane is fixed. Therefore, the behavior template generating unit 130 calculates and corrects only the rotation in the ZY plane. The formula of such a rotation operation is expressed by the following equation (1).

[Equation 1]

Here,? Can be obtained by a normal vector (Nref) of a template registered as an action and a normal vector (Nin) calculated according to the input information. Accordingly, the behavior template generation unit 130 may correct the position (x ') of the joint by rotating only the x-axis coordinate (x) projected on the 2D image. That is, the behavior template generated by the above operation is expressed as shown in FIG.

The behavior estimator 140 compares the input behavior template generated by the behavior template generator 130 with the behavior template stored in the data storage unit 120, and estimates the behavior of the user. That is, the behavior estimator 140 estimates the behavior of the user by confirming whether the input behavior template currently input is similar to the behavior template stored in the data storage unit 120.

Specifically, the behavior estimator 140 calculates a similarity score by matching the input behavior template with the behavior template stored in the data storage unit 120 using a template matching algorithm. In addition, the behavior estimator 140 may reduce the probability of false estimation by additionally using a negative template rejection algorithm because the likelihood of incorrect estimation becomes high as the similarity between actions becomes high. Here, the negative template rejection algorithm compares the similarities of the defined behaviors existing in the behavior template set, calculates a minimum acceptance score for each action, and uses the minimum acceptance score as a reference value when calculating the similar score. That is, the behavior estimating unit 140 does not return the estimated value if the input behavior template is compared with the stored behavior template, and when the behavior having the greatest similarity is estimated, if the minimum acceptable score is not exceeded. In addition, the behavior estimator 140 applies a sliding window filter to prevent the estimation from being erroneous or unexpected while taking continuous action. Here, the sliding window filter outputs the most estimated results within a certain range (window size) based on the results estimated in the past including the estimated results within the predetermined time (t).

As described above, the behavior recognition apparatus 100 according to the present invention estimates the behavior of the user by using the joint information and the depth information of the arms and legs, which have relatively many movements, so that the behavior recognition rate can be increased.

Also, the behavior recognition apparatus 100 according to the present invention generates a behavior template using the joint information and the depth information of the user, so that the behavior recognition apparatus 100 can recognize the behavior of the user robustly to the distance and the rotation.

3 is a flowchart illustrating a behavior recognition method of a behavior recognition apparatus according to an embodiment of the present invention. Fig. 4 is a flowchart specifically showing the steps of Fig. 3. Fig.

Referring to FIG. 3, a behavior recognition method of a behavior recognition apparatus according to an embodiment of the present invention includes a data acquisition step S10, a behavior template confirmation step S20, a matching step S30, a filtering step S40, And estimating step S50. Hereinafter, the respective steps of FIG. 3 will be described with reference to FIGS. 1 and 4. FIG.

In the data acquiring step (S10), the data acquiring unit 110 acquires the joint information and depth information of the user. The data acquisition unit 110 acquires the joint information and the depth information of the user using the joint tracking information provided by the depth camera.

In the behavior template checking step (S20), the information obtained in the data obtaining step (S10) is received as an input, and the existence of a behavior template is confirmed.

First, in the behavior template recognition step S20, if the behavior template is not stored in the data storage unit 120, the behavior recognition apparatus 100 switches to the behavior template registration mode and registers a behavior template (S21). Also, if the behavior recognition apparatus 100 stores the behavior template in the data storage unit 120, the behavior recognition apparatus 100 switches to the behavior recognition mode and starts to recognize the behavior of the user.

The matching step S30 is a step of receiving the joint information and the depth information of the user acquired in the data acquiring step (S10) and matching with the behavior template stored in the data storage unit 120. [ First, in the matching step S30, the behavior template generating unit 130 receives the joint information and depth information of the user acquired in the data acquiring step (S10), and generates an input action template. The behavior template generating unit 130 generates an input action template robust against translation and rotation using depth information. That is, the behavior template generating unit 130 performs a transformation operation to generate a robust behavior template and a rotation operation to generate a rotation-resistant behavior template. Since the behavior template generating unit 130 generates a behavior template by a conversion operation and a rotation operation are the same as those described above, a detailed description will be omitted.

Next, the behavior estimation unit 140 calculates a similar score by matching the input action template and the behavior template stored in the data storage unit using the template matching algorithm (S31). In addition, the behavior estimator 140 may reduce the probability of false estimation by additionally using a negative template rejection algorithm because the likelihood of incorrect estimation becomes high as the similarity between actions becomes high. Here, the negative template rejection algorithm compares the similarities of the defined behaviors existing in the set of behavior templates, calculates a minimum acceptable score for each of the behaviors, and uses it as a reference value when calculating the similar score. Next, the behavior estimating unit 140 determines whether the similar scores of all the stored behavior templates and the input behavior templates have been calculated (S32). If it is determined that the similarity score between the stored behavior template and the input action template has been calculated, the behavior estimating unit 140 estimates a behavior having a maximum score among the calculated behavior templates (S33). However, when estimating the behavior with the greatest similarity, it does not return the estimate if the score of the behavior does not exceed the minimum acceptable score.

In the filtering step S40, the behavior estimator 140 applies a sliding window filter to prevent the user's behavior estimation from being erroneous or unexpected. Here, the sliding window filter outputs the most estimated results within a certain range (window size) based on the results estimated in the past including the estimated results within the predetermined time (t).

In the behavior estimation step S50, the behavior estimation unit 140 estimates the behavior of the user according to the result output through the filtering step S40.

In order to confirm the accuracy of the behavior recognition method of the above-described behavior recognition apparatus, the following experiment was performed. 5A to 5D are diagrams showing experimental results of the behavior recognition method of the behavior recognition apparatus according to the present invention.

First, the goal of the experiment was to register random 3 kinds of behaviors to confirm the performance with recognition rate of 85% or more and false recognition rate of 10% or less and confirm the results by taking 100 random actions. In addition, the three behaviors are defined as shown in Table 1 below.

behavior action "Hello" Shake the right hand "SOS" Shake your arms above your head "None" No action taken

Also, for the accurate performance evaluation, the person who registered the behavior and the person who carried out the experiment were different, and each action template was created with a total of 8 behavior template images once every second.

5A to 5C, the behavior recognition apparatus 100 correctly recognizes the behavior defined by "Hello" (FIG. 5A), "SOS" (FIG. 5B), and "None" . In addition, as shown in FIG. 5D, the behavior recognition apparatus 100 can confirm that the behavior is recognized robustly with respect to transformation and rotation.

In addition, the recognition rate of each behavior of the behavior recognition apparatus is shown in Table 2 below.

[Table 2]

As shown in Table 2, it can be seen that the recognition rate is slightly higher only for the " Hello " operation, and the remaining operations are recognized almost accurately. That is, the recognition rate for all actions is about 86.67% and the false recognition rate is 4.44%. Therefore, it can be seen that the experimental target of the behavior recognition apparatus according to the present invention is satisfied.

It is to be understood that the present invention is not limited to the above-described embodiment, and that various modifications and changes may be made without departing from the scope of the present invention as defined in the appended claims. It will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention.

100: a behavior recognition device 110: a data acquisition unit
120: Data storage unit 130: Action template generation unit
140: Behavior Assessment Government

Claims (12)

A data acquiring step of acquiring joint information and depth information of a user;
A behavior template checking step of checking whether a behavior template exists in the data storage unit;
A matching step of creating an input action template from the joint information and depth information of the user acquired in the data acquiring step and matching the input action template with a plurality of behavior templates stored in the data storage unit;
A filtering step of filtering an estimated result using a sliding window filter to prevent a user's behavioral estimation from being misjudged after the matching step; And
And a behavior estimation step of outputting a behavior having a behavior template most similar to the input behavior template and estimating a behavior of the user. The method according to claim 1,
And acquiring joint information and depth information of a user by using a depth camera in the data acquiring step. The method according to claim 1,
Wherein the behavior template is formed as an image by linearly connecting the positions of the joints around the reference point of the user's body in the behavior template checking step. The method according to claim 1,
And if the behavior template does not exist in the behavior template recognition step, the behavior recognition apparatus switches to the registration mode and registers the behavior template. The method according to claim 1,
Wherein the matching step matches an input action template with a plurality of stored behavior templates to calculate respective similar scores, and finds a behavior template having a maximum score among the plurality of behavior templates. 6. The method of claim 5,
Wherein the matching step compares the similarity of the plurality of behavior templates to calculate a minimum acceptable score, and compares the score of the behavior template having the maximum score with the minimum acceptable score. The method according to claim 6,
Wherein in the behavior estimation step, if the score of the behavior template having the maximum score is larger than the minimum allowable score, the behavior estimation unit estimates the behavior of the user by outputting the behavior represented by the behavior template having the maximum score Recognition method. delete The method according to claim 1,
Wherein the sliding window filter outputs the most estimated result within a certain range based on a result estimated in the past including a result estimated within a predetermined time. A data acquisition unit for acquiring joint information and depth information of a user;
A data storage unit for storing a plurality of behavior templates according to the behavior of the user;
A behavior template generation unit that generates an input behavior template from the joint information and depth information of the user acquired by the data acquisition unit; And
And an estimation unit for matching the input behavior template with a plurality of behavior templates stored in the data storage unit and outputting a behavior having a behavior template most similar to the input behavior template,
Wherein the estimating unit filters the estimated result using a sliding window filter to prevent a user's behavior estimation from being misjudged. 11. The method of claim 10,
Wherein the behavior template is formed as an image by connecting the positions of the respective joints with a straight line around the reference point of the user's body. 11. The method of claim 10,
Wherein the estimating unit matches an input behavior template with a plurality of stored behavior templates to calculate respective similar scores, finds a behavior template having a maximum score among the plurality of behavior templates,
Compares the similarity of the plurality of behavioral templates to compute a minimum acceptable score to compare whether the score of the behavioral template with the maximum score is greater than a minimum acceptable score,
And when the score of the behavior template having the maximum score is larger than the minimum allowable score, the behavior estimation means estimates the behavior of the user by outputting the behavior represented by the behavior template having the maximum score.

Images