KR20130081126A - Method for hand-gesture recognition and apparatus thereof - Google Patents

Abstract

PURPOSE: A method for recognizing hand gesture and an apparatus thereof are provided to improve recognition performance of hand gesture and to process hand gesture recognition in real-time, by being strong to direction change. CONSTITUTION: A hand gesture recognition device acquires joint information of a tested person, and obtains direction element information for each direction element from the acquired joint information (S110,S120). The device generates direction histogram for each element of a vector expressing hand pose (S130). The device combines a number of direction histograms having different quantization levels (S140). The device recognizes hand gesture using the generated combined direction histogram (S150). [Reference numerals] (AA) Start; (BB) End; (S110) Acquire joint information; (S120) Draw direction element information; (S130) Generate direction histogram; (S140) Generate combined direction histogram; (S150) Recognize hand gesture using a random decision forest sorter

Description

Hand gesture recognition method and device therefor {METHOD FOR HAND-GESTURE RECOGNITION AND APPARATUS THEREOF}

The present invention relates to image processing, and more particularly, to a method and apparatus for hand gesture recognition.

Vision-based hand gesture recognition is one of essential technologies required in the field of human-computer interaction and is an active research field. Methods used for hand gesture recognition include kinematic model based methods, view based methods, low level feature based methods, and the like. In the kinematic model-based method, hand pose information is used. In the view-based method, a hand is modeled using a sequence of a plurality of views. In the low-level feature-based method, a low-level image feature of a hand region may be used. The kinematic model-based method among the hand gesture recognition methods may provide high recognition performance when a hand pose is accurately estimated. However, the hand gesture recognition methods have a problem in that hand pose estimation is difficult in an environment in which a complicated background exists.

Meanwhile, with the development of depth sensing technology in recent years, real-time depth cameras are actively used. For example, low-cost depth cameras include Kinect sensors released by Microsoft, which can provide RGB color images, depth images, and joint tracking information in real time. The development of this sensing technology makes it possible to obtain accurate hand pose estimation results in real time. In addition, a directional histogram can be used to represent the pose of the hand. The direction histogram is a histogram indicating the direction information obtained through the gradient of the image. Directional histograms have the advantage of being robust to light changes, simple, and capable of quickly extracting direction information and / or features. However, since the direction histogram has a disadvantage in that it is not robust to the direction change, when the direction histogram is used, there may be a problem that a hand gesture having various direction changes, which occurs in a real life environment, may not be properly recognized. Accordingly, in order to provide a hand gesture recognition technology applicable to a real-life environment, a hand pose feature and a gesture classification framework that can provide a simple, fast and high accuracy recognition performance are required.

An object of the present invention is to provide a hand gesture recognition method capable of providing high recognition performance.

Another technical problem of the present invention is to provide a hand gesture recognition apparatus capable of providing high recognition performance.

One embodiment of the present invention is a hand gesture recognition method. The method may further include obtaining joint information related to a hand gesture using a depth camera, deriving direction element information based on spherical coordinates using the obtained joint information, and using the derived direction element information. Generating a plurality of direction histograms, combining the generated plurality of direction histograms, generating a combined direction histogram, and applying a random decision forest classifier to the generated combined direction histogram, to the hand gesture. Deriving a final recognition result for the plurality of directional histograms having different levels of quantization.

According to the hand gesture recognition method according to the present invention, high recognition performance can be provided.

According to the hand gesture recognizing apparatus according to the present invention, high recognition performance can be provided.

1 is a flowchart schematically illustrating a hand gesture recognition method according to an embodiment of the present invention.
2 is a conceptual diagram schematically illustrating an embodiment of a method of generating a coupling direction histogram.
3 is a simulation result schematically showing the recognition accuracy of the random decision forest classifier according to the number of decision trees.
4 is a block diagram schematically illustrating an apparatus for recognizing a hand gesture according to an exemplary embodiment of the present invention.
5 is a table illustrating an embodiment of hand gesture recognition accuracy according to a direction histogram type and the number of decision trees.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the following description of the embodiments of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present disclosure rather unclear.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . In addition, the description of "including" a specific configuration in the present invention does not exclude a configuration other than the configuration, and means that additional configurations can be included in the practice of the present invention or the technical scope of the present invention.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

In addition, the components shown in the embodiments of the present invention are shown independently to represent different characteristic functions, which does not mean that each component is composed of separate hardware or software constituent units. That is, each constituent unit is included in each constituent unit for convenience of explanation, and at least two constituent units of the constituent units may be combined to form one constituent unit, or one constituent unit may be divided into a plurality of constituent units to perform a function. The integrated embodiments and separate embodiments of the components are also included within the scope of the present invention, unless they depart from the essence of the present invention.

In addition, some of the components are not essential components to perform essential functions in the present invention, but may be optional components only to improve performance. The present invention can be implemented only with components essential for realizing the essence of the present invention, except for the components used for the performance improvement, and can be implemented by only including the essential components except the optional components used for performance improvement Are also included in the scope of the present invention.

1 is a flowchart schematically illustrating a hand gesture recognition method according to an embodiment of the present invention.

Referring to FIG. 1, the hand gesture recognition apparatus may acquire joint information of a test subject (S110). The joint information may be obtained using joint tracking information provided by a depth camera. The depth camera may include, for example, a PrimeSense depth camera, a Microsoft Kinect, and the like. The hand gesture recognizing apparatus may use three-dimensional coordinate information of the joints of the shoulder, elbow, and hand related to the hand gesture among the acquired joint information of the whole body.

The hand gesture recognizing apparatus may derive direction element information for each direction element from the obtained joint information (S120). Here, the direction element may be four direction elements based on spherical coordinates.

,

및

이라 가정한다. 손 제스처 인식 장치는 손 포즈를 표현하기 위해, 어깨에 대한 팔꿈치의 방향과 팔꿈치에 대한 손의 방향을 계산할 수 있다. 이를 위해, 손 제스처 인식 장치는 3차원 데카르트 좌표를 구면 좌표로 변환하고, 팔꿈치 및 손의 방향을 계산할 수 있다. 이는 다음 수학식 1에 의해 나타내어질 수 있다.For example, the three-dimensional Cartesian coordinates of the shoulders, elbows, and hands are

,

And

Assume that The hand gesture recognizing apparatus may calculate the direction of the elbow with respect to the shoulder and the direction of the hand with respect to the elbow to express the hand pose. To this end, the hand gesture recognizing apparatus may convert 3D Cartesian coordinates into spherical coordinates and calculate elbow and hand directions. This can be represented by the following equation (1).

[Equation 1]

는 어깨에서 팔꿈치 간의 벡터를 나타내고,

는 팔꿈치에서 손 간의 벡터를 나타낼 수 있다.

과

은 어깨에 대한 팔꿈치의 방향을 나타내는 구면 좌표의 각도를 의미하고,

과

는 팔꿈치에 대한 손의 방향을 나타내는 구면 좌표의 각도를 의미할 수 있다. 따라서, 손 포즈 및/또는 방향 요소 정보는 팔꿈치 및 손의 방향을 나타내는 4개의 방향 요소로 구성된, 벡터

로 나타내어질 수 있다.
here,

Represents the vector from the shoulder to the elbow,

Can represent a vector between the hands at the elbows.

and

Means the angle of the spherical coordinates representing the direction of the elbow to the shoulder,

and

May refer to an angle of spherical coordinates representing the direction of the hand with respect to the elbow. Thus, the hand pose and / or direction element information consists of four direction elements representing the direction of the elbow and hand, the vector

It can be represented as.

Referring back to FIG. 1, the hand gesture recognizing apparatus may generate a direction histogram for each element of the vector representing the hand pose (S130). The direction histogram is a histogram indicating the direction information obtained through the gradient of the image.

과

에 대한 방향 히스토그램의 빈(bin)은

범위의 구간이 균등하게 분할되어 결정될 수 있고,

과

에 대한 방향 히스토그램의 빈은

범위의 구간이 균등하게 분할되어 결정될 수 있다. 이 때, 손 제스처 인식 장치는, 서로 다른 양자화 레벨을 갖는 복수의 방향 히스토그램을 생성할 수 있다. 즉, 손 제스처 인식 장치는, 다양한 각도 간격에 기반하여, 방향 히스토그램의 빈(bin) 개수를 결정할 수 있다.

and

The bin of the direction histogram for

The intervals of the range can be determined evenly divided,

and

The bin of the direction histogram for

The intervals of the range can be determined evenly divided. In this case, the hand gesture recognizing apparatus may generate a plurality of directional histograms having different quantization levels. That is, the hand gesture recognizing apparatus may determine the number of bins of the directional histogram based on various angular intervals.

The hand gesture recognizing apparatus may combine the plurality of direction histograms having different quantization levels to generate the combined direction histogram in order to increase the robustness against the direction change (S140). That is, the hand gesture recognition apparatus may combine a plurality of direction histograms having different numbers of bins. In this case, the hand pose may be represented by the coupling direction histogram, and the coupling direction histogram may be represented by the following equation (2).

&Quot; (2) "

는 히스토그램 빈에 대한 각도 간격을 나타낼 수 있다. 또한,

는 4개의 방향 요소에 대한 방향 히스토그램으로 구성된 벡터이고,

를 구성하는 각각의 방향 히스토그램은,

각도 간격의 히스토그램 빈을 가질 수 있다. n은 결합 방향 히스토그램 생성에 사용되는 방향 히스토그램의 개수를 나타낼 수 있다.here,

May represent an angular interval with respect to the histogram bin. Also,

Is a vector of directional histograms for four directional elements,

Each direction histogram constituting the

It can have histogram bins at angular intervals. n may indicate the number of directional histograms used for generating the combined direction histogram.

2 is a conceptual diagram schematically illustrating an embodiment of a method of generating a coupling direction histogram. In FIG. 2, the observation of the video sequence may be represented by x, where x represents a vector of frame observations within a window of size m. In the embodiment of FIG. 2, n direction histograms h ₁ to h _n may be combined to generate a coupling direction histogram h.

Referring back to FIG. 1, the hand gesture recognizing apparatus may recognize the hand gesture by using the coupling direction histogram (S150). For example, the hand gesture recognition apparatus may obtain a final recognition result for the hand gesture by applying a random decision forest classifier to the coupling direction histogram. That is, the combined direction histogram extracted from the test image may be input to the random decision forest classifier, and the hand gesture represented by the combined direction histogram may be recognized using the random decision forest classifier.

The random decision forest classifier is an ensemble classifier using a random subspace method and may be composed of a plurality of decision trees. The random decision forest classifier can effectively provide a high performance rate and a relatively high recognition rate.

The hand gesture recognizing apparatus may select a subset by a random replacement method from the combination direction histogram set extracted from the test image. The hand gesture recognizing apparatus may perform learning on a random decision forest classifier composed of T (T is a positive integer) decision trees using a subset selected from the set of combination direction histograms. The learned classifier can obtain the final recognition result for the hand gesture by calculating the average of the probabilities for all decision trees. This can be represented by the following equation (3).

&Quot; (3) "

는 손 제스처에 대한 최종 인식 결과를 나타낼 수 있다.
Here, P _t (c | H) may represent the probability for the gesture class c in the decision tree t,

May represent a final recognition result for the hand gesture.

3 is a simulation result schematically showing the recognition accuracy of the random decision forest classifier according to the number of decision trees. 3 shows the recognition accuracy of the hand gesture data set, and the hand gesture data set is composed of 30 classes.

Referring to FIG. 3, a random decision forest classifier that uses more than 85 decision trees may provide 100% accuracy.

4 is a block diagram schematically illustrating an apparatus for recognizing a hand gesture according to an exemplary embodiment of the present invention. The hand gesture recognition apparatus 400 may include a joint information acquirer 410, a direction element information derivator 420, a combined direction histogram generator 430, and a recognizer 440.

Referring to FIG. 4, the joint information acquisition unit 410 may acquire joint information of a test subject. The hand gesture recognizing apparatus 400 may use three-dimensional information about the joints of the shoulder, elbow, and hand related to the hand gesture among the acquired joint information of the entire body.

The direction element information deriving unit 420 may derive the direction element information for each direction element from the obtained joint information. Here, the direction element may be four direction elements based on spherical coordinates. Since the specific embodiment of the method for deriving the direction element information has been described above, it will be omitted.

The combining direction histogram generator 430 may generate a direction histogram for each direction element. In this case, the coupling direction histogram generator 430 may generate a plurality of directional histograms having different quantization levels, and generate the combined direction histogram by combining the plurality of directional histograms.

The recognition unit 440 may recognize a hand gesture using a coupling direction histogram. For example, the recognizer 440 may obtain a final recognition result for the hand gesture by applying the random decision forest classifier to the generated combining direction histogram. Since the specific embodiment of the method for obtaining the final recognition result has been described above, it will be omitted.

,

,

은 각각 10도, 20도, 30도 간격의 빈을 이용하여 생성된 하나의 방향 히스토그램을 나타낸다.5 is a table illustrating an embodiment of hand gesture recognition accuracy according to a direction histogram type and the number of decision trees. In Figure 5,

,

,

Represents one directional histogram generated using bins at 10, 20, and 30 degree intervals, respectively.

Referring to FIG. 5, when the number of decision trees of the random decision forest classifier is one, the highest hand gesture recognition accuracy may be provided for one direction histogram generated by using bins having a 10 degree interval. However, when the number of decision trees of the random decision forest classifier is two or more, the highest hand gesture recognition accuracy may be provided for the combined direction histogram. This means that the combined direction histogram can increase the hand gesture recognition accuracy.

The above-described methods, devices, features and / or recognition frameworks are not limited to hand gesture recognition, but may also be applied to full-body gesture recognition, human behavior analysis and / or human behavior recognition.

The method and apparatus for recognizing a hand gesture according to the present invention may be robust to a change in direction to improve the recognition performance of the hand gesture and enable real time processing of the hand gesture recognition. That is, the hand gesture recognition method and apparatus according to the present invention can efficiently recognize hand gestures having various direction changes generated in real life, and by using a low-dimensional image characteristic and a fast classifier, Real time processing can be enabled.

In addition, the present invention is an essential technology for human-computer interaction applications, and may be usefully used in various applications requiring control between a user and a TV, such as a gesture-based TV and a video game console.

In the above-described embodiments, the methods are described on the basis of a flowchart as a series of steps or blocks, but the present invention is not limited to the order of the steps, and some steps may occur in different orders or simultaneously . It will also be understood by those skilled in the art that the steps depicted in the flowchart illustrations are not exclusive, that other steps may be included, or that one or more steps in the flowchart may be deleted without affecting the scope of the present invention. You will understand.

The above-described embodiments include examples of various aspects. While it is not possible to describe every possible combination for expressing various aspects, one of ordinary skill in the art will recognize that other combinations are possible. Accordingly, it is intended that the invention include all alternatives, modifications and variations that fall within the scope of the following claims.

Claims (1)

Acquiring joint information related to a hand gesture using a depth camera;
Deriving direction element information based on spherical coordinates using the obtained joint information;
Generating a plurality of direction histograms using the derived direction element information;
Combining the generated plurality of direction histograms to generate a combining direction histogram; And
Deriving a final recognition result for the hand gesture by applying a random decision forest classifier to the generated combined direction histogram,
And the plurality of directional histograms have different quantization levels.

Images