KR100817298B1 - Method for detecting and tracking both hands - Google Patents

Abstract

The present invention relates to a method for detecting and tracking both hands, comprising the steps of modeling both arms including the shoulders, elbows, and hands from an image through a human body structure, detecting both hands from modeling, Comparison of the images, the amount of inclination and inclination of the two straight lines between the elbow and the hand, and the limiting radius of the human arm (the arm does not bend outward, the length of the arm does not extend over a certain length, Can be used to track the position of both hands. In addition, KLT (Kanade-Lucas-Tomasi) feature information is regressed and compressed and expressed as a slope value to extract and track the movement of overlapping objects in real time without increasing the computational complexity due to the extraction of a large amount of feature information. Extraction and tracking techniques can be applied to the real world.

Two-Handed Overlap, KLT Feature Points

Description

METHOD FOR DETECTING AND TRACKING BOTH HANDS}

1 is a flowchart illustrating a process of detecting and tracking movements of both hands in an image acquired by a camera according to an embodiment of the present invention;

2 is an embodiment of the present invention, a conceptual diagram of the modeling of the body and arm for both hand extraction and tracking,

3 is a model modeled for the arm of FIG.

4 is a view showing a video feature used in the present invention and a straight line representative of the feature.

<Explanation of symbols for the main parts of the drawings>

HEAD: Head LSP: Left Shoulder Point

LFA: Left Forearm LEP: Left Elbow Point

LA: left arm LH: left hand

RSP: Right Shoulder Point RFA: Right Forearm

REP: right elbow point RA: right arm

RH: Right hand

The present invention relates to a moving object extraction and tracking technology, and more particularly, to a two-hand detection and tracking method for stably extracting and tracking both hands of a moving person in a camera image despite the overlapping situation. will be.

Hand motion detection and tracking has been used in a variety of industries that use motion analysis, including computer vision applications and robotics, video surveillance technology, and intelligent transportation systems. Recently, as the necessity of detecting and tracking a moving object increases in the field of computer vision and the interaction between humans and computers, humans and robots, research on its application technology is being actively conducted.

As such, hand tracking and detection algorithms have been studied in various fields such as gesture recognition and motion analysis. Most of the hand detection and tracking techniques studied to date have been limited to one hand. This is because when two hands are simultaneously tracked, there is no distinct feature information for distinguishing both hands. Techniques designed for two-handed detection have recently been reported, which use a distance transform function to classify the types of overlapping hands or use shadowing information to the hands and elbows to distinguish them. do. In addition, a method of detecting an object by using unique feature information of the overlapping object, that is, color and shape, may be used to classify the type when the two hands overlap.

However, when unique information such as color and shape overlaps with very similar objects such as hands, it is difficult to separate and track them, and the location of overlapping hands can be known only when the overlapping form of both hands belongs to the classified type. There was this. On the other hand, when using the distance conversion function, there was a limitation that both hands should be seen from the image to the hand and the elbow. In addition, even when solving the problem of overlapping a general moving object, there is a distinctive feature information, but this feature information has a lot of limitations that takes a long time in calculation. That is, as the feature information increases, the complexity of calculation increases, which makes it difficult to track an object in real time.

Accordingly, the present invention has been made to solve the above problems of the prior art, an object of the present invention is to track both hands to reflect the structure of the human body and arm, as well as KLT (Kanade-Lucas-Tomasi) characteristic information It is to provide a two-hand detection and tracking method that detects and tracks the movement of both hands in real time without increasing the computational complexity by reducing the computational complexity by compressing to a slope value by compressing the regression analysis.

The two-handed detection and tracking method of the present invention for achieving the above object comprises the steps of: (a) modeling both arms, including the shoulders, elbows and hands through the body structure of the person from the image; (b) detecting both hands from the modeling; (c) determining a range of motion limits according to the structure of the human arm; And (d) tracking the position of both hands by changing the inclination of the two straight lines connecting the elbow and the hand by comparing the previous image with the previous image.

In this case, in the step (a), the modeling of the arm detects the position of the shoulder point using the position of the face, and detects the hand through the first regression analysis of the KLT (Kanade-Lucas-Tomasi) feature point. The elbow is estimated from the shoulder point position and the hand position. Meanwhile, before the step (a), the positions of the hands and the face are extracted using the skin color information.

In addition, in step (b), both hands match motion information and skin color information and detect the result using blob analysis.
Step (c) is a step of designating a virtual area of both hands, which is determined by a behavior limit radius of the human arm and a second order autoregressive model. The hypothetical region is defined by three limitations: the arms do not bend outward, the length of the arms do not extend beyond a certain length, and the arms can be bent relative to the elbows.

Finally, step (d) may include regression analysis of a previous image of the image to determine whether virtual regions of both hands overlap; And if the virtual regions do not overlap, check current positions of both hands to extract KLT (Kanade-Lucas-Tomasi) feature points, and regression analysis to calculate slopes of both straight lines connecting the elbows and hands, and overlap the virtual regions. In this case, regression analysis of the slope obtained from the previous image predicts the direction to move, and then re-determining whether the two hands overlap, and as a result of judging whether the two hands overlap, if the two hands do not overlap, the virtual region does not overlap. Follow and, if both hands overlap, the step consists of extracting the KLT (Kanade-Lucas-Tomasi) feature points to determine the position of both hands. In this case, the slope calculated when the virtual regions do not overlap is used to predict a moving direction of the next image, and the KLT feature points extracted when the two hands overlap, the straight line of the slope calculated when the virtual regions do not overlap. Perform a regression analysis to use.

Hereinafter, a method for detecting and tracking both hands of the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart illustrating a process of detecting and tracking movements of both hands in an image acquired by a camera as an embodiment of the present invention.

As shown in FIG. 1, first, an image is acquired from a camera (S1).

The obtained image is extracted with only motion information compared with the previous image. Also, feature information of the moving object is obtained from the acquired image. In this embodiment, since the feature information of the hand is extracted, the positions of the hand and the face are extracted using the skin color information (S2).

Now, motion information and skin color information are matched to extract both hands using blob analysis (S3).

After the two hands are extracted, the motion radius of the arm and the second autoregressive analysis model are used to determine the virtual area in which the two hands can be moved. In the present invention, the two hands are separated through the two-hand tracking module and separated if not. do. Since the two hands overlap each other, it is determined whether the predicted virtual areas of the two hands overlap each other, and first, it is determined whether the virtual areas overlap (S4).

If there is no overlap in the virtual region, the position where the blob coincides with the predicted virtual region becomes the position where the hand moves. At this time, the position of the elbow can be found by using the position of the head to predict the position of the shoulder point again. The inclination of the straight line connecting the elbow and the hand is stored for the moving direction information to be used in case of overlap (dotted line → S9) (S5 to S8).

에 관한 시계열이 기본적으로 한 시점 전의 변수값에 의존해서 결정되는 자기 회귀 과정이 있을 때, 그 관계를 수식으로 표현하면 Here, autoregressive analysis refers to a state in which a current value of a variable is determined depending on a past value. Generally

When there is a self-regression process in which the time series of is basically determined based on the value of a variable before a point in time,

--- 수학식 1

--- Equation 1

와

은 변수

의 한 시점 사이의 상호 관련을 구체적으로 나타내는 계수이며

는 한 시점 전의 값만에 의해서는 계통적으로 설명할 수 없는 불규칙적인 변동부분을 나타내는 확률변수이다. 이는 자기 회귀 과정 중에서 가장 간단한 것으로 1단계 자기 회귀 과정이라고 한다. 본 발명에서 사용한 자기 회귀는 2단계 방법으로 아래의 식으로 표현된다. Can be written. here

Wow

Is a variable

A coefficient that specifically describes the correlation between a point in time

Is a random variable that represents an irregular variation that cannot be systematically explained only by the value before one point in time. This is the simplest of the autoregressive processes and is called a one-step autoregressive process. Self-regression used in the present invention is represented by the following equation in a two-step method.

--- 수학식 2

--- Equation 2

로 나타내고 왼손과 오른손 각각에서 구한 직선을

라고 하면 거리 식은 수학식 3과 같이 표현된다. On the other hand, if there is overlap in the virtual region, the gradient value obtained from the previous image is applied to the second order autoregressive model to predict the direction in which the left and right hands move (S9). In step S10, if the two hands are not overlapped through the predicted direction, the two hands are close but not overlapped. When both hands overlap in each direction, Kanade-Lucas-Tomasi (KLT) feature information is extracted from an image in which current motion and skin color are matched (S11). The extracted feature information leaves only the important information necessary to use the straight line obtained from the slope (dotted line → S7). That is, the distance from the straight line is measured to use only feature information that is sufficiently close to the straight line. The position of both hands superimposed with this feature information is shown (S12). In addition, the current image is generated, and a virtual region of both hands is designated using the second regression analysis model from the previous image (S13 to S15). KLT characteristic information

And a straight line from each of the left and right hand

In this case, the distance expression is expressed as in Equation 3.

--- 수학식 3

--- Equation 3

The KLT values corresponding to the left hand and the right hand, respectively, can be obtained from Equation 4.

--- 수학식 4

--- Equation 4

값은 각 영상의 실험 조건에 따라 조절할 수 있다. At this time,

The value can be adjusted according to the experimental conditions of each image.

라 하 면, 국소 명암 변화 행렬은Here, the KLT feature value is defined as a feature value according to the texture pattern shown in the image using change information of the contrast of the image for tracking purposes on the image. You will find high contrast in both the x and y direction of the texture. In other words, the contrast

If we say, the local contrast change matrix is

---- 수학식 5

---- Equation 5

의 고유값(eigen values)이 미리 정의된 임계치

를 넘게 되는 곳은 이미지 상에서 KLT 특징값으로 선택된다. Can be expressed as At this time,

Threshold with predefined eigen values

The area beyond the is selected as the KLT feature value on the image.

2 is a conceptual diagram of a body and an arm for both hands extraction and tracking as an embodiment of the present invention.

Referring to Figure 2, the head (HEAD), left and right shoulder points (LSP, RSP), left and right forearms (LFA, RFA), left and right elbow points (LEP, REP), left and right arms (LA, RA), left and right hands (LH, Model for RH). As such, the body and arm are modeled for the extraction and tracking of both hands using the shoulder point, elbow point, and hand position for each hand.

Meanwhile, in order to distinguish each area of the body from the captured image, different colors are assigned to the image feature values or displayed using a specific symbol (+). It can be seen that the image feature value is indicated by a + mark on the hand. In addition, a feature value representing the image feature value is given another color and displayed as a line.

3 is a diagram modeling the arm of FIG. 2.

Referring to FIG. 3, the body and arm are modeled.

4 is a view showing a video feature used in the present invention and a straight line representative of the feature.

와

값을 이용하여 얻게 되는 실선으로 표시된 직선을 갖게 된다. 이에 새로이 획득한 KLT 특징값

과의 거리 D를 구하게 된다. 도 4로부터 수학식 1에서의 직선

가 자기 회귀 분석을 이용한 예측된 값과 KLT 특징값의 회귀선으로 검증된 직선은 팔꿈치와 손과의 직선을 나타냄을 확인할 수 있다. In FIG. 4, the K-type feature values of the x-type acquired from the new image are scattered in the region where the arm is to move. At this time, predicted by the second order autoregressive method

Wow

You will have a straight line marked with a solid line that you get using the value. Newly acquired KLT feature value

The distance D with is obtained. Straight line in Equation 1 from FIG.

It can be seen that the straight line verified by the regression line of the predicted value and the KLT feature value using the autoregressive analysis represents the straight line between the elbow and the hand.

Although the present invention has been described in more detail with reference to some embodiments, the present invention is not necessarily limited to these embodiments, and various modifications can be made without departing from the spirit of the present invention.

As described above, the two-hand detection and tracking method according to the present invention can be extracted and tracked in real time to be applicable to a system that requires use in a real environment such as a robot, and also has overlapping moving objects having similar properties. Each has the advantage of being tracked in situations.

Claims (7)

(a) The position of the shoulder point is detected using the position of the face, and the current value of an arbitrary variable in time series for the KLT (Kanade-Lucas-Tomasi) feature point selected as the feature point when the contrast of the image is greater than or equal to a preset threshold. Detecting a hand through a regression analysis determined based on the past value, and modeling both arms by estimating an elbow from the shoulder point position and the hand position; (b) detecting both hands from the modeling; And (c) determining a virtual region using a range of motion limits according to the structure of a human arm; And (d) comparing the previous image with the previous image to track the position of both hands by changing the inclination of the two straight lines connecting the elbow and the hand; Two-handed detection and tracking method comprising a. delete The method of claim 1, wherein before step (a), A method for detecting and tracking two hands, comprising extracting the positions of hands and faces using skin color information. The method of claim 1, wherein in step (b), both hands are: A method for detecting and tracking two hands characterized in that the motion information and skin color information are matched and detected using blob analysis. The method of claim 1, wherein step (d) Determining whether virtual regions of both hands overlap by performing a regression analysis in which a current value of an arbitrary variable in a time series is determined based on a past value of a previous image of the image; And When the virtual regions do not overlap, the current position of both hands is checked to extract the KLT (Kanade-Lucas-Tomasi) feature point selected as a feature point when the contrast of the image is greater than or equal to a preset threshold, and the regression analysis is performed to extract the elbow and the hand. Calculates the slopes of both straight lines and recursively analyzes the slopes obtained from the previous image when the virtual regions overlap, and again determines whether the two hands overlap, and determines whether the two hands overlap. If the two hands do not overlap, the virtual region does not overlap, and if both hands overlap, extracting the KLT (Kanade-Lucas-Tomasi) feature points to check the position of both hands. Two-handed detection and tracking method comprising the. The method of claim 5, wherein the slope calculated when the virtual regions do not overlap is used to predict a moving direction of a next image. The method of claim 5 or 6, wherein the extracted KLT feature points when the two hands overlap the two-hand detection, characterized in that to perform a regression analysis to use a straight line of the slope calculated when the virtual region does not overlap Tracking method.

Images