KR20160097513A - Paired-edge based hand tracking method using depth image - Google Patents

Abstract

The present invention relates to a paired-edge based hand tracking method, the method comprising the following steps: an initial hand detection step of using paired-edge group information of a shape in a depth image to detect a candidate finger and checking connection relation of the paired-edge group information to detect an initial hand; and a hand tracking step of tracking a shape including coordinates, which have a small depth change among stored coordinates, over a specific ratio in the next frame of a depth image to determine the tracked shape as a hand. According to the present invention, a main operation for initial hand detection and hand tracking is the same as an operation to detect paired-edge groups in an area of interest, so the initial hand detection and hand tracking are performed at the same time while keeping a fast speed. Further, since tracking is performed by using a simple method comparing a depth value of the current and the previous frame with respect to stored coordinates, an advantageous effect of requiring a less amount of operation is provided.

Description

{Pair-edge based hand tracking method using depth image}

The present invention relates to an edge pair-based hand tracking method in a depth image, and relates to a hand tracking method for detecting an initial hand and tracking a shape having a small depth change in a next frame.

Many images have been made by the rapid deployment of cameras, such as mobile handsets, laptops and tablets, cameras, video recorders for vehicles and CCTVs. On the other hand, in the field of smart TV, intelligent automobile, augmented reality (AR), wearable device and Internet of things (IOT), existing human-computer interface (HCI) computer interface (NIC), and natural user interface (NUI) / natural user experience (NUX).

Based on this trend, research on HCI based on image recognition requires image recognition for face, eyes, arms, legs and hands, and many research results have been published. The results of this research have been applied to the game industry and have been successfully commercialized. Games that can be adjusted by the user by recognizing the movements of the face, arms, legs and hands have been developed and widely used.

In order to precisely control the HCI, a technique capable of accurately recognizing the shape and movement of the hand is needed and studies are underway. General hand recognition studies can be divided into initial hand detection, shape recognition, and hand tracking.

The method of finding the initial hand based on the color image is largely influenced by the illumination, and it is difficult to segment the hand in the case of a cluttered background. To reduce the effect of lighting, studies using HSV color transformations have been conducted on the subdivision. However, there is still a problem that the hand is not recognized when it is influenced by the illumination and the hand of a color different from the established model.

 There is an approach to separate the foreground and background after finding the background for subdivision. After storing the background in the previous frames, a method of dividing the foreground and the background by calculation of the difference from the current frame has been proposed. However, if the background moves continuously or the foreground objects are fixed for a long time, There is a problem that separation becomes difficult.

There is a research result in which only the part with motion is detected and the part corresponding to the skin color range is recognized by hand among the detected areas. In this case, however, if the color of the object other than the hand is included in the skin color range, there is a problem.

A method using Haar-like features and Adaboost has also been proposed, which finds the initial hand or recognizes the shape of the hand. This method is applied to face detection very successfully, but it is difficult to obtain a high recognition rate because the background between objects in the hand affects the result and the hand shape is very diverse. Also, when considering rotation, the amount of computation can be increased.

In the depth image, it is easy to separate the reflected background and object, and the subdivision can be performed relatively easily in color image because it is not affected by illumination or color change. Depth images are stereo cameras and time-of-flight (TOF) methods. Depth images are widely used with the spread of MS Kinect. However, even if subdivision is easily performed using depth, it is not easy to find an initial hand.

To separate the background from the foreground in depth image, a method of storing the background or detecting only the part with motion was used. When HCI is used, the method of judging the nearest object in the depth image by early hand is widely used with the assumption that the hand is the closest object from the camera to other objects.

To improve accuracy in these methods, a method has been proposed that uses skin color, finds the body, and utilizes the spatial correlation of the body and the hand. Specify a range of depths and assume the objects in this range by hand. A method of finding a hand using this information after a face or body, which is relatively easy to detect than a hand, was proposed. The hand is complex in shape and difficult to detect because of large changes. In addition, when considering rotation, scale, and conversion, hand detection requires many operations. So far, the hand detection methods that have been studied are generally limited or require a large amount of computation.

Shape-based methods have been proposed for hand shape recognition. Convex hull was used to recognize the direction of the fingers and fingers laid out, or to recognize the fingers. After measuring the distance between the point and the pixels on the outline of the hand, a method of analyzing the shape of the hand or defining the fingertip using this distance has been tried. In this method, not only the distance but also the curvature of the outline, the distance between the detected elements, and the angle are utilized. In addition, a method of recognizing hands and fingers was proposed by looking at the shapes of circles and bars and analyzing their connections.

In the learning-based method, the Viola and Jones method is the most widely used method. In addition, new features and methods of using Adaboost are also presented. Using the skin color, candidate regions were found, and the faces were removed using the Viola and Jones method. In addition, the shape was recognized by SVM using SIFT after removing non-hand regions by template matching. For hand tracking, we used the distance between the hand found in the previous frame and the hand found in the current frame. Various methods such as Kalman filter, CAMSHIFT and optical flow can be used for hand tracking.

It is necessary to remove the assumption of the depth distance in the conventional method of finding the initial hand. In addition, a method for finding a hand after finding another object such as a face or a body must be an operation for finding an object other than a hand, and can not be applied to a case where only a hand exists in an image. A method of separating the hand and the background has been proposed. However, there is a restriction that the background and the camera should be fixed and the movement of the hand must be within a certain time.

In the hand shape recognition method, in the case of the shape-based method, the method of finding the initial hand and the method of recognizing the shape are often different. Therefore, if the initial hand detection and the detected hand shape analysis module are configured separately, and there is a possibility that a new hand may be continuously generated, there is a burden that these two methods must be continuously performed simultaneously.

In addition, there are inventions in which geometric information analysis of the fingers and hands is presented, but perform relatively complicated calculations. Therefore, it is necessary to increase the efficiency of the system by making the method of finding the initial hand and the shape analysis method similar. In the learning - based method, the problem of the initial hand detection is relatively small, but the performance is limited by the influence of the diversity of the hand shape and the background between the fingers.

Patent Literature 1 relates to a hand shape recognition apparatus and method using a finger pattern, in which a finger pattern is previously set, finger feature data is detected to detect a candidate pattern, and a finger pattern matching the candidate pattern is recognized as a hand Therefore, there are limitations in tracking various hand shapes and changing hand shapes.

Therefore, there is a need for a technique for detecting initial hand and detecting detected hand with little influence of noise, stable and low calculation amount.

1. Korean Patent Publication No. 10-2013-0109817

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems, and it is an object of the present invention to provide a hand tracking method for detecting an initial hand and tracking a shape having a small depth change in a next frame.

According to an aspect of the present invention, there is provided a method of detecting an object using a finger candidate (FC) using a PEG (paired-edge group) information in a depth image, Detecting an initial hand by detecting a connection relationship between the first frame and the second frame, and detecting an initial hand by detecting an association between the first frame and the second frame, And a hand tracking step of judging by hand the hand tracking.

In the preferred embodiment of the present invention, the initial hand detection step may include a finger candidate grouping (FC grouping) step of setting FCs having a depth difference within a certain range among the detected FCs as a group, (Connected-PEGs) generating the connected PEGs by confirming the PEG overlapping with the FC to generate connected PEGs, and if the number of FCs connected to the generated connected edge pair group is 5, And a palm area detecting step of detecting a palm area using the hand determining step of judging and the connecting position information of the finger and the palm.

In a preferred embodiment of the present invention, the connected edge pair group generation step includes a step of line-scanning the inputted depth image in a single direction and, when the finger candidate overlaps with a line connecting the start and end points of each edge pair, And judges that the candidate and each edge pair group are connected to each other.

In a preferred embodiment of the present invention, the palm area detecting step may include calculating and storing the detected palm area of the initial hand, calculating a size of the palm area according to the depth using the stored palm area size, And determining the position of the palm area by using the J-line connecting the start and end of the end edge pair coordinate at which the PEG generation stored in the hand is terminated .

In a preferred embodiment of the present invention, the step of tracking the hand comprises the steps of storing a range of ROIs of a rectangle using the maximum and minimum coordinate values of the connected PEGs constituting the detected hand, (X, y, z) coordinates of a hand tracking point (HTP) included in each connected-PEGs in the at least one connected PEG.

In the preferred embodiment of the present invention, the hand tracking step may include an FC detection step of detecting FC in the next frame of the depth image, a step of detecting the FCs in which the difference in depth is within a certain range among the detected FCs, Candidate PEGs in the region of interest, regenerating PEG within the region of interest to identify PEGs that overlap with FC to generate connected-PEGs, tracing connected PEGs using the HTPs contained in the connected PEGs, The HTP generation and the palm area are detected when the hand is determined by the hand determination step.

In a preferred embodiment of the present invention, the hand determining step determines that the HTP is located on a line connecting the start and end PEs, and when the depth value of the stored HTP is within a certain range of the z coordinates of the start edge pair and the end edge pair, Is included in the edge pair (PE).

In a preferred embodiment of the present invention, the hand judgment step may include counting the number of HTPs included in the connected PEG to include at least 25% of the total HTPs in each connected PEG, and at the same time, connected-PEGs are judged by the traced hand, and it is judged that the hand is disappeared when there is no connected-PEGs including 25% or more of the total HTP among the connected-PEGs.

In the present invention, object segmentation of depth images is relatively easy, and initial object detection is less affected by the background. Therefore, even if the background is complicated or the background is changed, it is not greatly affected by the initial object detection. Also, since there is no constraint on the depth range, the resolution of the depth image is sufficient and the initial object can be found at any depth position.

In addition, the present invention does not require pre-processing such as median or dilation for eliminating noises because it is affected less by the noise of the outline, and analyzes the connection relationship of the lines constituting the object, not the edge pixels, The computation amount is less than the method using the alignment of the points and the influence by the noise is small.

Since the hand tracking of the present invention does not analyze the shape of the hand, it can track various shapes of hands

In addition, the hand tracking of the present invention uses a simple method of comparing the depth value of the current frame with the depth value of the previous frame with respect to stored coordinates, so that the amount of computation is not so large.

In addition, the hand tracking of the present invention is a simple method of calculating the number of HTPs commonly present between connected-PEGs of a previous hand and a current hand using a hand tracking point (HTP) You can trace.

Figure 1 shows an edge pair (PE) and an edge pair group (PEG) according to the present invention.
Figure 2 is an image showing PEG production according to the present invention.
Figure 3 shows the angular calculation of PEG according to the invention.
4 illustrates calculation of thickness and length of a PEG according to the present invention.
5 is a view showing a finger PEG according to the present invention.
FIG. 6 is a graph illustrating the influence of angles of scan lines and finger directions in finger candidate generation according to the present invention. FIG.
7 is an image showing finger candidates incorporating finger candidates obtained from four directions of rotation according to the present invention.
8 is a view showing an outline of a finger detected by a finger candidate according to the present invention.
9 is a view showing a finger candidate and a finger candidate group according to the present invention.
FIG. 10 is a view showing an edge pair group included in connected-PEGs determined by hand and ROI of a finger candidate group according to the present invention.
11 is a view showing a J-line according to the present invention.
12 is an image showing the detected palm area according to the present invention.
13 is an image showing an extension of the palm area according to the present invention.
FIG. 14 is an image showing generation of a hand trace point (HTP) for hand tracking according to the present invention; FIG.
15 is a flowchart showing an operation procedure of a hand detection and hand tracking method according to the present invention.
16 is a flowchart showing a flow of a hand detection and hand tracking method according to the present invention.
17 is a view showing a hand trace according to the occlusion according to the present invention.
18 is an initial hand detection test image according to the present invention.
19 is a view showing an image of initial hand detection in which distortion occurs.
20 is an image showing the result of tracking the detected finger and palm according to the present invention.
Figure 21 shows tracked finger tip coordinates and measurement information for four depth cases according to the present invention;
22 is a view showing a hand detection result according to the present invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The drawings illustrate only the essential features for the sake of clarity of the invention and are not to be construed as limiting the drawings.

The edge pair-based hand tracking method according to the present invention includes an initial hand detection step and a hand tracking step.

In the initial hand detection step, a finger candidate (FC) is detected using geometric information of a paired-edge group (PEG) in a depth image, And detecting an initial hand (initial hand).

In the depth image, a PEG that can be a finger is detected using the edge pair (PE: paired-edge) on the left and right sides of the object, and a method of detecting the hand using the PEG is presented. The object is represented by PEG, which is a set of vertically connected edge pairs, and a finger candidate having a shape that can be a finger among the PEGs is detected.

A set of linked pairs of edge pairs (PEG) of PEGs can represent a single shape, which detects hand among the connected edge pair groups (PEGs).

The hand tracking step is a step of manually determining a shape that includes a coordinate having a small depth variation among a plurality of coordinates stored in the next frame of the depth image. The position and shape information in the next frame of the hand detected in the current frame is calculated. Tracking and shape analysis can be done by applying a simple method with a small amount of computation. In the initial hand detection, all five fingers detect the hand of the hand. However, in the hand tracking, a finger having an arbitrary shape is traced, and finger detection and shape analysis are performed.

The finger candidate detection step includes an edge pair detection step, a PEG generation step, a PEG geometric information calculation step, a finger candidate determination step, and an information storage step.

The edge pair generation step is a step of detecting an edge pair positioned on the left and right by reading the edge of the object in a raster scan order on the depth image.

A common method for displaying the outline of an object is to detect the edges of the object and then sort them in a spatially linked order and store the aligned edge coordinates or chain code. In the object detection method according to the present invention, in order to analyze an outline of an object, edge detection is performed with respect to the input image, and an edge pair method in which the left and right edges of the object are paired is used.

BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a diagram showing PE and PEG according to the present invention, which detects the edge of an object along a horizontal solid line. Fig. The ST point represents the left edge of the object and is called the start edge of the edge pair. The ED point represents the right edge and is called the edge of the edge pair.

If the depth difference between adjacent pixels in the direction of the scan line for detecting the edge is equal to or greater than the threshold value, the edge is determined. If the depth difference is 8 or more, it can be judged as an edge.

When edge pair detection is completed for one scan line, edge pairs are detected for the next line. This edge pair detection method proceeds in a raster scan order.

The PEG generation step is a step of collecting edge pairs connected in the vertical direction among edge pairs to generate PEG.

In order to represent an object using edge pairs, the vertically connected pairs of edge pairs are analyzed, and the vertically connected edge pairs are included in one group, and these groups are called PEG.

Vertical connections are determined using the connection of objects rather than the connection of edge points. (i, n) and the x-coordinates of the start and end edges of PE (i, n) are denoted by ST (i, n) and ED , And the vertical connection relation between the two edge pairs of the i and the (i + 1) th line is determined by the following condition I.

Condition I: Connectivity check between PE (i, n) and PE (i + 1, m)

If (ST (i, n) < = ED (i + 1, m)

PE (i, n) and PE (i + 1, m) are connected.

PE (i, n) and PE (i + 1, m) judged to be vertically connected by Condition I belong to one PEG. If PE (i + 2, p) detected in the (i + 2) th line is vertically connected to PE (i + 1, m) , p) are added. Edge pairs for all lines are searched for the depth image, and edge pairs connected to each other are analyzed by analyzing the vertical connection relationship of the edge pairs so that the pairs of edges connected to each other are included in one PEG.

In FIG. 1, an edge pair of an object of the n-th line is denoted by PEn, and a start edge and an end edge of PEn are denoted by STn and EDn, respectively. When Condition I is used to generate a PEG, ST1 to ST7 and ED1 to ED7 constituting the outline of this object belong to one PEG. The PEG thus constructed contains the outline of one object. It is determined that one object is composed even if the edge pixels are not adjacent to each other as in ST3 and ST4 because the connection relationship between the lines constituting the object is analyzed. The generation method of PEG is less affected by the noise of the outline, and it is possible to judge the composition of the object in a line unit by performing the raster scanning procedure, and it is simpler than the method of using the alignment of the existing edge points.

The PEG generation step may generate the PEG by checking the relationship between the edge pair of the current scan line and the edge pair of the previous scan line of the object.

In the case of a general input image, there exist objects having complex shapes other than a simple object as shown in FIG. We defined the PEG generation process for complex objects as five cases. Below are the conditions for judging these five cases and what should be done in each case. The edge pairs present in the current scan line and the previous scan line are denoted by Cur_PE and Prev_PE, respectively.

Case I ( Start ): There is no Prev_PE associated with Cur_PE. This means that Cur_PE is the starting point of the object. Since this is the start of a new object, create a new PEG and set Cur_PE to the tip of the object.

Case II ( End ): There is no Cur_PE associated with Prev_PE. This means that Prev_PE is the end point of the object. Since it means the end of the object, the edge pair is no longer added to the PEG, and Prev_PE can be the tip of the object.

Case III ( Continue ): Cur_PE and Prev_PE are connected to each other and are not connected to other edge pairs. This means that the object extends vertically. So we add Cur_PE to the PEG to which Prev_PE belongs.

Case IV ( Merge ): A number of Prev_PEs are connected to one Cur_PE. This means that multiple objects are gathered in Cur_PE. Creates a PEG with Cur_PE as the starting point, without adding edge pairs to the PEGs to which Prev_PEs belong. In addition, the connection relationship between the newly generated PEG and the PEGs to which Prev_PE belongs is stored.

Case V ( Branch ): A number of Cur_PEs are connected to a Prev_PE. This means that Prev_PE is split into multiple objects. Creates PEGs starting with Cur_PEs, without adding edge pairs to the PEG to which Prev_PE belongs. It also stores the connection relationship between the newly created PEGs and the PEG to which Prev_PE belongs.

FIG. 2 is an image showing PEG generation according to the present invention. Edge pairs are detected while the scan line is progressing from top to bottom, and a horizontal straight line means a boundary of PEG. In the left figure, the starting point of the finger corresponds to Case I, so the tip is set as tip and a new PEG is generated containing the tip. Each finger adds edge pairs to the PEG corresponding to Case III as the scan line goes down. At the point where two fingers meet, a case of merge corresponding to Case IV, in which the expansion of the PEG of the finger is stopped and a new PEG is generated.

The right picture shows the case where the finger is pointing downward. The end of the finger corresponds to Case II, in which case the edge pair is no longer added to the PEG. The location of the fingers corresponds to the branch case of Case V and generates new PEG as many as the number of cracked PEGs.

In the PEG geometric information calculation step, the object is represented by a set of connected PEGs, and the length, thickness, area, and angle information of the PEG are calculated in a line unit in the process of generating the PEG.

The PEG geometric information calculation step may include storing the midpoint of the tip of the generated PEG as a starting point of the angle measurement, calculating an angle between the midpoint of the fingertip and the midpoint of a pair of edges spaced by a predetermined distance or more, Calculating the length and thickness for the PEG of the interval calculated angle, updating the angle, length and thickness of the PEG when the edge pair is added to the PEG, and updating the angle between the start point and the end point of the added edge pair And calculating the area of the PEG by accumulating the numbers.

Each time an edge pair is added to the PEG, information about the angle, length, thickness, and area of the PEG is updated. In order to obtain the angle of the PEG, the angle of the midline of the edge pairs belonging to the PEG is measured. When the angle is measured in a small range, the influence of the distortion of the outline of the PEG is largely affected. So it is necessary to calculate the angle of the whole PEG.

Fig. 3 is a diagram showing the angle calculation of PEG according to the present invention, and Fig. 3 (a) shows this example. The bold line represents the outline of the PEG, and the circle represents the center of the edge pair obtained by scanning in the horizontal direction. When measuring an angle with respect to all the lines, the angle between all the vertically adjacent circles in Fig. 3 (a) must be calculated and converted back to the angle of the PEG.

In the present invention, the angle between the center point of one edge pair and the midpoint of a pair of edges spaced apart by 8 pixels is measured. By applying this method, the influence of the distortion of the outline can be reduced. 3 (b) is a diagram showing an example according to the present invention.

Once the tip is found, the tip's midpoint is stored as the starting point for the angle measurement. The circle in Fig. 3 (b) means this starting point. As shown in FIG. 3 (b), 64 boxes are formed in the shape of a rectangle around the circle. If the midpoint of the edge pair obtained from the subsequent scanline analysis meets one of these boxes, measure the angle between this position and the starting point of the angle measurement.

In FIG. 3 (b), the heavy point of line 9 is centered and the angle is the same as the number in the box. When calculating the angle at the intersection of the edge pair center and the box, set this intersection as the starting point of the new angle measurement and measure the angle in the same way. Fig. 3 (c) is a diagram showing an example of this method.

The length and thickness of the PEG are affected by the angle between the scan line and the PEG. FIG. 4 shows the calculation of the thickness and length of PEG according to the present invention, showing the results of obtaining two angles for PEG in FIGS. 3 (b) and 3 (c). In Fig. 4, the dark box indicates the starting point of the angle measurement. The angle formed by the scan line and the PEG is referred to as? _{SC_PEG} , the distance of the y coordinate between the start point of the angle measurement and the measurement position is referred to as PEG_height, the edge at the angle measurement position, The distance between the pair is called PEG_width. However, these values do not represent the length and thickness of the object corresponding to line 1 through line 9. This is because the object is skewed by θ _{SC-PEG_1} . Calculate the length and thickness for the PEG of the section where the angle is calculated. The length and thickness in consideration of the angles are calculated by the following equations (1) and (2).

This length and thickness are calculated for each unit of measurement of angle as shown in FIG. The area of the PEG is calculated by accumulating the number of pixels between the start and end edges of the added pair of edges whenever the pair of edges is added. Since the PEG is generated in units of scan lines and the information about the shape of the PEG is calculated and updated, the analysis and judgment of the PEG can be speeded up.

The finger candidate determination and information storing step is a step of finding a PEG which can be a finger using the information and storing information. Determining a PEG having a thickness, a length, and an area within a range that can be judged by a finger as a finger candidate, the PEG including a tip among the PEGs generated for the depth image; if the PEG is determined as a finger candidate, Storing coordinates of an angle pair, a length, a thickness, an area, an edge pair coordinate of the fingertip and an edge pair connected to the palm, and terminating the shape analysis if the PEG judges that the finger candidate is impossible.

The shape information such as the length, thickness, and area of the PEG is used to detect the PEG which can be a finger, and this PEG is indicated as a finger candidate. One finger must be generated as one PEG because it is determined whether it can become a finger candidate by analyzing the shape of one PEG.

As shown in the left image of FIG. 2, one finger starts from the tip and corresponds to Case IV at the portion where the other finger meets the adjacent finger, thus ending the expansion of the PEG. Or PEG is generated by Case V as shown in the right image of FIG. 2, and PEG expansion is stopped by the end of Case II. Thus, since the boundary of the PEG occurs at the position where the finger meets the palm, the finger is highly likely to be represented by a single PEG. When only one finger is open, the finger PEG can not be separated from the palm by Case IV and Case V. To solve this problem, when the thickness of the PEG changes abruptly, the expansion of the previous PEG is terminated, a new PEG is generated, and the connection between the two PEGs is stored.

FIG. 5 shows an image of the finger PEG according to the present invention, showing that the finger is separated into one PEG. In FIG. 5, the circle means the tip of the PEG, and the horizontal line is the line connecting the start edge and the end edge of the last edge pair of the PEG. FIG. 5 (a) shows a finger PEG separated by Case IV, and FIG. 5 (c) shows an example of finger PEG separation by a thickness change. In both cases, it can be seen that PEG is similar to the area of the finger.

To be a finger candidate, the PEG should have a tip created by Case I or Case II, and should be within the range of thickness, length, and area judged by the fingers. The geometric information of each PEG is continuously updated on a scan line basis.

When PEG is terminated by Cases II, IV and V, it is judged whether it can become a finger candidate. In addition, when the shape is analyzed on a line-by-line basis and it is judged that it can not be a finger candidate, the shape analysis is no longer performed. When it is judged as a finger candidate, it stores the coordinates of the angle, length, thickness, area, tip edge pair and PE connected to the palm. If it is judged that it can not be a finger candidate, the shape analysis for this PEG is stopped and only the connection relation is judged from the next scan line. Since the shape of the PEG is updated on a line-by-line basis, it is not necessary to store all pairs of edges belonging to the PEG. In this way, finger candidates are detected for the entire image.

The finger candidate detection step may include detecting and adding a finger candidate to an image obtained by rotating the depth image by 90 °, 180 °, and 270 ° after completion of detection of a finger candidate with respect to the depth image, Deg.] To integrate the finger candidates having a distance close to each other, and selecting the correct geometric information among the overlapped finger candidates.

When the scan input direction and the direction of the finger coincide with each other, the finger region is not classified as one PEG, and therefore, such a finger is not detected as a finger candidate. FIG. 6 is an image showing the influence on the angle of the scan line and the finger direction in the finger candidate generation according to the present invention, in which only three finger candidates are detected and the remaining finger is not detected. PEG has a tip, but it is judged that it can not be a finger because it is very short in length and thick in thickness. The thumb is also not detected as a finger candidate for this reason.

To solve this problem, a finger candidate is detected for images rotated 90 °, 180 ° and 270 ° together with the input image. When this method is applied, the finger candidates can be detected relatively accurately since the direction of the index and the thumb fingers is perpendicular to the scan line, as shown in Fig. In order to solve the problem of FIG. 6, only detection for an image rotated by 90 degrees is added. In order to simplify the implementation, finger candidates are detected in four directions. In order for PEG to be a finger candidate, it must include a tip, which is generated by Case I and Case II.

Since the tip generated by Case II is detected at the end of the PEG considering the order of the line scan, all PEGs may include the tip at the end. Do. Therefore, there is a problem that shape information is calculated and stored for all PEGs generated. If we include only the tips generated by Case I, it is not necessary to analyze shape information for PEGs not created in Case I, so that unnecessary operations and memory are not needed. If a finger candidate is searched only in the PEG generated by Case I for one image and rotated 180 °, the same results as those obtained by searching for a finger candidate considering both Case I and II in one image can be obtained. The scan is performed in the direction in which the y coordinate increases with respect to the two images rotated by 90 ° with the original image. If the scan is performed in the direction in which the y coordinate decreases with respect to the two images, the result of detecting the finger candidate for the image rotated by 180 ° and 270 ° can be obtained.

The finger candidates obtained in the four scanning directions may be overlapped. For example, a finger indicated by B in FIG. 6 is detected as a finger candidate in the original image, and is detected as a finger candidate even in an image rotated by 90 degrees. To eliminate this redundancy, the finger candidates obtained in each direction are rotated by 0 °, and the finger candidates having a short distance are integrated into one. At this time, it is necessary to select the correct geometric information among the duplicated finger candidates. In Fig. 6, the finger candidate indicated by A indicates that if the PEG indicates the finger region well, the finger candidate indicated by B does not contain much of the finger region in the PEG. That is, the closer the angle &amp;thetas; between the scan line and the finger is to 90 DEG, the better the PEG and the finger area match. For the same finger, the closer to 90 the angle tends to be, the shorter the distance between the start and end edges of the edge pairs. The maximum value of the distance between the start and end edges of the edge pairs constituting the finger candidate is stored and the accuracy of the data is improved by storing the geometric information of the short finger candidate when integrating the overlapping finger candidates.

FIG. 7 is an image showing finger candidates incorporating finger candidates obtained from the four scan input directions according to the present invention, in which the circle represents the tip of the finger candidate, and the straight line represents the line of the last edge pair of the finger candidate PEG.

The thumb finger candidate was detected in 270 ° rotated image, the small finger candidate was detected in the original image, and the remaining fingers were detected in 90 ° rotated image. This result shows that finger candidate PEG can be generated irrespective of the direction of the finger. Finger candidate detection through the 4 - way scan occupies the largest portion in the operation speed, and plays a very important role in detecting the rotated hand.

Since the hand detection method according to the present invention is less influenced by the outline noise of the depth image, a preprocessing process for eliminating noise is not required. Since the vertical connection relation of the straight lines connecting the start and end edges of edge pairs is determined instead of the edge line connection relationship, the influence of noise is small with a small amount of computation. In order to reduce the influence of vertical noise, if the length of the straight line connecting the start and end edges of the edge pairs is 5 pixels or less, it is not determined as an edge pair.

FIG. 8 is a view showing an outline of a finger detected with a finger candidate according to the present invention. In FIG. 8 (a), a portion denoted by 'A' has a very strong noise in the horizontal direction. This horizontal noise does not affect the vertical connections of the straight lines connecting the start and end edges of the edge pairs. In FIG. 8 (b), the portion denoted by 'B' shows noise generated in the vertical direction. If the vertical noise region is recognized as a new tip and a new PEG is generated, the PEG containing the finger tip and the PEG containing the noise tip will correspond to Case IV in the scan line where the PEG meets. Therefore, the PEG containing the real tip is excluded because the tip is short in length, and the PEG newly created by Case IV thereafter is excluded because it does not have a tip.

If the length of the straight line connecting the start and end edges of the edge pair is 5 pixels or less, the PEG including the tip is not generated due to noise because it is not determined as an edge pair. Therefore, it is possible to generate stable PEG without receiving the influence of the background noise of the depth image without performing a preprocessing process such as median or dilation for the depth image.

The hand detection step includes a finger grouping (FC grouping) step, a connected PEG (PEG) generation step, a hand judgment step and a palm area detection step.

In the hand detection step, the connection relationship of the detected finger candidates is analyzed, and the shape of the hand of all the five fingers can be detected and judged by hand. In the HCI system based on image recognition, it is assumed that a hand shape in which five fingers are extended is recognized as the start of user input.

The finger candidate grouping step is a step of setting the finger candidates having a depth difference within a predetermined range among the detected finger candidates as a group.

In order to reduce false positives, we used a range of hand sizes according to depth. The size of the hand along the depth is represented by a square. Through the experiment, the width of the square is obtained as a piecewise-linear equation as shown in Equation 3 below.

The size of the hand is determined by substituting the depth of the finger candidate for one finger candidate into the equation 3 and the difference between the current finger candidate and the depth of the finger candidates existing within the range of the hand size around the finger candidate is within a certain range And sets the finger candidates with one group. Finger candidates included in a group can be fingers constituting one hand.

FIG. 9 is a view showing a finger candidate and a finger candidate group according to the present invention, and FIG. 9 (a) shows a finger candidate detected from a depth image. Finger - like PEGs were detected as finger candidates. In FIG. 9 (b), finger candidates belonging to a group having five finger candidates among the finger candidate groups obtained from the input image are displayed. It appears that five fingers were included in one group and no other finger candidates were included.

The connected-PEGs generation step regenerates the PEG in the finger candidate group to identify the PEG overlapping the finger candidate to generate the connected PEG. It is possible to determine that the finger candidate and each PEG are connected when the input depth image is line scanned in a single direction and the finger candidate is overlapped with a line connecting the start and end points of each edge pair.

It is judged only by the distance between the finger candidates, so that even if five finger candidates exist in the finger candidate group, they may not be connected to each other. Therefore, it is confirmed whether the finger candidates in the finger candidate group are connected to each other. (ROI) is predicted by predicting the hand size with respect to the average depth of the finger candidate group using Equation (3), and the connection relation of the finger candidates in the ROI is grasped. To do this, line scan is performed on the input image in the region of interest to regenerate the PEG. In this case, scanning is performed in only one direction since it is not the detection of the finger candidate but the main purpose of grasping the connection relation of the finger candidates.

In the process of detecting the PEG, when the finger candidate overlaps the line connecting the start edge and the end edge of each edge pair constituting the PEG, it is judged that the finger candidate is connected to the PEG, and the finger candidate connected to each PEG is stored do. It also preserves the linkage between PEGs while generating PEG.

Once the PEG generation for the region of interest is completed, information about the connection between the finger candidates belonging to each PEG and the PEGs can be obtained. PEGs connected to each other are referred to as connected PEGs. The connected PEG means one object in the image.

The hand judgment step is a hand judgment step of judging by hand if the number of finger candidates connected to the generated connected PEG is five. There may be more than one linked PEG in the region of interest (ROI), checking the number of linked finger candidates for each linked PEG. The connected PEG with 5 connected finger candidates is judged by hand.

FIG. 10 is an image showing the PEG included in the connected PEG judged by the hand and the region of interest of the finger candidate group according to the present invention. The horizontal line inside the detected hand indicates the boundary of the PEGs, Were marked in circles. The outline of the area detected by hand was also displayed.

The step of detecting the palm area comprises the steps of detecting the palm area using the connection position information of the finger and the palm, calculating and storing the size of the palm area of the palm detected, using the size of the palm area And the size of the palm area according to the depth is stored and stored in the palm area of the palm area, and the J-line connecting the start and end edges of the coordinates of the end edge pair, And determining the location of the region.

When a hand is detected, determine the size and position of the square palm area. After calculating and storing the palm size of the detected hand, the tracking step calculates the size of the palm area according to the depth using the size of the stored palm area.

The size of the palm area is determined as the width of the palm area by five times the average thickness of the fingers stored in the finger candidate of the hand. Since the size of the palm area in the input image changes with depth, the size of the palm area measured is converted into the size at the reference depth and stored. To do this, the rate of change in width according to the experimentally obtained depth was measured and expressed as Equation (4). If the depth is 30, the width is 1, and if the depth is greater than 90, the width when the depth is 90 is used.

The calculated average palm size is stored in terms of depth 30, which is called Palm_size _INIT , the average depth of the detected hand is d _INIT , and the average thickness of the finger is indicated by Thick _AVG . Palm_size _INIT is given by equation (5).

The size of the palm area changed by the depth of the hand in the next frame is calculated using the following equation. In the case where a hand is detected in the hand trace of the next frame, if the depth of the hand is d _HAND , the width of the palm area is given by Equation (6).

After the size of the palm area is determined, the position of the palm area is determined. The palm area is determined to include the palm of the palm except for the fingers and the wrist as much as possible. The finger candidate contained in the hand stores the coordinates of the tip edge pair where the PEG starts and the end edge pair where the PEG generation ends. A straight line connecting the start edge and the end edge of the end edge pair is referred to as a J-line (junction line).

Since the J-line refers to the line where the finger and the palm meet, it is assumed that the J-line is the boundary of the palm. This assumption provides relatively accurate information on finding the boundary of the palm. In order to set the palm position using J-line, it can be classified into 4 types as follows.

Horizontal up line (HU-line): If the J-line is horizontal and the tip is above the J-line

Horizontal down line (HD-line): If the J-line is horizontal and the tip is below the J-line

Vertical left line (VL-line): If the J-line is vertical and the tip is on the left side of the J-line

Vertical right line (VR-line): If the J-line is vertical and the tip is on the right side of the J-line

11 shows an example of HU, HD, VL and VR-lines, showing a J-line according to the present invention.

This J-line means the position where the finger and the palm meet, and plays an important role in determining the position of the palm. There are 5 J-lines in the initial hand, and 0 ~ 5 J-lines in the hands detected by tracking. Since there may be only one type of J-line depending on the image input, or there may exist all four types, there are many cases of the number and types of J-lines. To simplify this, the x-coordinate and the y-coordinate position of the palm area are separated and judged.

1. Y position of palm area:

(1) When HU-line or HD-line exists: If there is one HU-line, the y-coordinate of HU-line is indicated as HU_ymin. If there are several HU-lines, the minimum value of y coordinates of HU-lines is indicated as HU_ymin. HU_ymin is the highest J-line among the HU-lines. If the HD_line is 1, the y-coordinate of the HD_line is denoted by HD_ymax. If there are multiple HD lines, their maximum y-coordinate is denoted by HD_ymax.

If both HU_ymin and HD_ymax are present, the midpoint between HU_ymin and HD_ymax is set to the y coordinate of the center of the palm area. If only HU_ymin is present, the y coordinate of the upper line of the palm area is HU_ymin. If only HD_ymax exists, the y coordinate of the lower line of the palm area is HD_ymax.

(2) When there is no HU-line and HD-line and maximum and minimum of y coordinates of Connected-PEGs: Set the minimum value among the y coordinates of the edge pairs constituting the hand object to HU_ymin and set the maximum value to HD_ymax . When the pixel coordinates of the maximum and minimum values are located at the boundary of the ROI, it is determined that there is no corresponding maximum or minimum value because the ROI is not the end of the actual ROI but the ROI. If there are two HU_ymin and HD_ymax thus obtained, the one with the maximum and minimum coordinates closest to the palm region of the previous frame is selected. If only one exists, the y-coordinate of the palm area is set in the same manner as the above-mentioned method.

(3) Case not applicable to (1) and (2): If there is a VL-line or VR-line, set the y-coordinate range of the palm area using the maximum and minimum of the y- . Otherwise, it is judged that the palm area is not detected.

2. x position of the palm area:

To obtain the x-coordinate of the palm area, set the minimum value of the VL-line to VL_xmin and the maximum value of the VR-line to VR_xmax. If both VL_xmin and VR_xmax are present, the average of these values is taken as the x coordinate of the center of the palm area. When only VL_xmin exists, the x-coordinate of the left line of the palm area is VL_xmin, and when only VR_xmax exists, the x-coordinate of the right line of the palm area is VR_xmax. If both VL_xmin and VR_xmax are not present, use the maximum and minimum values in the x direction of the object, or use the x-coordinates of the HU-line and HD-line, Setting.

12 is an image showing the detected palm area according to the present invention, wherein the bold line represents the J-line and the rectangle represents the detected palm area. 12 (a), the minimum y-coordinate among the HU-lines is selected, and the x-coordinate is selected from the VL-line to determine the position of the palm area. 12 (b) positions the palm area using the HU-line and the VR-line. 12 (c), the y-coordinate is determined by the HU-line. Since there is no J-line for determining the x-coordinate, the position of the palm area is determined by using the maximum and minimum coordinates of the x-coordinate. 12 (d) shows a case where there is no J-line, and the position of the palm area is determined by using the maximum value and the minimum value of the x and y coordinates.

And a hand tracking step of tracking an object including a coordinate having a small depth change among the coordinates stored in the next frame when the initial hand detection of the depth image is completed and then judging the object by hand.

Wherein the step of tracking a hand comprises the steps of storing a range of a region of interest of the rectangle using the maximum and minimum coordinate values of the connected PEGs constituting the detected hand and determining a hand tracking point (HTP) (x, y, z) coordinates of a tracking point.

The position and shape information in the next frame of the hand detected in the current frame is calculated. In the present invention, a simple method with a small computation amount is applied to track and shape analysis. In the initial hand detection, all five fingers detect the hand of the hand. However, tracking is performed on a hand having an arbitrary shape, and finger detection and shape analysis are performed.

When detecting a hand in the next frame, the hand can be detected in the range of the region of interest defined by the shape of the rectangle. To define the region of interest, the maximum and minimum values of the x and y coordinates of each PEG stored in the PEG generation step are used. The minimum values of the x and y coordinates for the connected PEG constituting the detected hand are represented by X _{MIN_HAND} and Y _{MIN_HAND} , and the maximum values of the x and y coordinates are represented by X _{MAX_HAND} and Y _{MAX_HAND} . A _rectangle with upper left corner and lower right corner (X _{MIN_HAND} , Y _{MIN_HAND} ) and (X _{MAX_HAND} , Y _{MAX_HAND} ) is the range of detected hands. Since this region of interest is the minimum size rectangle containing the hand detected in the current frame, the hand of the next frame will deviate from this region of interest if there is hand motion. Therefore, the upper left corner and the lower right corner are defined as (X _{MIN_HAND -} α, Y _{MIN_HAND -} α) and (X _{MAX_HAND} + α, Y _{MAX_HAND} + α), respectively. alpha can be set to 40 pixels. In this case, there is a problem that the width of the region of interest continuously increases in every frame.

FIG. 13 is an image showing an extension of the palm area according to the present invention, wherein a rectangle represents a region of interest, and xmin, xmax, ymin, and ymax are maximum values of x and y coordinates of a hand- It represents the minimum value. Xmin, xmax, ymin, and ymax are detected in the region of interest input in FIG. 13 (a). In FIG. 13 (b), the width of the region of interest is extended by? In four directions. 13 (c) shows a case where the generated region of interest is applied to the next frame. If the detected object exists outside the region of interest in the ymax direction, ymax increases by α even if there is no change in the input image. 13 (d) shows that the region of interest is increased in the y-direction. Repeated actions can expand the region of interest to the entire object connected to the hand. In order to prevent such an extension, the range of the interest region is limited so that the positions of the upper left corner and the lower right corner of the rectangular area of interest are located within 1.5 times the width obtained from the upper left corner and the lower right corner of the palm area square.

Because hands are flexible objects and have many movements and rotations, their shapes are very diverse in depth images. Therefore, it is difficult to find the corresponding hand in the current frame based on the hand shape of the previous frame. Several points are defined inside the detected hand object, and the (x, y, z) coordinates of these points are stored. This point is referred to as a hand track point. The hand is traced using the number of hand tracking points included in each connected PEG generated within the region of interest for tracking of the next frame. When generating hand tracking points, the number of hand tracking points is proportional to the size of the detected hands. The number of pixels of the PEGs constituting the detected hand is stored. The number of hand tracking points is determined by dividing the total number of pixels of the hand by 512. The hand tracking point thus obtained is again assigned to the PEG in proportion to the area of each PEG constituting the hand. It is judged that the connected PEG including most of the hand tracking points set in the previous frame is the same as the hand of the previous frame. It is advantageous for the hand tracking point to be located at the center of the hand in order for the hand tracking point to exist in the same hand in the next frame even if there is movement of the hand. Thus, the hand tracking points assigned to each PEG were uniformly distributed in the central part of the PEG.

FIG. 14 is an image showing generation of a hand tracking point for hand tracking according to the present invention, wherein a quadrangle represents an area of interest and the points inside the hand are hand tracking points. In Figure 14 (a), 19 hand track points were generated, and hand track points were uniformly distributed in the center of the detected PEG. Figure 14 (b) is a larger hand than Figure 14 (a) and 38 hand tracking points were created.

The hand tracking step includes a finger candidate detection step, a finger candidate grouping step, a connected PEG generation step, a hand determination step, a ROI setting, a hand tracking point generation, and a palm area detection step.

The finger candidate detection step detects a finger candidate in the next frame of the depth image, and the finger candidate grouping step sets the finger candidates having a depth difference within a predetermined range among the detected finger candidates as a group.

The linked PEG generation step regenerates PEG within the region of interest to identify PEG overlapping the finger candidate to generate linked PEG.

When the region of interest and the hand track point are determined for a hand detected in one frame, the process in one frame is terminated. When the operation for the next frame starts, a finger candidate is detected, a finger candidate group is generated, and the hand detected in the previous frame is tracked. A 0 ° direction scan is performed on the stored region of interest for each hand detected in the previous frame to find a connected PEG.

The hand judgment step is a step of judging the PEG connected by using the hand tracking point included in the connected PEG as a tracked hand. During the scan, find finger candidates and hand tracking points in each connected PEG.

Wherein the hand tracing step comprises the steps of: if the hand tracing point is located on a line connecting a pair of starting and ending edges and the depth value of the stored hand tracing point is within a certain range of the z coordinate of the starting edge pair and the ending edge pair, It can be judged to be included.

Expressing a hand using the connected PEG and judging the finger candidates and the palm area connected thereto is the same as finding the initial hand. In the process of detecting pairs of edges for each scan line, the hand trace point is located on the line after the start and end PE, and the z coordinate of the stored hand trace point, that is, the depth value is within a certain range of the z coordinate of the start edge pair and the end edge pair It is judged that the hand trace point is included in this edge pair.

 If the difference in z coordinates is less than 8, the hand tracking point is judged to be included in the edge pair. The connected PEG with this edge pair is judged to contain one hand trace point.

Wherein the hand determining step includes counting the number of the hand tracking points included in the connected PEG to include at least 25% of the total hand tracking points among the connected PEGs, Judging from the traced hand, it can be judged that the hand has disappeared when there is no connected PEG including 25% or more of the total hand trace points among the connected PEGs.

The number of hand tracking points included in the PEG is counted while generating the connected PEG. Once the scan for the region of interest is complete, the traced hand will include more than 25% of the total hand tracking points among the connected PEGs, and the largest number of hand tracking points among the connected PEGs in the region of interest.

If there is no connected PEG corresponding to this condition, it is judged that the hand has disappeared.

If it is judged by hand in the hand judgment step, it is possible to detect a region of interest, a hand tracking point generation and a palm area for a next frame. The initial hand is detected for the entire image, and the region of interest is generated by the number of hands detected in the previous frame to perform hand tracking. In this process, the finger candidate included once in the detected hand is excluded from the finger candidate list so that the finger candidate contained in the hand is not detected as duplication in the other hand. The method of detecting a hand in the hand tracking of the present invention does not analyze the shape of the hand, so that it is possible to trace a hand of various shapes. Also, since the number of hand tracking points included in the connected PEG is counted, the amount of computation is very small.

The edge pair-based hand detection system according to the present invention includes a finger candidate detection module and a hand detection module.

The finger candidate detection module can detect a finger candidate using the PEG information of the object in a depth image. Depth images are read in raster scan order to detect edge pairs located on the left and right sides of the object, and edge pairs connected in the vertical direction among these edge pairs are collected to form PEG. PEGs are also interconnected, and an object can be represented as a set of connected PEGs. The length, thickness, area, and angle of PEG can be calculated in the unit of line in the process of PEG generation, and the PEG which can be a finger can be searched using these geometric information.

Since the finger candidates are detected and the shape information is analyzed by a simple operation which is processed in a line unit, the operation amount is relatively small and a fast operation speed can be obtained.

The hand detection module can detect a hand by grasping a connection relationship between the detected finger candidates. The palm area is detected using the connecting position of the finger and the palm by finding the initial hand by grasping the connection relation between the detected finger candidates. Since the initial hand is based on the analysis of the shape of the connected PEGs, there is no condition for the depth region or restriction on the face or body first. The constrained traces do not have much computation because they use a simple method of comparing the depth values of the current frame and the previous frame for the stored coordinates.

15 is a flowchart illustrating an operation procedure of a hand detection and hand tracking method according to the present invention. When the depth image is input, the finger candidate is detected and the shape information is stored. If there is a hand detected in the previous frame, a tracking operation is performed on this hand. After hand tracking is completed, the initial hand is detected for the entire input image.

16 is a flowchart illustrating a flow of a hand detection and hand tracking method according to the present invention. In the finger candidate detection step, a PEG is generated for the input image and a shape of the PEG is analyzed to generate a finger candidate that can be a finger. After the finger candidate is detected for the input image and the image rotated by 90 °, 180 °, and 270 °, the finger candidates obtained from each rotation image are rotated and integrated again to the original coordinates.

In the hand tracking step, the PEG is generated in the region of interest of the previously detected hand. In this process, the finger candidate and hand tracker included in the linked PEG are searched, and the hand is tracked using the number of hand tracking points included in the connected PEG.

When the hand is judged, the palm area and the hand track point are generated, and the area of interest is updated for tracking of the next frame. If the hand can not be found within the area of interest, it is judged that the hand has disappeared. This operation is repeated for the number of hands detected in the previous frame. Early hand detection In the process, the connected PEG including five finger candidates is judged by hand. First, a finger candidate group and a region of interest of each finger candidate group are generated in consideration of the distance between the detected finger candidates.

Calculate the number of finger candidates contained in the connected PEG while generating the linked PEG within the region of interest. If the number of finger candidates is 5, judge by hand and store new hand. Create a palm area and hand tracking point for this hand and update the area of interest. When the operation for all the finger candidate groups is finished, the process for the next frame starts.

Image analysis is processed line by line. Most operations such as edge pair detection and vertical direction connection analysis, PEG generation and shape analysis, finger candidate search, finger candidate and hand trace point detection included in PEG, and connected PEG generation for hand area definition are performed on a line basis . Since necessary information is calculated and judged on a line-by-line basis, it is not necessary to store all the outlines of the object, and only the calculated information can be stored. In addition, it is possible to make a judgment during operation, thereby reducing unnecessary operations and improving the efficiency of the operation of reading data from the memory. In addition, the operation to be used is not complicated, which is advantageous in terms of operation speed.

In the input depth image, there are many small noises in the outline of the hand object. Since it is less influenced by such noise, a preprocessing process such as median or dilation is not necessary. Therefore, the operation speed can be improved by omitting the preprocessing process.

In the depth image, object segmentation is relatively easy and the initial hand detection is less affected by the background because it analyzes the connected shape of five PEGs that can be a finger. Therefore, even if the background is complex or the background is changed, it is not greatly affected by the initial hand detection. In addition, if there is no constraint on the depth range and the resolution of the depth image is sufficient and the finger is divided, the initial hand can be found at any depth position.

The order of initial hand detection and detected hand tracking is very similar. The difference between the two methods is that the initial hand detection is detected by the connected PEG generated in the region of interest including the five finger candidates and in the case of the hand tracking, the ratio of the hand tracking points contained in the connected PEG generated in the region of interest Use your hand to track. Other methods are the same as the method of using finger candidates calculated in the previous step and generating and analyzing PEGs linked in the region of interest. Therefore, the calculation efficiency is high because the calculated hand candidate is used in both cases, compared with the method in which the initial hand detection and the hand tracking process are different from each other.

The finger detection and shape are calculated by a relatively simple operation, and the palm area is detected. The angle of the finger is defined as the direction of the center of the pair of edges contained in the PEG of the finger. Using the calculated angles, the thickness and length of the finger can be updated in line units, and the change in length and thickness can be stored at regular intervals. The palm area is defined using the maximum and minimum values of the x and y coordinates of the PEG connected to the connecting position of the finger and the palm.

Hand is a flexible object and it is very difficult to trace using hand shape because deformation and rotation are relatively severe. Using a hand tracking point, a very simple method of tracking the number of hand tracking points in common between the previous hand and the current connected PEG was applied. In this method, the shape of the hand is so complex that tracking can be maintained even if shape analysis is difficult. Also, even if a part of the hand is obscured in the current frame, tracing is possible if the number of hand-tracking points included in the currently connected PEGs is sufficient.

Fig. 17 is an image showing a hand trace according to the occlusion according to the present invention, and Fig. 17 (a) shows a detected hand when no occlusion occurs. The outline of the hand represents the outline of the detected hand, the rectangle represents the palm area, and the points represent the hand tracking point. Fig. 17 (b) shows a case where a part of the thumb and the palm of the hand is covered. Even in this case, all the finger tips not covered are detected, and the palm area is well shown. Fig. 17 (c) shows a case in which four fingers are obscured. In this case as well, a tip of a single finger not covered is detected, and the palm area is well represented.

If a part of the detected hand is occluded, it can be seen that the hand tracepoint is distributed in the center of the uncovered PEG.

More specifically, an edge pair-based hand tracking system and method according to the present invention will be described with reference to the embodiments and experimental examples.

Experimental Example 1: Evaluation of Hand Detection Performance and Operation Speed

Experimental images were obtained using a depth image of 640x480 resolution obtained from Kinect and executed on a computer using an Intel i7-4770 @ 3.4GHz CPU. In the step of detecting the finger candidates for the speed improvement, the horizontal direction scan line is performed only for the odd line. Therefore, the resolution in the step of detecting the finger candidate is equal to 320x480. All of the lines were scanned when generating PEGs linked in the region of interest for initial hand detection and hand tracking. That is, in order to precisely search finger candidates and hand track points, the final detected hand was analyzed on a 640x480 image.

In order to measure the initial hand detection performance, we used a video sequence with five fingers in unfolded, with continuous shifts in movement and rotation. This sequence has two relatively complex backgrounds, each consisting of 770 frames.

18 is an initial hand detection test image according to the present invention, in which a circle indicates a position of a finger tip and a box indicates a palm area. The straight line in the area where the finger and the palm meet represents the J-line, and the straight line connecting the finger tip and the J-line midpoint is displayed. The tracking function was excluded to measure only the initial hand detection performance. The initial hand detection performance is shown in Table 1 below.

It was judged that the initial hand was correctly detected when 5 finger tips were found and a frame with more than 50% of the detected palm area containing the palm was obtained. Of the 770 frames, the initial hand was detected at 759 frames and the detection rate was 98.57%. In the case of recognition failure, there were 7 frames in the case of a false positive (false positive) and 2 frames in which no finger was detected. The cases where more than 50% of the palm area were not palms occurred in two frames.

FIG. 19 shows an image of initial hand detection in which a distortion occurs. FIG. 19 (a) shows a finger-like object generated by distortion of a depth image in the wrist portion. FIG. 19 (b) shows a case where the initial hand is not recognized, because the shape distortion of the tip portion of the fourth finger is not recognized as a finger. Fig. 19 (c) shows a frame whose palm area is less accurate. If only the initial hand is searched, the average processing speed of one frame measured for 770 frames is 9.49 msec.

Experimental Example 2: Evaluation of Hand Tracking Performance

To evaluate the tracking performance, a test sequence was generated for cases with 0, 1, and 2 hands. In each case, the hand continues to rotate and move. In a sequence with zero hand, the number of spread fingers varies from 0 to 4, and if the number of hands is one and two, the number of spread fingers varies from 0 to 5. A sequence with zero hand count consists of 339 frames, and a sequence with 1 hand and 2 hands consists of 1310 and 766 frames, respectively.

FIG. 20 is a view showing a result of tracking the detected finger and palm according to the present invention, and FIGS. 20 (a) and 20 (b) show an example of a case where there is one hand and two hands. In this case, the detection rate is measured frame by frame based on the number of finger tips detected and the accuracy of the palm area as in the experiment of FIG.

Fig. 20 (a) shows the result when the number of hands to be tracked is one. The numbers on the upper left of each figure represent the frame numbers of the degrees. The frames other than the frame 10 are enlarged in the hand portion. It can be seen that the finger tip and palm area are well detected even when the number of spread fingers changes, or when the hand moves and turns. Frames 308, 343, and 382 show good tracking in the x, y, and z directions even when there is no spread finger.

Fig. 20 (b) shows a case in which the number of hands to be tracked is two. In this figure, the number in the lower left corner indicates the frame number of each image. Images other than frames 80, 86 and 150 are enlarged images of the hand. In 80 frames, the far hand is recognized first, and the near hand is not recognized as the initial hand because the number of spread fingers is not 5. In the 86 frame, all the fingers of a nearby hand are unfolded and recognized as an initial hand, then tracked from the next frame. Since all five fingers are recognized as an initial hand, there is no restriction on the range of depth for the initial hand recognition, and thus it is possible to detect any range of hand depth. At 150 frames, the detected hand is tracked even if it moves in the z axis direction. Frames from 330 to 450 seem to look good even when the number of fingers changes, and frame 525 shows that two fingers are tracked well even when there are no fingers spread out. Frames from 660 to 760 show that hand tracking works well even with rotation.

For the cases where the number of hands is 0, 1, and 2, the recognition rate and the operation speed are shown in Table 2 below.

If the hand is 0, the hand is not recognized because it has no initial hand, and the hand is not recognized in every frame. The average speed of processing one frame at this time is 7.88 msec. When the number of detected hands is 1, the recognition rate is 98.24%, and the operation speed is 9.93 msec. When the number of hands is 2, the recognition rate is 99.48% and the operation speed is measured as 12.11msec. As the number of hands to track increases, the connected PEG must be generated within the region of interest of each hand, thus slowing down the operation.

To analyze the tracking performance, the trajectory of the index finger was traced to track the (x, y, z) coordinates and this coordinate was compared with the ground truth. The tracking error is defined as the Euclidean distance between two points when the coordinates of the traced finger tip are (xT, yT, zT) and the ground truth coordinates are (xG, yG, zG) This error distance is given by the following equation.

Here, the z coordinate position uses the integer type depth input from Kinect, and the depth range has a value from 0 to 255.

FIG. 21 is a diagram showing tracked finger tip coordinates and measurement information for four depths according to the present invention, in which the two axes of the plane represent the x and y axes, and the vertical axes represent the z axis representing the depth. The solid line represents the coordinates of the tracked finger tip, and the dotted line represents the ground truth. For each depth case, we can see that the two cases are roughly matched. The following Table 3 shows the mean and maximum error distances.

The first column of Table 3 shows the number of frames of the sequence used in the experiment for each depth, and the second column shows the depth range for each depth. The third and fourth columns show the mean and maximum values of the tracking error distances. The average error is 1.957 in the nearest case and 3.291 in the farthest case. The average error increases with the distance. When the distance increases, the distortion of the depth image increases and the position accuracy of the finger tip detection decreases. The average tracking error distance is 2.458.

FIG. 22 is a picture showing the hand detection result according to the present invention. FIG. 22 shows detection results in the case where the number of hands is 0 to 5 in the input image. In each case, the hand is detected and both the palm area and the finger tip are detected. The average running time according to the number of hands is measured for more than 100 frames. Table 4 below shows the results of measuring the time required to process one frame in each case.

When one hand is used, the processing speed of one frame is 12.58 msec and the processing speed of five hands is 22.10 msec. It shows that a large number of hands can be processed at the same time.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those skilled in the art that various changes and modifications may be made therein without departing from the spirit and scope of the invention as defined in the appended claims. It will be possible.

Claims (5)

In the depth image, finger candidates (FCs) are detected using the information of paired edge groups (PEGs) of objects, An initial hand detection step of detecting an initial hand
And a hand tracing step of tracing a shape including a coordinate having a depth variation smaller than a previous one of the coordinates stored in the next frame of the depth image by a predetermined ratio or more and judging the shape manually. 2. The method of claim 1,
A finger grouping step (FC grouping) of setting FCs having a difference in depth within a predetermined range among the FCs detected as a group;
Regenerating the PEG within the FC group to identify connected PEGs with the FC to generate connected PEGs;
A hand judgment step of judging by hand if the number of FCs connected to the generated connected edge pair group is 5, and
And a palm area detecting step of detecting a palm area using information on the connection of the finger and the palm. 2. The method according to claim 1,
Storing the range of the ROI of the rectangle using the maximum and minimum coordinate values of the connected PEGs constituting the detected hand, and
(X, y, z) coordinates of a hand tracking point (HTP) included in each connected-PEGs in the region of interest, characterized by further comprising the step of: . 4. The method according to claim 3,
An FC detection step of detecting FC in the next frame of the depth image;
A finger candidate grouping step of setting the FCs having a depth difference within a certain range among the FCs detected as a group;
Regenerating PEG within said region of interest to identify connected PEGs by identifying PEG overlapping FC;
Determining hands connected PEGs using the HTP included in the connected-PEGs;
Wherein if the hand judgment is made at the hand judgment step, the region of interest setting, the HTP generation and the palm area for the next frame are detected. 5. The method according to claim 4,
It is determined that the HTP is included in the edge pair (PE) when the HTP is located on the line connecting the start and end PE and the depth value of the stored HTP is within a certain range of the z coordinate of the start edge pair and the end edge pair. Edge pair based hand tracking method.

Images