KR101955596B1 - System and method for tracking human using two depth images - Google Patents

Abstract

The present invention provides a human tracking system and method using two depth images. The system comprises: a first depth sensor for viewing a predetermined target area at an angle from a side; A second depth sensor for looking down the target area vertically; Wherein the position of the first object in the target area is determined and set to the first position when the first object detected from the first depth image by the first depth sensor is determined as a human body, Determines the position of the second object in the target area and sets it to the second position when the second object detected from the second depth image by the second depth image is determined to be the human body and compares the first position and the second position with each other And determining the first object and the second object as a human body when it is determined that the first object and the second object are at the same position.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a system and method for tracking two depth images,

The present invention relates to a system and method for detecting and tracking a human body from a depth image of a target area.

Recently, a depth sensor camera combining RGB camera and depth sensor is popular. In addition, techniques for recognizing the human body by recognizing the shape and the distance of the object by analyzing the depth image generated by the depth sensor camera and analyzing the shape and movement of the object have been developed.

In particular, if a skeleton model corresponding to the joints of the human body is identified from the depth image, it can be determined that the human body has been detected. Furthermore, the movement of the human body can be recognized by recognizing the motion of the skeletal model.

As a related art related to this, Japanese Patent Application Laid-Open No. 10-1636171 entitled Skeleton tracking method and skeleton tracking system using the same, can be mentioned.

In the above conventional technique, N photographing information for an arbitrary human body having a plurality of joints from N depth sensor cameras (N is an integer equal to or greater than 2) is generated from the photographing information, Extracting N positional information from each of the joints, calculating integrated position information for each of the joints from N positional information in each of the joints, To form a skeleton model for the human body. Thus, it is disclosed to calculate the combined position information for each of the joints from the N positional information in each of the joints.

In such a conventional technique, since N depth sensor cameras of 2 or more are used, the number of necessary equipments is large, the system configuration cost is high, and information processing burden is large. Also, it is impossible to determine the object as a human body because it can identify the skeletal model.

Therefore, there is a need for a more improved method for precisely distinguishing between objects and human bodies from depth images generated by shooting in a specific target area in which objects and human bodies are mixed.

In addition, there is a need to more accurately identify the movement of the human body in the target area, and to analyze the positional relationship between the human body and the object identified.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide a method for accurately identifying a human body in a depth image in which objects and a human body are mixed, and a system for implementing the method, in order to solve the above problems.

The present invention also provides a method for identifying the motion of the identified human body and a system for executing the method.

According to an aspect of the present invention, there is provided a human body tracking system using two depth images, including: a first depth sensor for viewing a predetermined target area obliquely from a side; A second depth sensor for looking down the target area vertically; Wherein the position of the first object in the target area is determined and set to the first position when the first object detected from the first depth image by the first depth sensor is determined as a human body, Determines the position of the second object in the target area and sets it to the second position when the second object detected from the second depth image by the second depth image is determined to be the human body and compares the first position and the second position with each other And determining the first object and the second object as the human body when it is determined that the first and second objects are at the same position.

Here, when the shape of the first object detected from the first depth image is analyzed by a specific algorithm, and the skeleton structure of the human body is acquired, the first object can be determined as a human body. Further, the algorithm is characterized by including processing by a NiTE library.

In addition, the analyzing apparatus can determine the human body when the shape of the second object detected in the second depth image includes a hemispherical head shape and a shoulder shape extending from the head shape. Further, the second depth sensor is offset by an offset so as to detect a shape detected within a predetermined reference distance, and the reference distance is a distance that can best detect the head and shoulders of the human body, Is set. Alternatively, the second depth sensor scans forward in a linear form along the first axis direction to generate a first line image, and at a position shifted in the second axis direction, And the analyzing device can perform object detection and human determination using one or more of the line images.

In addition, the analyzing apparatus may further include: a determination unit configured to determine, when the first object and the second object are determined as a human body, to use the at least one of the first position and the second position to move the determined human body in the target area And a function of tracking the determined gesture of the human body using the skeleton structure.

The comparison of the first position and the second position may further include defining the target region as a rectangular shape having four vertexes (A2, B2, C2, D2) in the second depth image, (A2, b2, c2, d2) for each side of the square shape of the second position (xb, yb) is defined as a second coordinate plane, and the percentage of the position with respect to the length of each side (1, ty), c2 (tx, 1), d2 (0, ty)) of each of the intersection points and calculates the entire first depth image as a first coordinate (A1, B1, C1, D1) corresponding to each of the four vertexes of the target area in the first coordinate plane to define a transformation target area, and the coordinate of each of the set vertexes (X2, y2), C1 (x3, y3), and D1 (x4, y4) , d2) of the deformed target zones (A1, b1, c1, d1) on the side of the intersection a1 and the intersection c1 are determined and the coordinates of the straight line L1 connecting the intersection a1 and the intersection c1 are calculated by using the formula The formula of the straight line L2 connecting the intersection b1 and the intersection d1 is obtained and the intersection Pc of the straight lines L1 and L2 is determined using the formula 3, The coordinates Px and Py of the first position Pa in the plane can be determined and the position of the determined interchange point Pc and the position of the first position Pa can be compared.

(1)

(2)

(3)

Further, according to another embodiment of the present invention, there is provided a method of acquiring a first depth image, the method comprising: acquiring a first depth image by a first depth sensor that obliquely faces a predetermined target area at a side; Determining a position of the first object within the target area and setting the first object to a first position if the first object is detected from the first depth image and the detected first object is determined to be human; Obtaining a second depth image by a second depth sensor viewing the target area vertically; Determining a position of the second object in the target area and setting it to a second position if a second object is detected from the second depth image and the detected second object is determined to be a human body; And determining the first object and the second object as the human body when the first position and the second position are compared with each other to determine the same position, the human body tracking method using two depth images is provided do.

The determining whether the first object is a human body may further include determining the human body when the skeletal structure of the human body is acquired by analyzing the shape of the first object detected from the first depth image using a specific algorithm And determining whether the second object is a human being is determined when the shape of the second object detected in the second depth image includes a hemispherical head shape and a shoulder shape extending from the head shape, And determining that the object is a human body.

The comparison of the first position and the second position may further include defining the target region as a rectangular shape having four vertexes (A2, B2, C2, D2) in the second depth image, (A2, b2, c2, d2) for each side of the square shape of the second position (xb, yb) is defined as a second coordinate plane, and the percentage of the position with respect to the length of each side (1, ty), c2 (tx, 1), d2 (0, ty)) of each of the intersection points and determines the entire first depth image as a coordinate plane (A1, B1, C1, D1) corresponding to each of the four vertices of the target area in the first coordinate plane to define a transformation target area, and the coordinates (A1 (x2, y2), C1 (x3, y3), and D1 (x4, y4) ) To each side of the deformation target zone (A1, b1, c1, d1) to determine the corresponding coordinates. Using the formula (2), the intersection (a1) and the intersection (c1) (L2) connecting the intersection (d1) and the intersection (d1) of the straight lines (L1, L2) is determined and the intersection Pc of the straight lines (L1, L2) And determining a coordinate of the first position Pa and comparing the position of the determined intersection Pc with the position of the first position Pa.

INDUSTRIAL APPLICABILITY The present invention including the above-described structure can accurately identify a human body even in a depth image in which objects and a human body are mixed.

Further, the present invention can identify the motion of the identified human body. As a result, it is possible to detect and alert the dangerous situation that may occur to the human body.

1 shows a configuration of a human tracking system using two depth images, according to an embodiment of the present invention.
2 is an overall flowchart of a human body tracking method using two depth images according to the present invention.
3 is a flowchart illustrating a procedure for determining whether a first object detected from a first depth image is a human body.
Fig. 4 shows an embodiment of the human determination procedure of Fig.
5 is a flowchart illustrating a procedure for determining whether a second object detected from the second depth image is a human body.
Fig. 6 is a diagram for explaining an example of implementing the procedure of Fig. 5; Fig.
7 is a diagram for explaining a method of matching the coordinate plane of the second depth image and the coordinate plane of the first depth image.
8 is a diagram showing an example of implementing a procedure for comparing the first position and the second position as described above.
FIG. 9 shows a method of specifying the position of an object determined to be human in each target area.
Fig. 10 is a diagram for explaining a procedure for tracking the movement of the human body.

Hereinafter, a preferred embodiment of a human tracking system and method using two depth images according to the present invention will be described with reference to the accompanying drawings. For the sake of reference, terms referring to the respective elements of the present invention have been exemplarily named in consideration of their functions, and therefore, the terms of the present invention should not be predicted and limited in terms of the term itself.

First, a configuration of a human tracking system using two depth images according to an embodiment of the present invention will be described with reference to FIG. Referring to the drawings, the human tracking system of the present invention includes two depth sensors 110 and 120 (a first depth sensor and a second depth sensor) for generating two depth images of a predetermined target area, And an analyzer 200 for detecting a human body from the depth images and analyzing and tracking the detected movement and gestures of the human body.

The target zone Z is the area to be detected for presence, movement and gesture of the human body. In addition, not only a human body but also objects can be placed in the target area, and the positional relationship between the human body and the object (for example, a human body approaching an object or a moving object approaching a human body, etc.) .

This target zone Z may be set mainly at the bottom, but may also be set at the sidewall surface or at a specific height. Further, the target zone Z may be set as a polygonal shape by a plurality of vertexes such as a triangle, a square, and the like. Alternatively, it may be set to any type of area, and at least one reference point for setting a predetermined coordinate axis inside or outside the set area may be defined. In the present invention, it is assumed that the target area is set to a rectangular shape having four vertexes.

The first depth sensor 110 may be installed to obliquely look at the target zone Z from the side. That is, the first depth sensor 110 may be installed on any sidewall facing the target area, installed on a cradle, or installed on the ceiling. The first depth sensor 110 may adjust the angle of view and the focus so that the whole of the human body H or the object T that may appear in the target zone Z can be photographed.

Also, the first depth sensor 110 may include a plurality of cameras arranged in parallel, and the first depth image may be generated from a stereo image.

The first depth image generated by the first depth sensor 110 may be transmitted to the analysis apparatus 200.

The second depth sensor 120 can be disposed and operated so as to look down the target zone Z from the ceiling down, especially the center of the target zone. The second depth sensor 120 scans forward in a linear form along a first axis direction to generate a first line image and moves along a first axis direction at a position shifted by a predetermined distance in a second axis direction It is possible to generate the second line image and to continuously scan the entire target area Z by repeatedly generating the line image.

Each generated line image can be transmitted to the analyzer in real time.

On the other hand, the second depth sensor 120 can be offset to perform distance measurement only to an object located within a predetermined distance. For example, if the distance from the second depth sensor 120 to the bottom of the target zone Z is 3 m, if the offset is set to 2 m, only the shape of the object 1 m or more and 3 m or less from the floor Will be. Further, if it is desired to detect a moving human body walking on the target zone Z, the offset can be set to 1.5 m, thereby obtaining a better line image including the standing head and shoulder shape of the standing human body .

The analyzer 200 analyzes the input first depth image and the second depth image (or line image), respectively, to detect the first object and / or the second object, the detected first object and / It is possible to perform a function of determining whether the second object is a human body. It is also possible to perform the function of analyzing the position, movement and gesture of the determined human body in the target area.

On the other hand, the analysis apparatus 200 may be provided with a control apparatus 201. The control device 201 can provide an interface between the present system and the operator.

2 is an overall flowchart of a human body tracking method using two depth images according to the present invention. The illustrated method can be performed by a human tracking system using the above two depth images.

The analysis apparatus 200 obtains a first depth image obliquely viewed from a side of the predetermined target area from the first depth sensor 110 included in the human body tracking system (S110). Subsequently, it is detected whether the first object exists from the obtained first depth image (S120). If the first object is detected from the first depth image, it is analyzed whether the detected first object is human (S130). If it is determined that the first object is a human body, the position of the first object (or human body) in the target area is calculated, and the calculated position is set to the first position Pa (S140).

Meanwhile, the analysis apparatus 200 acquires a second depth image looking vertically down the target zone Z from the second depth sensor 120 included in the system (S210). Here, the procedure (steps S110 to S140) for acquiring and processing the first depth image and the procedure for acquiring and processing the second depth image (steps S210 to S240) ) May be performed simultaneously or alternately in parallel.

The second depth image may include a line image in the first axis direction with respect to the target zone Z. [ The analysis apparatus 200 analyzes the obtained second depth image (or line image) to detect a second object (S220), and analyzes whether the detected second object is a human body (S230). If it is determined that the second object is a human body, the position of the second object (or human body) in the target area can be calculated, and the calculated position is set to the second position Pb (S240).

When the first position Pa of the first object determined as the human body and the second position Pb of the second object determined as the human body are respectively identified, It is determined whether the position Pb is the same position (S150). If the first position Pa and the second position Pb are at the same position or within a predetermined range (S160), it is assumed that the first object and the second object represent the same human body, The second object can be determined as a human body (S170). When it is determined to be the human body, the position, movement, and tracking of the determined human body are started (S180). The above procedures can be repeatedly returned to the beginning and then repeated (S190).

In another embodiment, the analysis apparatus 200 detects a second object from the second depth image and determines whether the detected second object is human. If the second object is determined to be human, the position (Pb, i.e., the second position) in the target area of the second object is converted to the position in the first depth image (Pc). On the other hand, the analysis apparatus 200 can detect the first object from the first depth image to determine whether the object is a human, and calculates the position (Pa, i.e., the first position) of the first object determined to be human. It is also possible to determine whether the converted position Pc and the first position Pa are the same and confirm the human body.

Each procedure / step of the method is described in more detail below.

3 is a flowchart illustrating a procedure for determining whether a detected first object is a human body when a first object is detected from the first depth image. The first depth image is an image obliquely viewing the target zone Z. Therefore, the whole of the human body can be photographed by the first depth sensor. An algorithm for determining the human body by judging the entire shape of the human body from the first depth image is required.

First, it is possible to detect whether or not a predetermined object other than the wall or floor of the target zone Z exists from the first depth image (S120), and the shape of the detected object (i.e., the first object) (S122). Then, it is determined whether a skeleton model of the human body is formed. If a skeleton model of the human body is configured (S124), the first object can be determined as a human body (S130). The determination to the human body at this time is not definite but may be a provisional determination (that is, the human body is confirmed through steps S150 to S170). On the other hand, in step S124, if the skeletal model of the human body is not configured, it is determined that the first object is an object, and the procedure returns to step S110 to detect whether another object has been photographed.

Fig. 4 shows an embodiment of the human determination procedure of Fig. Here, since the first depth image is an image obtained by obliquely viewing the target area, the target area in the form of a rectangular plane can be deformed into a trapezoid shape (see FIG. 7).

In the first depth image of FIG. 4 (a), a box-shaped object T and one human body H are simultaneously photographed in the target zone Z. FIG. The analysis apparatus 200 can identify the object T and the human body H and set them as the first object. On the other hand, if an image of each of the two detected first objects is analyzed by an arbitrary algorithm, for example, a NiTE algorithm, a box-shaped skeletal model (TS), limbs and joints The human skeletal model (HS) can be obtained. The analyzer 200 recognizes the human skeleton model HS and determines the corresponding first object H as a human body.

On the other hand, the NiTE algorithm for determining a human body is a well-known algorithm for extracting a skeletal model of a human body from the shape of an object. However, it should be understood that the present invention is not limited to the NiTE algorithm for analyzing the skeletal model of the human body, and it is understood that it is possible to utilize other algorithms that can be developed as well as selecting other known algorithms .

Next, FIG. 5 is a flowchart for explaining a procedure for determining whether the detected second object is a human body when the second object is detected from the second depth image. 6 is a diagram showing an example in which the procedure of FIG. 5 is implemented.

The second depth image is generated by looking down the target zone Z from the ceiling or vertically downward. In particular, since the second depth sensor 120 generates a line-shaped depth image with respect to the target area, the analyzer 200 executes an algorithm for detecting the human body from each of the generated line images.

6 (a) shows a vertical view of the presence of two objects T1 and T2 and one human body H in the target area.

6 (b) shows an example of the respective line images obtained at points (1), (2) and (3) in FIG. 6 (a). Here, although a number of line images can be generated in the space between each of the points (1), (2), and (3), the line image shown in FIG.

Here, it is assumed that the second depth sensor 120 sets an offset of 1.3 m from the bottom of the target zone. Therefore, the object T2 having a height of 0.5 m was not detected in the line image corresponding to (1) point.

On the other hand, in the line image corresponding to the point (2), the box-shaped upper portion of the object T1 having the height of 1.5 m and the upper body portion of the human body H having the height of 1.7 m were obtained.

③ No object was detected at the point.

At this time, the analysis apparatus 200 can detect two second objects T1 and H from the line image corresponding to the point (S220). Then, the analysis apparatus 200 analyzes the shape of each of the two detected second objects (S222). The shape of the human body H to be photographed from the depth image viewed vertically may be mainly a head shape and a shoulder shape. Accordingly, the analysis apparatus 200 will attempt to identify the head shape and the shoulder shape in the obtained line image (S224). That is, when the hemispherical head shape and the shoulder shape extending to both sides symmetric to the head shape are identified, the human body can be determined (S230). Therefore, the second object T1 is not judged as a human body, and the second object H will be judged as a human body. On the other hand, if it is determined in step S224 that the second object is not a human body, it is determined that the second object is an object, and the procedure returns to step S210 to acquire an object from another second depth image .

On the other hand, for example, if a rectangle-shaped target zone Z having four vertexes is defined, the second depth image is an image viewed downward from the target zone Z vertically. Therefore, (Z2, or the second target region), the first depth image is an image obliquely viewing the target region Z, so the target region will appear to be transformed into a trapezoidal shape (Z1, or first target region).

Therefore, matching the position of the object and / or the human body obtained from the first depth image with the position of the object and / or the human body obtained from the second depth image requires additional processing. For example, a procedure for transforming the target area of the distorted deformed shape of the first depth image into the rectangular shape of the second depth image and matching the coordinate plane should be performed.

Or a second position Pb which is the position of the human body in the target area at the second depth image and converts the identified second position Pb to a specific point Pc in the target area of the first depth image , It can be determined whether the converted specific point Pc indicates the same position as the first position Pa identified in the first depth image or within a predetermined range. In the present invention, this method is used.

This method will be described with reference to Fig. 7 shows an example of converting the second position Pb in the coordinate plane in which the target zone Z2 is seen in the second depth image into a point Pc in the target zone shown in the first depth image Lt; / RTI >

In Fig. 7 (b), the target area can be defined as a rectangular (e.g., square) shape having four vertices and sides, thereby indicating that the second position can be expressed in a rectangular coordinate system. In FIG. 7 (a), the target area set at the bottom of the predetermined area is seen to be deformed in a trapezoidal shape as it looks obliquely.

FIG. 7 (b) shows each vertex A2, B2, C2, and D2, a second target zone Z2, and a second target zone Z2 for the second target zone Z2 viewed from the second depth image, (A2, b2, C2, d2) and a second position (Pb) set on each side constituting the side of the first lens group.

The coordinates of an arbitrary point in the target area in the second depth image can be expressed by a ratio with the sides generated by the vertexes A2, B2, C2, and D2 as axes and the length of each side as 100% . That is, the coordinates of the illustrated second position Pb may be defined as Pb (xb, yb) or Pb (tx, ty).

Therefore, the coordinates of the foot (or intersection) of the perpendicular to each side of the second position Pb are a2 (tx, 0), b2 (1, ty), c2 ). ≪ / RTI >

On the other hand, the second position Pb can be converted to the position of the target zone Z1 of the first depth image.

7A shows a first target zone Z1 obliquely viewed from a first depth image obliquely facing the target zone and its respective vertexes A1, B1, C1, D1, a first position Pa (xa , ya).

In the first depth image, the entire depth image is defined as a first coordinate plane (represented by a rectangle including a dotted line in FIG. 7A), and the coordinate axes (orthogonal coordinate system) of the image itself are used as they are.

First, a vertex corresponding to each of the vertexes A2, B2, C2, and D2 constituting the second target zone Z2 of the second depth image can be assigned to the first depth image (A1, B1, C1, D1). The trapezoidal shape connecting the designated vertices (A1, B1, C1, D1) is defined as a modified target area (Z1, or first target area). On the other hand, the coordinates of each vertex in the first coordinate plane can be determined. (X1, y1), B1 (x2, y2), C1 (x3, y3)

Next, each of the intersections a2, b2, c2, and d2 from the second position Pb set on each side of the target zone Z2 of the second depth image is moved to the target zone Z1 of the first coordinate plane To the positions a1, b1, c1, and d1 of the respective sides (see Fig. 8). To this end, the coordinates of each vertex of the first target zone Z1 and (Equation 1) may be used.

(1)

Subsequently, the formula of the straight line L1 connecting the intersection a1 and the intersection c1 is obtained by the formula (2), and the formula of the straight line L2 connecting the intersection b1 and the intersection d1 is obtained. And determines the intersection Pc of the obtained straight lines L1 and L2. Then, the coordinates of the determined intersection Pc are compared with the coordinates of the first position Pa. When the two are coincident with each other or within a predetermined range, the object at the second position (i.e., the second object temporarily determined as a person) And the object at the first position (i.e., the first object temporarily determined as a person) indicate the same person, whereby the first object and / or the second object can be determined as a person.

(2)

(3)

8 is a diagram showing an example of implementing a procedure for comparing the first position and the second position as described above. When the second position Pb is determined from the first depth image in the rectangular target zone Z2 having four vertexes A2, B2, C2, and D2 as shown in FIG. 8 (b) The coordinates of the position Pb can be determined as a percentage of each side. For example, if the ratio tx of the waterline to the transverse axis of the second position Pb to the transverse length of the second target zone is 0.7 and the ratio to the longitudinal length of the waterline to the second target zone length, ty ) Is set to 0.6.

On the other hand, in the first depth image as shown in Fig. 8 (a), four vertexes A1, B1, C1, and D1 of the deformed target zone Z1 can be designated and the coordinates of each vertex can be determined have. A1 (30,60), B1 (70,60), C1 (25,90), D1 (75,90).

Then, the coordinates of each of the intersections a1, b1, c1, and d1 are calculated as follows using Equation (1).

(x, y) = ((30 x 0.3) + (70 x 0.7), (60 x 0.3) + (60 x 0.7)

b1 (x, y) = ((70 x 0.6) + (75 x 0.4), (60 x 0.6)

(90 x 0.3) + (90 x 0.7)) = (60, 90) c1 (x, y) =

d1 (x, y) = ((25x0.4) + (30x0.6), (90x0.4) + (60x0.6)) =

Next, the formula of the straight line L1 in the vertical direction and the formula of the straight line L2 in the horizontal direction are determined by using the coordinates of each of the intersections a1, b1, c1, and d1 and (Formula 2).

L1: y-60 = {(90-60) / (60-58)} (x-58) y = 15x-810

L2: y-72 = {(72-72) / (28-72)} (x-28) y = 72

Next, Pc (59,72) can be obtained by obtaining an intersection point of two straight lines (L1, L2) using (Equation 3).

When the coordinates (xa, ya) of the first position (Pa) of the first object determined as the human body from the first depth image are similar to each other with an error of a predetermined range from or coinciding with (59,72) It is possible to determine that the object at the first position and the object at the second position are the same human body.

On the other hand, with reference to Fig. 9, a method of specifying the position of an object determined as a human body in each target area will be described. In order to compare the position of the human body in the first depth image with the position of the human body in the second depth image, it is necessary to determine each position as a specific point or area.

The human body H determined in the first depth image can generally consist of a model (HS) including a head, two arms, and two hand and trunk skeletal forms. Therefore, the determined position Pa of the human body H can be determined by the position of the center of gravity of the human body model, and simply, the center of the end of the two legs can be determined as the position of the human body. Fig. 9 (a) shows that the center of the foot of the human skeleton model (HS) is determined as the position Pa.

The human body determined in the second depth image may include a head shape that is generally hemispherical and a cylindrical shape that extends from both sides of the head. Thus, the determined position of the human body can be defined as the midpoint between both shoulders. FIG. 9 (b) shows that the center of the shoulder shape is determined as the position Pb.

In addition, each of the target zones Z1 and Z2 may be divided into a plurality of grid units, as shown in FIG. 7 and the like, and the position comparison procedure described above may be performed for each grid unit. If the position Pa of the first object determined as a person is within a certain grid and the position Pb of the second object determined as a person is within the grid, then both can be determined to indicate the same person will be.

Fig. 10 is a diagram for explaining tracking of the movement of the human body by tracking the position of an object fixed to the human body. If the first position Pa and the second position Pb coincide with each other and the first object and / or the second object can be determined as the human body, the coordinates of the second target zone Z2 shown in the second depth image The second position Pb can be determined to be the position of the human body with respect to the plane and the movement of the human body can be tracked by calculating the movement of the second position Pb at an arbitrary time.

The figure shows that the moving direction and speed of the human body can be calculated by calculating the change between the previously determined second position Pp and the currently determined second position Pc.

On the other hand, the gesture of the human body at every moment can be judged by referring to the skeletal model constructed from the first depth image, and the continuous gesture can be identified by comparing the previous gesture with the current gesture.

According to the human body tracking system and method using the two depth images, the object and the human body are firstly classified by the method of constructing the skeleton model by the NiTE algorithm, and the depth of the head shape and the shoulder By detecting the object and the human body in the second way, it becomes possible to distinguish between the object and the human body which are complementary and accurate.

In addition, when a skeleton model is constructed from an image obliquely viewed from a target area, the skeleton may not be configured accurately depending on the direction, posture, or position of the human body. That is, there is a possibility that it can not be accurately determined by the human body.

Likewise, when head and shoulder are detected from the image viewed from the target area vertically, it may not be possible to distinguish between the head and the shoulder according to the direction, posture or position of the human body.

Therefore, in step S170, when the first position and the second position are not at the same position, that is, in the case where the first object is determined as a human body, or in all cases where the second object is determined as a human body, And then perform the following processing.

For example, the human tracking system using two depth images according to the present invention can be installed in a place where there is a possibility of a collision and a safety accident at the time of operation of objects and human bodies in factories and manufacturing industries, It is implemented to continuously monitor the positional change of the detected object and the approach of other objects when an object expected to be the human body is detected (all cases where the human body is determined from the first depth image or the human body is determined from the second depth image) There is a need, and it can meet that need.

The embodiments of the present invention described above are merely illustrative of the technical idea of the present invention, and the scope of protection of the present invention should be interpreted according to the claims. It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the essential characteristics thereof, It is to be understood that the invention is not limited thereto.

Claims (11)

A first depth sensor for obliquely viewing a predetermined target area from a side;
A second depth sensor for looking down the target area vertically;
(1) detecting a first object from a first depth image generated by the first depth sensor, analyzing a shape of the first object detected by a specific algorithm, and determining, when a skeleton structure of the human body is obtained, (2) determining a position of the first object determined as a human body in the target area by determining a position of the first object (Pa); (2) The second object is detected from the second depth image, and when the detected second object includes a hemispherical head shape and a shoulder shape extending from the head shape, (3) determining whether the first position Pa and the second position Pb are the same by comparing the first position Pa and the second position Pb with each other; The first object and the second water Includes; the analyzing device that performs the function of a fixed body

Comparing the first position Pa and the second position Pb with each other,
The target region is defined as a rectangular shape having four vertexes (A2, B2, C2, D2) at the second depth image,
(A2, b2, c2, d2) for each side of the square shape of the second position (Pb) is defined as the second coordinate plane and the square shape is defined as a second coordinate plane, and a percentage (Tx, 0), b2 (1, ty), c2 (tx, 1), d2 (0, ty)
(A1, B1, C1, D1) corresponding to each of the four vertexes of the target region in the first coordinate plane, thereby forming a transformed target region (X1, y1), B1 (x2, y2), C1 (x3, y3), D1 (x4, y4)) of the set vertexes,
(A1, b1, c1, d1) at each side of the deformed target zone to determine the corresponding coordinates, using the equation (1)
(1)

The formula of the straight line L1 connecting the intersection a1 and the intersection c1 and the formula of the straight line L2 connecting the intersection b1 and the intersection d1 are obtained by using the equation (2)
(2)

The intersection Pc of the straight lines L1 and L2 is determined using the following equation (3)
(3)

Determining coordinates (xa, ya) of the first position (Pa) in the first coordinate plane,
(Pa) and the coordinates (xa, ya) of the first position (Pa) are equal to each other or have an error in a predetermined range, the first position (Pa) And the second position (Pb) is determined to be the same position. delete The method according to claim 1,
Characterized in that the algorithm comprises processing by a NiTE library. delete The method according to claim 1,
Wherein the second depth sensor comprises:
An offset is set so as to detect a shape detected within a predetermined reference distance,
Wherein the reference distance is set to a distance that can best detect the head and shoulders of the human body that can enter the target area. The method according to claim 1,
Wherein the second depth sensor comprises:
A first line image is generated by scanning forward in a linear form along a first axis direction and a second line image is generated along a first axis direction at a position moved in a second axial direction, To scan,
Wherein the analyzer performs object detection and human determination using one or more of the line images. The method according to claim 1,
The analyzing apparatus, when the first object and the second object are determined as the human body,
A function of tracking the movement of the defined human body in the target area using at least one of the first position and the second position, and
And further performs a function of tracking the determined gesture of the human body using the skeleton structure. delete Obtaining a first depth image by a first depth sensor that obliquely views a predetermined target area at a side view;
When the first object is detected from the first depth image, the shape of the first object detected is analyzed by a specific algorithm to determine the first object as a human body when the skeleton structure of the human body is acquired, Determining a position of the first object in the target area and setting it to a first position (Pa);
Obtaining a second depth image by a second depth sensor viewing the target area vertically;
Determining that the second object is a human body when the detected second object includes a hemispherical head shape and a shoulder shape extending from the head shape when the second object is detected from the second depth image; Determining a position of the second object within the target area and setting it to a second position (Pb);
Determining the first object and the second object as a human body when the first position Pa and the second position Pb are compared with each other to determine the same position,

Comparing the first position Pa and the second position Pb with each other,
The target region is defined as a rectangular shape having four vertexes (A2, B2, C2, D2) at the second depth image,
(A2, b2, c2, d2) for each side of the square shape of the second position (Pb) is defined as the second coordinate plane and the square shape is defined as a second coordinate plane, and a percentage (Tx, 0), b2 (1, ty), c2 (tx, 1), d2 (0, ty)
(A1, B1, C1, D1) corresponding to each of the four vertexes of the target region in the first coordinate plane, thereby forming a transformed target region (X1, y1), B1 (x2, y2), C1 (x3, y3), D1 (x4, y4)) of the set vertexes,
(A1, b1, c1, d1) at each side of the deformed target zone to determine the corresponding coordinates, using the equation (1)
(1)

The formula of the straight line L1 connecting the intersection a1 and the intersection c1 and the formula of the straight line L2 connecting the intersection b1 and the intersection d1 are obtained by using the equation (2)
(2)

The intersection Pc of the straight lines L1 and L2 is determined using the following equation (3)
(3)

Determining coordinates (xa, ya) of the first position (Pa) in the first coordinate plane,
(Pa) and the coordinates (xa, ya) of the first position (Pa) are equal to each other or have an error in a predetermined range, the first position (Pa) Further comprising determining the second position (Pb) to be the same position. delete delete

Images