JP2017174305A - Object tracking device, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make robust and high-accuracy tracking possible irrespective of interhuman occlusion.SOLUTION: An ID assignment execution unit 40 correlates an object detected in a preceding frame and an object detected in the latest frame on the basis of their similarity and assigns them an ID. An integrated object division unit 40a divides, from an integrated object consisting of a plurality of integrated objects that are integrated by occlusion, an overlapping range whose depth information resembles each object before the occlusion, as an object to be tracked. An object processing unit 40b includes the functions of: correlating each object divided from the integrated object to each object before the integration; correlating each object divided from the integrated object of the preceding frame to each object divided from the integrated object of the latest frame; and correlating each object divided from the integrated object in the preceding frame to each object after occlusion is canceled.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an object tracking apparatus, method and program for tracking a moving object such as a person on a moving image, and more particularly to an object tracking apparatus and method which enable robust and high-precision tracking regardless of inter-person occlusion. And program.

  Non-Patent Document 1 discloses a background subtraction method in which a background image is generated based on statistical characteristics of depth (depth) data, and an object is detected by subtracting the frame image or the background image. Non-Patent Document 2 discloses a method of detecting and tracking an object by determining occlusion based on the presence or absence of overlapping objects.

  Patent Document 1 discloses a technique for tracking an object using only a depth video that cannot specifically identify an object without using an RGB color video that can specifically identify the object.

  In Patent Document 2, the object curve along the contour of each object is compared between the frames, and the objects are associated with each other, so that the movement direction of each object after occlusion cannot be predicted from the movement vector before occlusion. In some cases, a technique that enables accurate tracking of an object is disclosed.

Japanese Patent Application No. 2014-198941 Japanese Patent Application No. 2015-038594

SH Cho, K. Bae, KM Kyung, S. Jeong, and TC Kim, "Background Subtraction Based Object Extraction for Time-of-Flight Sensor," in Proc.IEEE 2nd Global Conf. On Consumer Electronics, pp. 48-89 , 2013. H. Sabirin, H. Sankoh, and S. Naito, "Utilizing attributed graph representation in object detection and tracking for indoor range sensor surveillance cameras," IEICE Transactions on Information and Systems 98 (12), pp. 2299-2307, 2015.

  In the above conventional technique, as shown in FIG. 8, tracking is performed based on the similarity between the objects A and B detected before the occurrence of occlusion and the objects a and b detected after the occurrence. However, if the object is a human body, the posture and orientation may change greatly during the occlusion period, and especially when the occlusion period is long, the image characteristics change greatly before and after that. Although the same object is extracted before and after, both cannot be associated with each other, and there is a technical problem that incorrect association can be performed.

  SUMMARY OF THE INVENTION An object of the present invention is to solve the above technical problem and to provide an object tracking apparatus, method, and program capable of accurately associating each object before and after the occurrence of occlusion regardless of changes in the posture and orientation of each object during the occlusion period Is to provide.

  In order to achieve the above object, the present invention is characterized in that an object tracking apparatus for tracking an object on a frame image in which each pixel has depth information has the following configuration.

  (1) Means for detecting an object on each frame image based on depth information, means for determining whether or not the object detected in the current frame is an integrated object caused by occlusion, and from the integrated object to the previous frame And a means for dividing a portion similar to the object detected in the above as an object to be tracked, and a means for tracking each object based on its similarity between frame images.

  (2) The means for determining whether or not the object is an integrated object is that the ratio of the area of each overlapping range of the object detected in the current frame and the plurality of objects detected in the previous frame is equal to or greater than a predetermined threshold. And the object detected in the current frame is determined as an integrated object.

  (3) The dividing means divides each overlapping range in which the arrangement of depth information is similar to each object detected in the previous frame from the integrated object as a tracking target object.

  According to the present invention, the following effects are achieved.

  (1) Even when the orientation and orientation of an object change greatly before and after the occlusion period, the amount of change is sufficiently small when viewed in units of frames, so the influence on similarity determination is small. Therefore, by dividing the part corresponding to each object from the integrated object during the occlusion period and continuing tracking as each object, it is possible to track the object robustly and accurately regardless of the length of the occlusion period. Become.

  (2) Occlusion occurrence is determined based on the ratio of the area of the overlapping range of the object detected in the current frame and the multiple objects detected in the previous frame, so the number of objects between frames This makes it possible to make a judgment with higher accuracy than the conventional method that judges based on the change.

It is the functional block diagram which showed the structure of the principal part of the object tracking device which concerns on one Embodiment of this invention. It is the figure which showed the detection method of an object. It is the figure which showed the example of a setting of a rectangular area. It is the figure which showed the example of a display of the movement locus | trajectory of an object. It is the flowchart which showed operation | movement of one Embodiment of this invention. It is the figure which showed typically the method of determining an occlusion relationship based on the overlapping area of the rectangular area of an object. It is the figure which represented typically the method of tracking the part divided | segmented from the integrated object as an object. It is a figure for demonstrating the subject of a prior art.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing the configuration of the main part of an object tracking apparatus according to an embodiment of the present invention, and here, the configuration unnecessary for the description of the present invention is omitted.

  As illustrated in FIG. 2, the object detection unit 10 detects each object on each frame image of a video of only depth data (depth video) that cannot specifically identify the object. In the present embodiment, background images taken in an environment where no object exists are accumulated, and a closed region where the difference between each frame of the depth video and the background image is greater than or equal to a predetermined threshold is detected as an object.

  The rectangular area setting unit 20 sets a rectangular area (rectangular area) circumscribing the outline of each object, as shown in FIG. The occlusion determination unit 30 includes an overlapping area calculation unit 30a, and determines the presence / absence of occlusion based on the ratio of the area of the overlapping range in which the detected rectangular regions of the objects overlap, as will be described in detail later.

  The ID allocation execution unit 40 includes an integrated object dividing unit 40a and an object association unit processing unit 40b, and associates an object detected in the previous frame with an object detected in the current frame based on the similarity. Assign the ID of the corresponding object in the previous frame to the object in the frame.

  The integrated object dividing unit 40a divides each overlapping range in which the arrangement of depth information is similar to each object before occlusion as an object to be tracked from an object (integrated object) obtained by integrating a plurality of objects by occlusion.

  When a new occurrence of occlusion is detected, the object associating unit 40b first divides each object divided from the integrated object with each object before integration, and divides from the integrated object of the previous frame during the occlusion period. A second function for associating each object with each object divided from the integrated object of the current frame, and a third function for associating each object divided from the integrated object in the previous frame with each object after occlusion elimination, the same ID for the corresponding object Assign.

  As shown in FIG. 4, the flow line display unit 50 regards objects assigned the same ID between frames as the same object, and displays the movement trajectory of each object on the display. In the present embodiment, each object is represented by a circle symbol mark and displayed in a different color.

  Such an object tracking device can be configured by mounting an application (program) for realizing each function on a general-purpose computer or server. Alternatively, it can be configured as a dedicated machine or a single-function machine in which a part of the application is hardware or ROMized.

  FIG. 5 is a flowchart showing the operation of an embodiment of the object tracking method and the tracking program according to the present invention. In step S1, a depth image having a depth value in pixels is captured in frames. Note that when there are a plurality of cameras that shoot depth videos, a plurality of frame images are captured.

  In step S2, difference data between the frame image and the background image generated in advance is projected onto the 2D coordinate system. At this time, if frame images are acquired from a plurality of cameras, a camera calibration process for generating a projection matrix is executed in advance.

  In step S3, clustering based on the depth value on the 2D coordinates is executed in pixel units. In step S4, each object is detected from the current frame based on the clustering result. In step S5, the similarity between each object detected in the previous frame and each object detected in the current frame is calculated brute force, and objects having higher similarity are associated with each other.

  In step S6, a rectangular area circumscribing each object is set, and the area ratio of the overlapping range between the rectangular areas is calculated. In step S7, it is determined that objects whose overlapping area ratio satisfies a predetermined condition are in an occlusion relationship.

  FIG. 6 is a diagram schematically showing a method for determining the occurrence of occlusion based on the ratio of the overlapping area of the rectangular regions of the respective objects.

  In the present embodiment, the overlapping of the rectangular area between the object detected in the previous frame and the object detected in the current frame is determined. As a result, as shown in FIG. 6A, the overlapping range of the rectangular areas Ka and Kb of the objects A and B detected in the previous frame and the rectangular area Kc of the object C detected in the current frame If both of the area ratios are 50% or more, it is determined that the object C is an integrated object generated by the occlusion of the two objects A and B.

  On the other hand, as shown in FIG. 5B, the rectangular areas Ka and Kb of the objects A and B detected in the previous frame and the rectangular area Kc of the object C detected in the current frame are superimposed. Even if it is detected, the occlusion state is not determined if there is only one combination in which the area ratio of the overlapping range is 50% or more (here, only the combination of objects A and D).

  Further, as shown in FIG. 5C, the ratio of the area of the overlapping range of the rectangular area Ka of the object A detected in the previous frame and the rectangular area Kf of the object F detected in the current frame is 50% or more. However, if there is only one, it is not determined as an occlusion state.

  With respect to the object of the current frame that is determined not to be in the occlusion state by the above occlusion determination, the process proceeds to step S8, and the ID of the object of the previous frame having a high similarity is assigned. On the other hand, for the integrated object determined to be in the occlusion state, the process proceeds to step S9, where the integrated object is divided based on the depth value of each pixel.

  In the present embodiment, if an integrated object is generated by occlusion of n objects, each region from the integrated object is considered based on the consideration that n regions having different distributions of depth values are mixed in the region. Based on the pixel depth value, a plurality of parts are divided into new tracking target objects.

  Here, first, a depth model x represented by the following equations (1) to (4) is generated for each object before integration detected in the previous frame and the overlapping range in the current frame. Here, variable a is a function of width w and height h of the rectangular area circumscribing each object or overlapping range before integration, variable c is the center position (coordinates) of width w and height h, and variable d is a function of the depth value of each pixel.

  Next, in order to determine the similarity between the distribution related to the depth value of each object before integration and the distribution related to the depth value of the overlapping range in the current frame, the Battery distance (Bhattacharyya distance) in the following equation (5) is Calculated.

  Here, σi and μi are the variance and arithmetic mean (median) of the depth values of the overlapping range in the current frame, and σj and μj are the variance and arithmetic mean (median) of the depth values of the objects detected in the previous frame. It is. Then, among the overlapping ranges in the current frame, the overlapping range near the batcher distance B with each pre-integration object detected in the previous frame is divided from the integrated object to be a new tracking target object.

  In step S10, the degree of similarity between each object before integration detected in the previous frame and each object in the current frame is compared brute force. In step S11, the ID of each object before integration having the highest similarity is assigned to each object. In step S12, the flow line (trajectory) of each object is displayed and output.

  FIG. 7 is a diagram schematically showing a tracking method in a period from the occurrence of occlusion of a portion divided from the integrated object to the cancellation thereof.

  In the present embodiment, when the two objects A and B detected at time t0 are integrated into one object C by occlusion, the upper portion Ka similar to each object A and B from the integrated object C at time t1. And the lower part Kb are divided.

  Since each part Ka and Kb should be similar to each object A and B before integration, each part Ka and Kb of object C is assigned the identifier IDa of object A and the identifier IDb of object B, respectively. To be tracked. The integrated object C is excluded from the tracking target.

  At the next time t2, the right part and the left part similar to the objects Ka and Kb corresponding to the parts divided in the previous frame are divided from the integrated object C detected in the current frame, and the identifier of the object Ka is divided into each. IDa and the identifier IDb of the object Kb are assigned to be tracked in the future. The integrated object C is excluded from the tracking target.

  At the next time t3, the lower part and the upper part similar to the objects Ka and Kb corresponding to the parts divided in the previous frame are divided from the integrated object C detected in the current frame, and each of the objects Ka The identifier IDa and the identifier IDb of the object Kb are assigned to be subject to future tracking. The integrated object C is excluded from the tracking target.

  After that, when the occlusion of the integrated object C is resolved and the objects A and B are detected independently, at time t4, the object Ka corresponding to the lower part and the object Kb corresponding to the upper part divided in the previous frame. Are associated with the objects A and B, respectively, so that it is possible to correctly assign the identifier IDa to the object A and the identifier IDb to the object B.

  The object tracking is executed based on a particle filter using the depth model x. In the present embodiment, as shown in the following equation (6), an object that is not in the occlusion state of the previous frame is set as a weighted standard sample.

  The weight element Δ of the depth similarity model is given by the following equation (7) using the observation target frames Z0 to Zt-1.

  Here, as shown in the following equation (8), a case will be described in which object tracking is performed by observing a maximum of five previous frames.

  D is given by the following equation (9), and is a measurement result of the similarity of the depth model calculated based on the above equation (1) between the current object and the object of the reference frame. Similarity is calculated based on the Euclidean distance of the center of gravity and the depth of each object.

Here, constants α = 0.5 and β = 0.1 are determined in order to give a high score to the center of gravity so that a pair of objects that are separated in tracking are not erroneously associated with each other. The variance σ of the depth value of the current object is called the depth value component in the similarity measurement so that it is within the distance between the detected objects. Thus, tracking is to find the object in the current frame t that has the smallest distance from the center of gravity of the object in the t _AVG frame.

Here, I is the number of objects detected in the reference frame (average over 5 frames). Therefore, the label L of the object for which c gives the minimum value in the frame t _AVG is assigned to the jth object of the frame t.

  As described above, the object tracking device according to the embodiment of the present invention has been described, but the present invention is not limited thereto, and may be an object tracking method that performs the same processing as the object tracking device, An object tracking program for causing a computer to execute the tracking method may be used.

  DESCRIPTION OF SYMBOLS 10 ... Object detection part, 20 ... Rectangular area setting part, 30 ... Occlusion determination part, 30a ... Area calculation part, 40 ... ID allocation execution part, 40a ... Integrated object division part, 40b ... Object matching part, 50 ... Flow line Display section

Claims (11)

In an object tracking device that tracks an object on a frame image in which each pixel has depth information,
Means for detecting an object on each frame image based on depth information;
Means for determining whether the object detected in the current frame is an integrated object caused by occlusion;
Means for dividing a portion similar to the object of the previous frame from the integrated object as an object to be tracked,
An object tracking device comprising means for tracking each object between frame images based on the degree of similarity.   The means for determining whether or not the object is an integrated object is that the ratio of the area of each overlapping range of the object detected in the current frame and the plurality of objects detected in the previous frame is equal to or greater than a predetermined threshold. The object tracking device according to claim 1, wherein the object detected in the current frame is determined as an integrated object.   The object tracking device according to claim 1, wherein the dividing unit divides each overlapping range in which the arrangement of depth information is similar to each object of the previous frame from the integrated object as a tracking target object. .   The object tracking according to claim 3, wherein the similarity of the arrangement of the depth information is determined based on a virtual distance between the depth information of each object in the previous frame and the depth information of each overlapping range in the current frame. apparatus.   5. The object tracking device according to claim 1, wherein the tracking unit calculates the similarity of each object between frame images based on a particle filter. In an object tracking method in which a computer tracks an object on a frame image in which each pixel has depth information,
A procedure for detecting an object on each frame image based on the depth information;
A procedure for determining whether or not the object detected in the current frame is an integrated object caused by occlusion;
Procedure to divide the part similar to the object of the previous frame from the integrated object as the object to be tracked,
And a procedure for tracking each object between frame images based on the similarity.   In the procedure for determining whether or not the object is an integrated object, if the ratio of the area of each overlapping range of the object detected in the current frame and the plurality of objects detected in the previous frame is equal to or greater than a predetermined threshold, The object tracking method according to claim 6, wherein the object detected in the current frame is determined as an integrated object.   The object tracking method according to claim 6 or 7, wherein in the dividing step, each overlapping range in which the arrangement of the depth information is similar to each object of the previous frame is divided as an object to be tracked from the integrated object. . In an object tracking program that causes a computer to track an object on a frame image in which each pixel has depth information,
A procedure for detecting an object on each frame image based on the depth information;
A procedure for determining whether or not the object detected in the current frame is an integrated object caused by occlusion;
Procedure to divide the part similar to the object of the previous frame from the integrated object as the object to be tracked,
An object tracking program for causing a computer to execute a procedure for tracking each object between frame images based on the degree of similarity.   In the procedure for determining whether or not the object is an integrated object, if the ratio of the area of each overlapping range of the object detected in the current frame and the plurality of objects detected in the previous frame is equal to or greater than a predetermined threshold, The object tracking program according to claim 9, wherein the object detected in the current frame is determined as an integrated object.   11. The object tracking program according to claim 9, wherein, in the dividing step, each overlapping range in which an arrangement of depth information is similar to each object of the previous frame is divided as a tracking target object from the integrated object. .