JP6336872B2 - Object tracking method, apparatus and program - Google Patents

Description

  The present invention relates to an object tracking method, apparatus, and program for tracking a dynamic object typified by a person on a video, and particularly to a depth in which an object cannot be individually recognized without using an RGB color image that can individually recognize the object. The present invention relates to an object tracking method, apparatus, and program for tracking a dynamic object using only video of data.

  Many methods related to object tracking for surveillance camera images and sports images have been proposed. Most of them perform tracking based on video color information on the assumption that certain shooting conditions are satisfied.

  On the other hand, in crime prevention and marketing applications, the demand for human detection / tracking using surveillance camera images is rapidly increasing, but there are many cases where it is difficult to use color information from the viewpoint of insufficient resolution and lighting conditions. In addition, from the viewpoint of privacy, the use of color information that can specify a person is limited, and instead, there is an increasing expectation for a technology that performs object tracking only from a depth video in which a person cannot be specified.

  Non-Patent Document 1 proposes a method of tracking based on detection of an object area in depth data using a depth sensor typified by Microsoft Kinect. Non-Patent Document 2 discloses a technique for realizing improvement in tracking performance by combining color information and depth data.

Parvizi and Wu, "Multiple Bject Tracking Based on Adaptive Depth Segmentation," Canadian conference on Computer and RBot Vision 2008, pp.273-277, 2008 Tran and Harada, "Depth-Aided Tracking Multiple Bjects under Cclusion," Journal of Signal and Information prCessing, vol. 4, pp.299-307, 2013

  In Non-Patent Document 1, an object tracking method using only depth data is proposed, but there is no mention of tracking at the time of occlusion in which objects overlap each other, and performance degradation is expected in terms of actual use. Non-Patent Document 2 is a combination method with color information and does not correspond to a use scene in which only depth data is used.

  An object of the present invention is to provide an object tracking method, apparatus, and program capable of solving the above technical problem and accurately tracking an object with or without occlusion while respecting the privacy of the object by using only the depth image. There is to do.

  In order to achieve the above object, the present invention is characterized in that an object tracking apparatus for tracking an object on a depth image has the following configuration.

  (1) A means for detecting an object from each frame of a depth video, a position assignment, a size, and a motion vector of each object are compared between frames, an ID assigning means for assigning the same ID to the same object, and an occlusion between objects is determined. The ID assigning means assigns the same ID to each object for which occlusion has been resolved based on the position, size, and motion vector of each object before and after the occurrence of occlusion.

  (2) A means for setting a rectangular object area circumscribing each object is provided, and the means for determining occlusion determines that there is a possibility that occlusion occurs between objects in which at least part of the object area overlaps. Accumulated object position, size and motion vector.

According to the present invention, the following effects are achieved.
(1) Without using RGB color images that can specifically identify objects, it is possible to track objects using only depth images that cannot specifically identify objects, enabling object tracking that respects the privacy of each object. Become.

  (2) Since the identity of each object before and after occlusion is determined based on the similarity calculated from the position, size and motion vector of each object, accurate tracking of the object regardless of the occurrence of occlusion Is possible.

  (3) Presence of occlusion is predicted based on the presence or absence of overlapping rectangular occlusion areas circumscribing objects before objects overlap and cannot be recognized. Can be accurately identified.

It is a functional block diagram of the object tracking device concerning one embodiment of the present invention. It is the figure which showed an example of the depth image | video. It is the figure which showed the example of a setting of a rectangular object area | region. It is the figure which showed the determination method of occlusion. It is the figure which showed an example of the object tracking image | video. It is a figure for demonstrating the object tracking method. It is the flowchart which showed operation | movement of one Embodiment of this invention. It is the flowchart which showed operation | movement of ID allocation processing.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing the configuration of the main part of an object tracking apparatus according to an embodiment of the present invention, and here, the configuration unnecessary for the description of the present invention is omitted.

  As illustrated in FIG. 2, the object detection unit 10 detects each object on each video frame of a video of only depth data (depth video) that cannot specifically identify the object. In the present embodiment, background images taken in an environment where no object exists are accumulated, and a closed region where the difference between each frame of the depth video and the background image is greater than or equal to a predetermined threshold is detected as an object.

  As shown in FIG. 3, the object area setting unit 20 sets a rectangular object area circumscribing or including the outline of each object. The ID allocation processing unit 30 compares the position, size, and motion vector of each object between video frames, and regards objects whose similarity calculated based on the object information exceeds a predetermined threshold as the same object. Assign the same ID. The position of each object is represented by the center coordinates of the object, for example.

  In the occlusion processing unit 40, the occlusion determination unit 41 determines that there is a possibility that occlusion may occur between the objects A and B in which at least part of the object areas Ka and Kb overlap as shown in FIG. Then, an ID unique to each object is registered in the occlusion list 42, and an occlusion flag (Foc) 43 is set.

  In the ID allocation processing unit 30, the occlusion occurrence allocation unit 31 includes an unallocated object that has not been allocated the same ID as the previous frame in the current frame, and the number of objects decreases from the previous frame to the current frame, If the current position of the unassigned object is within the prediction error range from the current position predicted based on the position and motion vector of the objects A and B registered in the occlusion list 42, the occlusion is performed between the objects A and B. Is determined to have occurred. Then, for example, IDs of the objects A and B are assigned to the unassigned objects as IDs unique to the objects determined to be occluded.

  The allocation unit 32 at the time of occlusion cancellation includes an unallocated object that could not be assigned the same ID as the previous frame in the current frame, the number of objects increases from the previous frame to the current frame, and the size of the unallocated object is Similar to objects A and B registered in the occlusion list, the current position of the unassigned object is predicted based on the position and motion vector of the objects A and B registered in the occlusion list. If it is within the range, it is determined that the occlusion between the objects A and B has been resolved. Then, the IDs of the objects A and B registered in the occlusion list are assigned to the respective unassigned objects.

  As shown in FIG. 5, the flow line display unit 50 regards objects assigned the same ID between frames as the same object, and displays the movement trajectory of each object on the display. In the present embodiment, each object is represented by a circle symbol mark and displayed in a different color.

  Such an object tracking device can be configured by mounting an application (program) for realizing each function on a general-purpose computer or server. Alternatively, it can be configured as a dedicated machine or a single-function machine in which a part of the application is hardware or ROMized.

  Next, as shown in FIG. 6, after three objects A, B, and C are detected [(a) in FIG. 6], IDa, IDb, and IDc are assigned as object IDs, respectively. When the occlusion occurs in [Fig. (B)], [Fig. (C)], and then the occlusion disappears, the object tracking operation of [Fig. (D)], [Fig. , 8 will be described.

  First, referring to the flowchart of FIG. 7, in the step S1 of the nth cycle, the depth video frame shown in FIG. 6A is captured. In step S2, three objects A, B, and C are detected by the object detection unit 10. In step S3, rectangular object areas Ka, Kb, Kc circumscribing the objects A, B, C are set by the object area setting unit 20, respectively. In step S4, IDa, IDb, and IDc are assigned to objects A, B, and C as object IDs, respectively.

  In step S1 of the subsequent (n + 1) period, the depth video frame shown in FIG. 6B is captured. In this frame, the two objects A and B are close to each other, but since the objects have not yet overlapped, the three objects A, B, and C are detected in step S2.

  In step S3, rectangular object areas Ka, Kb, and Kc are set for the objects A, B, and C, respectively. In step S4, the ID allocation processing unit 30 executes ID allocation processing for allocating a tracking object ID that is consistent throughout all frames to the objects A, B, and C detected in step S2.

  FIG. 8 is a flowchart showing the operation of the ID assignment process. In step S101, it is determined whether or not it is the first frame. If it is the first frame, the process proceeds to step S111, and a new object ID is assigned to all the objects detected in step S2.

  On the other hand, if it is determined that it is not the first video frame, the process proceeds to step S102, where each object detected in the previous video frame and each object detected in the current video frame are separated by the distance between objects ( Position), the object size, and the similarity calculated based on the motion vector of the object. In the present embodiment, corresponding objects IDa, IDb, and IDc are assigned to the objects A, B, and C, respectively.

  In step S103, the object information regarding the objects A, B, and C for which the corresponding ID has been successfully allocated is updated. In the present embodiment, the current position of each object is updated, and its motion vector is updated based on the amount and direction of movement of each object A, B, C from the previous frame to the current frame. In step S104, it is determined whether or not there is an unassigned object to which an ID has not been assigned, and since it is determined that there is no object here, the process ends.

  Returning to FIG. 7, in step S5, it is determined whether or not there is occlusion. Here, a part of the rectangular object area Ka of the object A and a part of the rectangular object area Kb of the object B are overlapped. It is determined that there is a possibility of occurrence of occlusion in the middle, and the process proceeds to step S6.

  In step S 6, object IDs (IDa, IDb) of the objects A and B are registered in the occlusion list 42. In step S7, the occlusion flag Foc is set. In step S8, the flow lines (trajectories) of the objects A, B, and C are displayed and output.

  In this way, in this embodiment, even if the objects themselves do not overlap, it is determined that there is a possibility that occlusion occurs if part of the object area overlaps. Even if objects overlap and cannot be identified, information immediately before the occlusion of each object can be acquired.

  In step S1 of the subsequent (n + 2) period, the (n + 2) video frame as shown in FIG. 6C is captured. In this video frame, the objects A and B overlap and appear to be integrated, and each object cannot be individually identified. Accordingly, two objects C and D (= A + B) are detected in step S2. In step S3, rectangular object areas Kc and Ka + b are set for the objects C and D, respectively. In step S4, an ID assignment process for each object C, D is executed.

  Proceeding to FIG. 8, if it is determined in step S101 that it is not the first video frame, the process proceeds to step S102, and the degree of similarity between each object detected in the previous video frame and each object detected in the current video frame. Are calculated based on the position of each object, the object size, and the motion vector of the object, and the same ID is assigned to an object having a high degree of similarity.

  Here, IDc is assigned to the object C, but no ID is assigned to the object D. Therefore, the process proceeds from step S104 to step S105, and it is determined based on the occlusion flag Foc whether occlusion is in progress. Here, since it is determined that occlusion is in progress, the process proceeds to step S106.

  In step S106, it is determined whether or not the number of objects is decreasing from the previous frame to the current frame. Here, since it is determined that the number of objects has decreased from “3” to “2”, the process proceeds to step S107, and the ID allocation to the unallocated object D is performed by the allocation unit 31 when the occlusion occurs.

  In the present embodiment, it is determined whether or not the current position of the unassigned object D is within the prediction error range from the current position predicted based on the motion vectors of the objects A and B registered in the occlusion list 42. The As a result, if it is within the prediction error range, it is determined that the unassigned object D is an object in occlusion in which the objects A and B are integrated, and a unique ID is assigned.

  On the other hand, if the current position of the unassigned object D is outside the prediction error range, a new object ID is assigned. In step S109, the object information of objects A and B is updated.

  In step S1 of the subsequent (n + 3) period, the video frame shown in FIG. 6D is captured. In this video frame, the occlusion between the objects A and B has begun to be eliminated, and in step S2, the objects A and B can be individually recognized, so that the three objects A, B and C are detected again. In step S3, rectangular object areas Ka, Kb, and Kc are set for the objects A, B, and C, respectively. In step S4, ID assignment processing for each object A, B, C is executed.

  In step S101 in FIG. 8, since it is determined that the current video frame is not the first video frame, the process proceeds to step S102, and the objects C and D detected in the previous video frame and the objects A, B, detected in the current video frame are detected. The similarity with C is calculated based on the position of each object, the object size, and the motion vector of the object, and the same ID is assigned to an object with a high similarity.

  Here, the object IDc is assigned to the object C. However, for the objects A and B, since it is determined in step S104 that the corresponding object has not been detected in the previous video frame, the process proceeds to step S105.

  In step S105, since it is determined that the occlusion flag Foc is set, the process proceeds to step S106. Here, it is determined that the number of objects has increased from “2” to “3”, and therefore, from step S108 to step S110. The ID allocation to the unallocated object is performed by the occlusion resolution allocation unit 32.

  In this embodiment, the size of the unassigned object is registered in the occlusion list and is similar to the objects A and B, and the current position of the unassigned object is registered in the occlusion list and is based on the motion vectors of the objects A and B. It is determined whether or not the predicted current position is within the prediction error range.

  If it is within the prediction error range, the object ID (IDa, IDb) registered in the occlusion list is assigned to each corresponding unassigned object. In the present embodiment, object IDa is assigned to object A, and object IDb is assigned to object B. If the current position of the unassigned object is outside the prediction error range, a new object ID is assigned.

  According to the present embodiment, an object that respects the privacy of each object can be tracked without using an RGB color image that can specifically identify the object, and can be tracked using only a depth image that cannot specifically identify the object. Tracking becomes possible.

  In addition, according to the present embodiment, the identity of each object before and after the occlusion is determined based on the similarity calculated from the position, size, and motion vector of each object. This enables accurate tracking of objects.

  Furthermore, according to the present embodiment, the presence / absence of occlusion is predicted based on the presence / absence of overlap of the rectangular occlusion areas circumscribing the objects before the objects overlap and cannot be recognized. Each object in the occlusion relationship can be accurately identified.

  DESCRIPTION OF SYMBOLS 10 ... Object detection part, 20 ... Object area | region setting part, 30 ... ID allocation process part, 31 ... Occlusion occurrence allocation part, 32 ... Occlusion cancellation allocation part, 40 ... Occlusion process part, 50 ... Flow line display part

Claims (4)

In an object tracking device that tracks an object on a depth image,
Means for detecting an object from each frame of the depth image;
ID allocation means for comparing the position, size and motion vector of each object between frames and assigning the same ID to the same object;
Means for setting a rectangular object area circumscribing each object;
Means for determining that there is a possibility that occlusion occurs between objects that overlap at least part of the object region, and registering the position, size, and motion vector of the object in the occlusion list ,
The object assigning device, wherein the ID assigning means assigns the same ID to each object for which occlusion has been resolved based on the position, size and motion vector of each object after the occurrence of the occlusion list and occlusion. The ID assigning means is
Means for determining whether there is an unassigned object that cannot be assigned the same ID as the previous frame in the current frame;
There is an unallocated object in the current frame, the number of objects decreases from the previous frame to the current frame, and the current position of the unallocated object is predicted based on the motion vector of the object that is determined to be occluded. Means for assigning a unique ID to the unassigned object if it is within the range;
There is an unallocated object in the current frame, the number of objects increases from the previous frame to the current frame, and the size of the unallocated object is similar to the object that has been occluded, and the current position of the unallocated object is determined as occlusion. 2. An apparatus according to claim 1 , further comprising means for assigning the ID of the object determined to be occluded to the unassigned object if it is within a prediction error range from the current position predicted based on the motion vector of the object. Object tracking device. In an object tracking method in which a computer tracks an object on a depth image,
The procedure to detect an object from each frame of depth video,
A procedure for comparing the position, size and motion vector of each object between frames and assigning the same ID to the same object ,
To set a rectangular object area that circumscribes each object,
Determining that there is a possibility that occlusion may occur between objects in which at least part of the object region overlaps, and registering the position, size, and motion vector of the object in the occlusion list ,
In the procedure of assigning the ID, an object tracking method characterized in that the same ID is assigned to each object whose occlusion has been resolved based on the position, size and motion vector of each object after the occurrence of the occlusion list and occlusion. . In an object tracking program that causes a computer to track an object on a depth image,
The procedure to detect an object from each frame of depth video,
A procedure for comparing the position, size and motion vector of each object between frames and assigning the same ID to the same object ,
To set a rectangular object area that circumscribes each object,
Determining that there is a possibility that occlusion may occur between objects in which at least part of the object region overlaps, and registering the position, size, and motion vector of the object in the occlusion list ,
In the procedure of assigning the ID, an object tracking program characterized in that the same ID is assigned to each object whose occlusion has been resolved based on the position, size and motion vector of each object after the occurrence of the occlusion list and occlusion. .