JP2020057424A - Device and method for tracking mobile object and program - Google Patents

Abstract

To provide a method and device for tracking a mobile object, and a program each of which allows for tracking of a plurality of mobile objects with little calculation cost even if the mobile objects move complexly.SOLUTION: A device for tracking an object comprises an acquisition unit, a detection unit, an extraction unit, and a coordination unit. The acquisition unit acquires a plurality of frames. The detection unit detects an object form the plurality of frames. The extraction unit extracts, with respect to the detected object, a first movement trajectory including a first frame, a second movement trajectory composed of only frames preceding the first frame, and a third movement trajectory composed of only frames after the first frame. The coordination unit makes the third and second movement trajectories correspond to each other which satisfy a condition that similarity between the first and third movement trajectories is greater than or equal to similarity between the second and other third movement trajectories, and that similarity between the first and third movement trajectories is greater than or equal to similarity between the first and third movement trajectories.SELECTED DRAWING: Figure 1

Description

  Embodiments of the present invention relate to a moving object tracking device, method, and program.

In the time series of images, multiple moving objects included in multiple frames are detected, and the same objects are associated with each other between the frames to track the moving objects, record the results of the tracking, and record the results of the tracking. A moving object tracking system for identifying a moving object based on the moving object is disclosed.

However, calculation costs increase because a combination of branches (paths) whose appearance, disappearance, and detection failure of a face detected in a plurality of time-series images are nodes is considered.

JP 2011-170711 A

The problem to be solved by the present invention is to provide a moving object tracking system and method with low calculation cost even for complicated movement of a plurality of moving objects.

The object tracking device according to the embodiment includes an acquisition unit, a detection unit, an extraction unit, and an association unit. The acquisition unit acquires a plurality of frames. The detection unit detects an object from the plurality of frames.
For the detected object, the extracted object includes only a first movement trajectory including a first frame, a second movement trajectory including only a frame before the first frame, and only a frame after the first frame. The third moving trajectory to be constructed is extracted. The associating unit may be configured such that a similarity between the second movement trajectory and the third movement trajectory is equal to or greater than a similarity between the second movement trajectory and another third movement trajectory, and the second movement trajectory and the third movement trajectory are similar to each other. The third trajectory and the second trajectory satisfying a condition that the similarity of the trajectory becomes equal to or more than the similarity of the first trajectory and the third trajectory are associated.

FIG. 1 is a configuration diagram illustrating an example of a moving object tracking system according to a first embodiment. 4 is a flowchart illustrating an example of a moving object tracking system according to the first embodiment. FIG. 4 is an explanatory diagram showing an example of dividing a moving trajectory. FIG. 4 is an explanatory diagram showing an example of calculation using a Euclidean distance. FIG. 2 is a configuration diagram illustrating an example of a moving object tracking system according to a second embodiment. FIG. 9 is a schematic diagram illustrating a shield according to a second embodiment.

  Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

(First embodiment)
FIG. 1 is a configuration diagram illustrating an example of a moving object tracking system 1 according to the first embodiment. As shown in FIG. 1, the acquiring unit 10, the detecting unit 11, the extracting unit 12, the managing unit 13, the setting unit 14, and the dividing unit 15
, An association unit 16, an integration unit 17, and an output unit 18. The moving object tracking system may be realized, for example, by causing a processing device such as a CPU (Central Processing Unit) to execute a program, that is, by a software or by a hardware such as an IC (Integrated Circuit). Alternatively, it may be realized by using software and hardware together. The moving image acquired by the acquiring unit 10 may be one stored in a storage device.

The storage device is, for example, a hard disk drive (HDD) or a solid state drive (SSD).
, A ROM (Read Only Memory), a magnetic card, a magnetic card, an optical storage device, and the like.

FIG. 2 is a flowchart illustrating an example of a flow of a processing procedure of the moving object tracking system 1 according to the first embodiment.

First, the moving object tracking system 1 acquires a moving image captured by the imaging device or a moving image stored in the storage device (step S101). The moving image includes a plurality of frames (images).

Next, the detecting unit 11 detects a plurality of the moving objects from the moving image acquired by the acquiring unit (Step S102). The moving object refers to, for example, a person, a car, and the like. Hereinafter, a case where the moving object is a person will be described as an example. The following method can be applied as a specific processing method for detecting a person.
For example, see the literature (N. Diall, B. Triggs, "Histograms of O.
iented Gradients for Human Detection, "I
EEE Computer Vision and Pattern Recognit
ion, pp. 886-893, 2005) can be applied. Also,
By using the Visual Tracking technology and estimating the position of the same object in the frame being processed by tracking the object detected in the frame immediately before the frame being processed, it is possible to increase the accuracy of detecting a person. . The Visual Tracking technology is described, for example, in the literature (K. Zhang, L. Zhang, MH Yang, “Real-
time compressive Tracking ”, European Conf
eance Computer Vision, pp. 866-879, 2012) can be applied.
Next, the extraction unit 12 associates each person in each frame, and extracts a movement trajectory (tracklet; hereinafter, a movement trajectory is referred to as a tracklet) of the associated person (step S103). As a method of extracting a tracklet, for example, a document (H.P.
Irsiavash, D .; Ramanan, C .; C. Fowlkes ", Global
y-Optimal Greedy Algorithms for Tracking
a Variable Number of Objects ", IEEE Comp
uter Vision and Pattern Reg., p. 1
201-1208, 2012) can be applied.

Next, the management unit 13 manages a tracklet for each person (step S104). The tracklet may manage the time zone in the moving image in which the moving person is included, and may be managed by the frame number of the image, the reproduction or recording time, or the like. Further, the position and size of the person detected by the detection unit 11 may be stored together. These pieces of information may be stored in the storage unit 19. The tracklet for each person may be stored in the storage unit 19 together with the ID of the person. The IDs of the persons need only be given so as to be distinguishable from each other, and may not be able to identify the persons themselves.

Next, the setting unit 14 sets the frame for which the position is calculated as the frame of interest (step S105).

Next, the dividing unit 15 assigns a first block including at least the frame of interest, a second block positioned before the first block in chronological order, and Is divided into a third block located after the first block (step S105). For example, as shown in FIG. 3, when a time t (i-th) frame in an N-frame image is set as a target frame, a block including the i-th frame is a first block, and the block including the i-th frame is more time-series than the first block. A block preceding the first block is referred to as a second block, and a block subsequent to the first block in a chronological order is referred to as a third block. More specifically, Equation 1 below is used.

Where start (t) is the number of the start frame of the tracklet t, and end (t) is the tracklet t
Is the end frame number.

Next, the associating unit 16 acquires the tracklet included in the second block from the managing unit 13,
The acquired tracklets are associated with the tracklets included in the first block and the third block based on the similarity of the tracklets (step S106).

This is executed in two stages, that is, the tracklet selection of the second block and the tracklet selection of the third block.

First, in the tracklet selection of the second block, ap such that end (ap) = i−1 is selected from the second block. i is the frame of interest. More specifically, it is preferable to use a tracklet whose length in the second block is within a predetermined threshold. Tracklets that end before i-1 in chronological order are excluded because they have been processed before this processing flow. In this case, the processing time can be greatly reduced by greatly reducing the number of association candidates.
In the tracklet selection of the third block, each {ap} selected in the previous stage is associated with a tracklet bq satisfying the following expression (2) from the third block.

Here, D (,) indicates the correspondence between two tracklets and is calculated from the similarity of motion between tracklets and the similarity of appearance.

Motion similarity MotionMatch (t1, t2) is t1 in the short time between tracklets t1 and t2.
Suppose that the person takes a linear movement. FIG. 4 shows that t1 is extended to t1 ′ by the start time of t2.
As shown in the above, the calculation is performed using the Euclidean distance.

The appearance similarity AppearanceMatch (t1, t2) is calculated by selecting a representative appearance of a person from each tracklet and extracting and comparing a feature amount from the two appearances. Regarding the selection of a representative appearance, as shown in FIG. 4, a person located between tracklets may be selected. Also, regarding extraction and comparison of feature amounts, see the literature (K. Zhang,
L. Zhang, M .; H. Yang ", Real-time compressive
Tracking ”, European Conference Computer Vi
sion, pp. 866-879, 2012) can be applied.

Here, if there is no tracklet extraction result corresponding to the person detection result in the second block, the same processing as steps S101 to S104 described in FIG. 2 is performed to extract a person and tracklet. Becomes possible.

The integration unit 17 integrates a set of tracklets associated by the association unit 16 as a new tracklet. The management unit 14 manages the detection result of the detection unit 12 and the integrated tracklet of the integration unit 17.

The output unit 18 outputs the result of the associated person and the tracklet. The result of the person's tracklet may be superimposed on the moving image and displayed, or only the result of the desired person's tracklet may be output. It is preferable to superimpose the result of the tracklet of the person, since the complicated trajectory becomes clear to the user. Alternatively, the ID and position of the person may be output only for the frame of interest. As described above, the IDs of the persons need only be given so as to be distinguishable from each other, and the persons themselves may not be able to be identified.

As described above, according to the moving object tracking system 1 according to the first embodiment, it is possible to reduce the calculation cost even for a complicated movement of a plurality of moving objects. In particular, the associating unit 16 uses the tracklets of the second block and integrates them with the tracklets of the first and third blocks, so that there is no need for duplicate calculation in associating the tracklets. In addition, since redundant calculations are not required, calculation costs can be reduced.

(Second embodiment)
FIG. 5 is a block diagram showing a moving object tracking system 2 according to the second embodiment. The moving object tracking system 2 according to the present embodiment includes an acquisition unit 10, a setting unit 11, a detection unit 12, an extraction unit 13, a management unit 14, a division unit 15, a correspondence unit 16, an interpolation unit 20, an integration unit 17, an output unit And a unit 18. The difference from the first embodiment is that an interpolation unit 20 is provided.

The interpolating unit 20 interpolates the position information of the person in the first block from each tracklet of the person associated between the second block and the third block. Here, the position information refers to information including any one of the position of the person in the frame, the size of the person in the frame, and the movement trajectory (tracklet) preceding the frame of interest in chronological order.
Specifically, non-detection of a person in a time series between tracklets associated by the association unit 16 is interpolated. The associated tracklet is (t1, t2), and the position, size, and frame number ([x1, y1], [h1, w1], f1) of the person at the end of t1 are The position of the person,
Size and frame number ([x2, y2], [h2, w2], f2). By setting df = f2-f1, [f1 + 1,…, f1 +
df-1], the position and size ([xs, ys], [hs, ws]) of the person can be estimated by Equation 5 for each frame f1 + s.

The output unit 18 outputs the position of the person in the frame of interest using the association result of the association unit 16 and the estimation result of the interpolation unit 20. Further, the size and the result of association before the frame of interest in chronological order may be output together.

The integrating unit 17 may integrate the associated tracklet and the interpolated person area into a new tracklet using the association result of the association unit 16 and the interpolation result of the interpolation unit 20.

  Here, a specific case where interpolation is necessary will be described with reference to FIG.

FIG. 6 is a schematic diagram illustrating a case where a person is overshadowed by a building (upper part) and a blockage caused by passing of a person. The figure shows a case where the drawing changes from the left drawing to the right along the time axis.

For example, if a person passes each other (lower part in FIG. 6), it is possible to make a correspondence between front and rear based on walking speed and the like. The speed can be estimated by determining whether the moving object is a person or a car and learning the average speed in advance, thereby estimating the occlusion time.

The same applies to the shielding by a building (the upper part in FIG. 6). For example, the presence or absence of a building that blocks a moving object when the moving object is a car or the like, or the presence or absence of a building that blocks a moving object in the vicinity of imaging, or position information By acquiring such information in advance, it is possible to estimate the shielding time.

From the estimated shielding time, an appropriate length of the first block may be set from the assumed shielding time. For example, in a general monitoring video, the length M is set to twice the frame rate (that is, the number of frames in 2 seconds). By setting the shielding time, it is possible to appropriately cope with changes in the environment due to the building, the traffic volume, and the like, and the result of tracking the person can be performed more robustly.

As described above, according to the moving object tracking system 1 according to the second embodiment, it is possible to reduce the calculation cost even for a complicated movement of a plurality of moving objects. In particular, the interpolation makes it possible to output the tracking result in the frame of interest without knowing the entire tracking result.

(Hardware configuration)
The moving object tracking system of the above embodiment includes a control device such as a CPU (Central Processing Unit), a storage device such as a ROM and a RAM, an external storage device such as an HDD and an SSD, a display device such as a display, and an input device such as a mouse and a keyboard. It has an apparatus and an imaging device such as a camera, and can be realized by a hardware configuration using a normal computer.

The program executed by the apparatus of the above embodiment is provided by being incorporated in a ROM or the like in advance.

Further, the program executed by the apparatus of the above embodiment can be read by a computer such as a CD-ROM, a CD-R, a memory card, a DVD, and a flexible disk (FD) in an installable format or an executable format file. It may be provided by being stored in a simple storage medium.

Further, the program executed by the apparatus of the above-described embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Further, a program executed by the device of the above embodiment is
It may be provided or distributed via a network such as the Internet.

The program executed by the apparatus of the above embodiment has a module configuration for realizing the above-described units on a computer. As the actual hardware, for example, the control unit reads a program from an external storage device onto a storage device and executes the program, whereby the above-described units are realized on a computer.

Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying constituent elements in an implementation stage without departing from the scope of the invention. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiment. Further, components of different embodiments may be appropriately combined.

For example, the steps in the flowchart of the above-described embodiment may be executed in a different order and executed simultaneously or in different orders for each execution, as long as they do not violate the nature of the steps.

As described above, according to the moving object tracking system 1 according to the embodiment, it is possible to reduce the calculation cost even for a complicated movement of a plurality of moving objects. In particular, the associating unit 16 uses the tracklets of the second block and integrates them with the tracklets of the first and third blocks, so that there is no need for duplicate calculation in associating the tracklets.

1, 2 ... moving object tracking system, 10 ... acquisition unit, 11 ... detection unit, 12 ...
Extraction unit, 13 management unit, 14 setting unit, 15 division unit, 16 association unit, 17 integration unit, 18 output unit, 19 ... Storage unit, 20 ... interpolation unit

Claims (7)

An acquisition unit that acquires a plurality of frames;
A detection unit that detects an object from the plurality of frames,
For the detected object, a first movement trajectory including a first frame, a second movement trajectory including only frames before the first frame, and a second movement trajectory including only frames subsequent to the first frame. An extraction unit for extracting three trajectories;
The similarity between the second trajectory and the third trajectory is equal to or greater than the similarity between the second trajectory and the other third trajectory, and the similarity between the second trajectory and the third trajectory. And the second moving trajectory and the second moving trajectory satisfying a condition that is equal to or greater than the similarity between the first moving trajectory and the third moving trajectory.
An associating unit for associating with a movement trajectory;
An object tracking device comprising: From the second moving trajectory and the third moving trajectory associated with the associating unit, the first
An interpolation unit that interpolates the position information of the object after a frame,
The object tracking device according to claim 1, further comprising: The object tracking device according to claim 1, wherein the associating unit uses the similarity of an object and the movement of a moving trajectory as the similarity. The object tracking device according to claim 3, wherein the associating unit uses, as the similarity, a product of the similarity of the object and the similarity of the movement of the movement trajectory. The object tracking device according to claim 1, wherein a length of the second movement trajectory associated with the association unit is within a predetermined threshold. Obtaining a plurality of frames;
Detecting an object from the plurality of frames;
For the detected object, a first movement trajectory including a first frame, a second movement trajectory including only frames before the first frame, and a second movement trajectory including only frames subsequent to the first frame. Extracting three trajectories;
The similarity between the second trajectory and the third trajectory is equal to or greater than the similarity between the second trajectory and the other third trajectory, and the similarity between the second trajectory and the third trajectory. And the second moving trajectory and the second moving trajectory satisfying a condition that is equal to or greater than the similarity between the first moving trajectory and the third moving trajectory.
Associating with a movement trajectory;
Object tracking method having Obtaining a plurality of frames;
Detecting an object from the plurality of frames;
For the detected object, a first movement trajectory including a first frame, a second movement trajectory including only frames before the first frame, and a second movement trajectory including only frames subsequent to the first frame. Extracting three trajectories;
The similarity between the second trajectory and the third trajectory is equal to or greater than the similarity between the second trajectory and the other third trajectory, and the similarity between the second trajectory and the third trajectory. And the second moving trajectory and the second moving trajectory satisfying a condition that is equal to or greater than the similarity between the first moving trajectory and the third moving trajectory.
Associating with a movement trajectory;
A program for causing a computer to execute.