JP2021149686A - Device, method, and program for tracking mobile object - Google Patents

Abstract

To provide a mobile object tracking device that can prevent delay of tracking and reduction of accuracy even when the congestion of mobile objects are heavy.SOLUTION: The mobile object tracking device includes: congestion level estimation means 50 for estimating the congestion level of mobile objects taken in an image; candidate position number allocation means 51 for allocating the candidate position number of each mobile object according to the congestion level; candidate position setting means 52 for determining the same number of candidate positions of the mobile objects at the current time as the number of candidate position numbers allocated to each mobile object, from the positions of the mobile objects in the past; and object position determination means 53 for obtaining the object position of the mobile objects at the current time on the basis of a plurality of candidate positions of each mobile object.SELECTED DRAWING: Figure 2

Description

The present invention relates to a technique for tracking a moving object based on captured images taken sequentially.

For the purpose of security, etc., moving objects such as people are tracked by analyzing the images taken sequentially by the camera, and a particle filter is known as one of the methods used in the tracking process. There is. According to the particle filter, a large number of hypotheses representing the candidates (candidate positions) of the moving destination of each moving object being tracked are set, and the characteristics of the moving object appear at each candidate position in the captured image based on the degree of appearance. Determine the position of the moving object.

For example, Patent Document 1 describes a moving object tracking device that sets a candidate position for each part constituting a moving object. In this moving object tracking device, the total number of candidate positions to be set for each moving object is predetermined, and the number of candidate positions to be assigned to each part is determined according to the past tracking reliability of the part to change the posture. By assigning the hypothesis of the part that is difficult to track due to occlusion to other parts, the hypothesis that is effective for tracking is prevented from decreasing.

Japanese Unexamined Patent Publication No. 2016-17603

However, in the prior art, the number of candidate positions per moving object is a fixed value, so when applied to tracking in an area where many people gather, such as an event venue, if the degree of congestion increases, one captured image will be used. There is a problem that the number of candidate positions of the above increases too much, causing a large delay in the tracking process.

Further, in order to prevent delay, it is conceivable to determine the number of candidate positions per captured image and divide the number equally by the moving objects being tracked. However, in that case, when the degree of congestion increases, the number of candidate positions per moving object becomes too small, and there is a problem that the tracking accuracy deteriorates.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a moving object tracking technique capable of suppressing a delay in tracking processing and a decrease in accuracy even when the degree of congestion of moving objects becomes high.

(1) The moving object tracking device according to the present invention is a moving object tracking device that tracks one or a plurality of moving objects based on sequentially captured images, and is crowded with the moving objects captured in the captured images. Congestion degree estimation means for estimating the degree, candidate position number allocation means for allocating the number of candidate positions of each moving object according to the congestion degree, and the same number of candidate positions as the number of candidate positions assigned to each moving object in the past A candidate position setting means for obtaining a candidate position of the moving object at the current time from the position of the moving object, and an object for obtaining an object position of the moving object at the current time based on the plurality of candidate positions in each moving object. It is provided with a position determination means.

(2) In the moving object tracking device according to the present invention described in (1) above, the congestion degree estimation means outputs the congestion degree at an arbitrary position in the photographed image when the photographed image is input. The captured image is input to the estimator learned in advance to estimate the degree of congestion at an arbitrary position in the captured image, and the candidate position number allocation means is the object position of the moving object on the captured image or The number of candidate positions for each moving object is assigned according to the degree of congestion of the candidate positions.

(3) In the moving object tracking device according to the present invention according to the above (1) or (2), the candidate position number allocation means allocates a large number of candidate positions of the moving object as the degree of congestion increases.

(4) In the moving object tracking device according to the present invention described in (3) above, in the candidate position number allocation means, the total sum of the candidate positions of the moving objects to be tracked at one time falls within the upper limit of the total number. The number of candidate positions of each moving object is compressed so as to.

(5) In the moving object tracking device according to the present invention according to the above (1) to (4), the candidate position setting means has a wider range for setting the candidate position of the moving object as the degree of congestion is smaller. do.

(6) In the moving object tracking device according to the present invention described in (1) above, in the region where the degree of congestion is a certain level or higher, the candidate position number allocation means, the candidate position setting means, and the object position determination means are used. The position of the moving object group on the captured images sequentially captured without processing is stored.

According to the present invention, it is possible to suppress a delay in tracking processing and a decrease in accuracy even when the degree of congestion of a moving object becomes high.

It is a block diagram which shows the schematic structure of the moving object tracking apparatus. It is a functional block diagram which shows the outline of the moving object tracking apparatus. It is a schematic diagram which shows the processing by a congestion degree estimation part. It is a figure which shows the function of the number of candidate positions proportional to the degree of congestion. It is a schematic flow chart which shows the overall processing of the moving object tracking apparatus which concerns on embodiment. It is a figure which shows the function of the number of candidate positions according to the degree of congestion in a modification.

Hereinafter, embodiments of the present invention (hereinafter referred to as embodiments) will be described with reference to the drawings. The present embodiment is a moving object tracking device in which a photographing unit and a display unit are connected to a computer, and the computer analyzes captured images sequentially captured by the photographing unit to track a person who is a moving object.

The moving object tracking device 1 shown as an example in the present embodiment obtains a candidate for a position at the current time from a past position for each moving object, and the image feature of the moving object is set at the candidate position in the captured image at the current time. The position at the current time is obtained based on the degree of appearance. Hereinafter, the candidate position of the moving object is referred to as a candidate position. It is assumed that a plurality of candidate positions are set for one time and one moving object, and the number of candidate positions is referred to as the number of candidate positions. Further, the position of a moving object is referred to as an object position. It is assumed that one object position is set for one time and one moving object. That is, the candidate position is a candidate for the object position, and one object position is determined by integrating a plurality of candidate positions. The moving object tracking device 1 is characterized in that the number of candidate positions of each moving object is assigned according to the degree of congestion.

FIG. 1 is a block diagram showing a schematic configuration of a moving object tracking device 1. The moving object tracking device 1 includes a photographing unit 2, a communication unit 3, a storage unit 4, an image processing unit 5, and a display unit 6.

The photographing unit 2 is a so-called surveillance camera. The photographing unit 2 is connected to the image processing unit 5 via the communication unit 3, photographs the monitoring space at predetermined time intervals to generate an image, and sequentially inputs the generated images to the image processing unit 5. For example, the photographing unit 2 is installed on a pole set up at an event venue, which is a monitoring space, with a fixed field of view overlooking a person existing in the monitoring space, and photographs the monitoring space at a time interval of 1/5 second. To generate a color image. The photographing unit 2 may generate a monochrome image instead of the color image.

The communication unit 3 is a communication circuit, one end of which is connected to the image processing unit 5 and the other end of which is connected to the photographing unit 2 and the display unit 6. The communication unit 3 acquires an image from the photographing unit 2 and inputs it to the image processing unit 5. Further, the communication unit 3 outputs the tracking result of the moving object from the image processing unit 5 to the display unit 6.

The photographing unit 2, the communication unit 3, the storage unit 4, the image processing unit 5, and the display unit 6 are appropriately connected in a form according to the installation location of each unit. For example, when the photographing unit 2, the communication unit 3, and the image processing unit 5 are installed remotely, the photographing unit 2 and the communication unit 3 can be connected by an internet line. Further, the communication unit 3 and the image processing unit 5 can be connected by a bus. In addition, a LAN (Local Area Network), various cables, or the like can be used as the connection means.

The storage unit 4 is a memory device such as a ROM (Read Only Memory) and a RAM (Random Access Memory), and stores various programs and various data. For example, the storage unit 4 stores learning data and information of an estimator that is a trained model, and inputs and outputs such information to and from the image processing unit 5. That is, information used for learning the estimator, information generated in the process of the processing, and the like are input / output between the storage unit 4 and the image processing unit 5.

The image processing unit 5 is composed of arithmetic units such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), an MCU (Micro Control Unit), and a GPU (Graphics Processing Unit). The image processing unit 5 operates as various processing means / control means by reading a program from the storage unit 4 and executing the program, reads various data from the storage unit 4 as necessary, and stores the generated data in the storage unit 4. Remember. For example, the image processing unit 5 learns and generates an estimator, and stores the generated estimator in the storage unit 4 via the communication unit 3.

The display unit 6 is a liquid crystal display, an organic EL (Electro-Luminescence) display, or the like, and displays the tracking result of a moving object input from the image processing unit 5 via the communication unit 3.

FIG. 2 is a schematic functional block diagram of the moving object tracking device 1, in which the storage unit 4 functions as the congestion degree storage means 40 and the object information storage means 41, and the image processing unit 5 has the congestion degree estimation means 50 and the number of candidate positions. It functions as an allocation means 51, a candidate position setting means 52, and an object position determination means 53. Further, the communication unit 3 cooperates with the image processing unit 5 to function as the image acquisition means 30 and the tracking result output means 31.

The image acquisition means 30 acquires captured images sequentially captured from the photographing unit 2 and outputs the captured images to the congestion degree estimating means 50 and the object position determining means 53.

The congestion degree estimation means 50 estimates the congestion degree of a moving object captured in a captured image. The congestion degree estimation means 50 estimates (approximate value) of the congestion degree at the current time by, for example, counting the number of object positions one hour ago for the entire captured image or for each block obtained by dividing the captured image into a plurality of blocks. You may. However, when the degree of congestion is calculated from the tracking result and the degree of congestion is used for tracking, if the tracking accuracy decreases, the estimation accuracy of the degree of congestion based on the tracking result also decreases, and the decrease in tracking accuracy doubles. It ends up. Therefore, it is desirable to estimate the congestion degree by a process independent of tracking so that the error of tracking and congestion degree estimation is not doubled.

That is, preferably, the congestion degree estimation means 50 estimates the congestion degree of a moving object photographed at an arbitrary position on the photographed image at the current time from the photographed image at the current time without being based on the photographed image in the past. In the present embodiment, the congestion degree estimation means 50 inputs the photographed image into the estimator learned in advance so as to output the congestion degree at an arbitrary position in the photographed image when the photographed image is input, and within the photographed image. Estimate the degree of congestion at any position in. Specifically, the congestion degree estimation means 50 inputs a photographed image into an estimator learned in advance to output a congestion degree map that estimates the congestion degree of each pixel when the photographed image is input, and the photographed image. The congestion degree map of the above is output, and the obtained congestion degree map is stored in the congestion degree storage means 40.

The estimator can be specifically realized by using deep learning technology. That is, the estimator can be modeled by a CNN (convolutional neural network) that outputs a congestion degree map of the captured image when the captured image is input. For learning, for example, a large number of learning images taken by a crowd and a probability density function having a variance according to the size of the head, with the position of the center of gravity of each person's head in each of the learning images as the average value. Is set and the value of the function for each head is added for each pixel to prepare a congestion degree map. Then, learning is performed in advance to bring the output when each of the training images is input to the model closer to the congestion degree map corresponding to the image. The trained model thus obtained is stored in the storage unit 4 as an estimator forming a part of the program of the congestion degree estimation means 50. For example, MCNN (multi-column convolutional neural network) described in “Single image crowd counting via multi-column convolutional neural network”, Zhang, Y., Zhou et al., CVPR 2016 is an example of an estimator. The crowd density map described in is an example of a congestion map.

Further, the congestion degree estimating means 50 detects an ultra-high congestion region having a congestion degree of the threshold value T3 or more, and stores the information of the ultra-high congestion region in the congestion degree storage means 40. The ultra-high congestion area occurs when moving objects are densely packed at a position where the depression angle is small (a position as if taken from the side). For example, if the moving object is a human, in an ultra-highly congested area, only the vicinity of the head and shoulders can be seen, and head-to-head occlusion occurs, making tracking difficult. The threshold value T3 is predetermined through a prior experiment.

The congestion degree storage means 40 stores the estimated congestion degree with respect to the captured image. Specifically, the latest predetermined number of congestion degree maps are cyclically stored. FIG. 3 is a schematic diagram showing the processing of the congestion degree estimation means 50. The congestion degree map output with respect to the captured image of FIG. 3 (a) is shown in FIG. 3 (b). Further, FIG. 3C shows the degree of congestion on the straight line 510 of FIG. 3A. Higher values are output in more crowded places, and the degree of congestion at the positions of persons 501, 502, and 503 is estimated to be 0.2, 1.2, and 2.8, respectively.

The object information storage means 41 stores information (object information) of a moving object being tracked. Specifically, the template of the moving object, the object position of the moving object, and the hypothesis of the moving object are stored in association with the object ID that identifies each of the moving objects being tracked. Further, the object information storage means 41 stores a shape model of a moving object.

An example of a template of a moving object is a template image obtained by cutting out an area of a corresponding person from a captured image. Instead of or in combination with the template image, a feature amount such as a color histogram or a luminance gradient calculated from the template image may be used. The hypothesis of a moving object is information in which an object ID, a hypothesis ID, a candidate position, a likelihood, and the like are associated with each other, and is generated as many as the number of combinations of the moving object and the candidate position. The object position and candidate position of the moving object are represented by a set of x-coordinates and y-coordinates (x, y), with the horizontal and vertical directions of the captured image as the x-axis and the y-axis, respectively. An example of a shape model of a moving object is graphic data of an ellipse that approximates a standing person. Instead of the ellipse, it may be a rectangle, or a figure in which three ellipses that approximate the head, body, and legs are connected. Preferably, the shape model is composed of reference graphic data and data in which the position and the size of the graphic data are associated with each other in correspondence with the difference in the size of the image of a person at each position in the captured image.

The candidate position number allocation means 51 allocates the number of candidate positions to each moving object according to the degree of congestion. As an example, the candidate position number allocating means 51 allocates a large number of candidate positions of moving objects as the degree of congestion increases.

Basically, increasing the number of candidate positions increases the possibility that candidate positions with a small deviation (positional deviation) from the true object position will be included, so comparisons between moving objects can be tightened and tracking accuracy can be improved. Can be done. For example, template matching that deviates from the true object position is not a strict comparison between the features of moving objects because the comparison with the features of the background is mixed.

However, the processing load increases as the number of candidate positions increases. As the degree of congestion increases, the distance between moving objects becomes shorter and the number of moving objects in close proximity increases, so that the tracking accuracy tends to decrease due to misalignment. On the other hand, as the degree of congestion decreases, the distance between moving objects increases and the number of nearby moving objects decreases, so that the tracking accuracy is less likely to decrease due to misalignment.

Therefore, as the degree of congestion increases, the number of candidate positions for each moving object can be increased to prevent a decrease in tracking accuracy due to congestion, and when the degree of congestion is low, the number of candidate positions for each moving object can be reduced to reduce the processing load. Will be able to.

By doing so, for example, when estimating one congestion degree for the entire captured image, even if a processing delay occurs during the period when the congestion degree is high, the delay is eliminated in a short time during the subsequent period when the congestion degree decreases. can. Further, for example, when estimating one degree of congestion for each image captured by a plurality of imaging units, processing is performed while preventing a decrease in tracking accuracy due to congestion by utilizing the bias of the distribution of moving objects among the captured images. The load can be suppressed.

Further, the candidate position number allocating means 51 may allocate the number of candidate positions of the moving object according to the degree of congestion of the object position of each moving object in the captured image. That is, the number of candidate positions is increased for a moving object that is predicted to have a high degree of congestion at the object position, and the number of candidate positions for a moving object that is predicted to have a low degree of congestion at the object position is suppressed to a small number. By doing so, it is possible to suppress the processing load while preventing the tracking accuracy from being lowered due to congestion by utilizing the bias of the distribution of moving objects in the captured image.

Specifically, the candidate position number allocation means 51 sets the congestion degree map stored in the congestion degree storage means 40, the object position of each moving object stored in the object information storage means 41, and the number of moving objects. With reference to this, the number of candidate positions of each moving object is determined, and the determined number of candidate positions is output to the candidate position setting means 52.

A specific example of a method of calculating the number of candidate positions to be assigned to each moving object by the candidate position number allocation means 51 is shown. The functions (a) to (d) shown in FIG. 4 are functions for calculating the number of candidate positions according to the degree of congestion, and based on this, the number of candidate positions is calculated from the degree of congestion.

(D) is the simplest function for calculating the number of candidate positions, and the number of candidate positions increases in proportion to the degree of congestion. On the other hand, in (c), when the degree of congestion is less than the threshold value T1, the number of candidate positions is constant, and when the degree of congestion is greater than or equal to the threshold value T1, the number of candidate positions increases in proportion to the degree of congestion. The number of candidate positions when the degree of congestion is less than the threshold value T1 is the minimum value of the number of candidate positions assigned to one moving object, and is set through a prior experiment. By setting the minimum value, it is possible to prevent the number of candidate positions from being set to be extremely small because the degree of congestion at the position where the moving object actually exists becomes a low value due to the influence of the error in estimating the degree of congestion. On the other hand, the range in which the tracked person can move is limited to some extent, and even if the number of candidate positions is increased more than necessary, improvement in accuracy cannot be expected. Therefore, in (b), when the number of candidate positions reaches a certain number (maximum value) and the degree of congestion is equal to or higher than the threshold value T2, the number of candidate positions is kept constant at the maximum value even if the degree of congestion further increases. (A) is a function in which both the minimum value and the maximum value are set for the number of candidate positions. Further, T3 in FIG. 4 is a threshold value for discriminating an ultra-highly congested region.

[Compression of the number of candidate positions by the total number upper limit]
When the number of candidate positions for the tracked person is determined by the above function, as the number of people existing in the highly congested area increases, the total number of candidate positions of all the tracked persons becomes large and the processing is delayed.

For example, when the monitoring space is a signal waiting area at an intersection, the degree of congestion changes periodically depending on the signal cycle, so even if a delay due to a high congestion situation occurs temporarily, the delay is restored after the congestion is resolved. It is possible. On the other hand, there is a possibility that congestion will continue for a long time at the entrance and exit of the event venue. In such a monitoring space, it is difficult to recover when a delay occurs. Therefore, it is desirable to set an upper limit of the total number of candidate positions in advance so that the delay does not occur.

Specifically, the number of candidate positions of each moving object once obtained for each person so that the sum of the candidate positions of all the moving objects to be tracked at one time falls within the upper limit of the total number is compressed. Number candidate position for a tracking person i can be a congestion degree d _i and the function f to the number of candidate positions before compression obtained from (x) f (d _i) and the following equation (1). The upper limit of the total number is the upper limit of the candidate positions assigned to one time. In other words, it is the upper limit of the total number of candidate positions. The upper limit of the total number is set in advance to a value within a range in which processing delay does not occur depending on the performance of the moving object tracking device and the like.

Here, min (1.0, upper limit of total number / Σ _j f (d _j )) is the compression rate of the number of candidate positions, and by setting 1.0 as the maximum value, the total number of candidate positions Σ _j f ( If d _j ) does not reach the upper limit of the total number, the number of candidate positions is maintained. This is because it is premised that the number of candidate positions before compression is a necessary and sufficient number. By setting the upper limit of the total number in this way, it is possible to allocate the number of candidate positions according to the degree of congestion within the range not exceeding the upper limit of the total number, so that the tracking accuracy is lowered even in the monitoring space where congestion continues for a long time. It is possible to prevent the occurrence of delay while preventing the occurrence of delay.

[Minimum number of candidate positions for each tracked person]
When the number of candidate positions is compressed using the upper limit of the total number, the number of compressed candidate positions becomes extremely small, and the tracking performance in an extremely low congestion area may deteriorate. In order to prevent this, the lower limit of the number of candidate positions for the tracked person may be set. In equation (2), after assigning the lower limit to each moving object, the number of candidate positions already assigned is subtracted from the upper limit of the total number to obtain the residual number, and the number of candidate positions before compression of each moving object is subtracted from the lower limit. Distribute according to the ratio of. The lower limit is the lower limit of the candidate positions assigned to one moving object. The lower limit is set to the minimum number of allocations that can obtain tracking accuracy higher than the target value in a low congestion state through prior experiments.

For each moving object, the candidate position number allocation means 51 obtains an area where the moving object can exist from the object position of the moving object, acquires the degree of congestion of the area from the congestion degree map, and the moving object exists. It is the degree of congestion for the area to be obtained. In the present embodiment, the candidate position number allocation means 51 extrapolates the object position at the current time to the past object position stored in the object information storage means 41 for each person, and the extrapolated object position at the current time. Get the degree of congestion. However, for a person whose past object position does not exist more than once, the degree of congestion of the object position one time ago is acquired. In that case, it is desirable to use the congestion map one hour ago. In another embodiment, the candidate position number allocation means 51 reads the extrapolated current time object position and the degree of congestion around it for each person, and sets their representative values as the area in which the moving object can exist. The degree of congestion. The representative value can be an average value, a mode value, or a maximum value. However, for a person who does not have two or more object positions in the past, the representative value of the object position one time ago and the degree of congestion around it is used. Further, the congestion degree may be acquired by applying the candidate position one hour ago to the congestion degree map one hour ago. In that case, the degree of congestion of the moving object is calculated by weighting and averaging the degree of congestion one hour ago with the likelihood of the candidate position one hour ago.

The candidate position setting means 52 obtains the candidate positions of the moving object at the current time from the positions of the moving objects in the past by the same number as the number of candidate positions assigned to each moving object. That is, for each moving object being tracked, the past object position or the past candidate position of the moving object, or the past object position and the candidate position of the current time of the moving object from the past object position and the candidate position are used as the moving object. Set as many as the number of assigned candidate positions, and set a hypothesis that includes each candidate position of the moving object.

Specifically, the number of candidate positions input from the candidate position number allocation means 51 and the past object position or / and past hypothesis of each moving object stored in the object information storage means 41 are referred to as necessary. Then, a hypothesis of the current time is set for each moving object, and the set hypothesis is stored in the object information storage means 41. Further, a congestion degree map may be input from the congestion degree storage means 40 and referred to.

The candidate position setting means 52 sets, for each of the moving objects, the candidate positions where the existence of the moving object is predicted at the current time. The i-th candidate position _{Pi, t} at the current time t can be calculated by any of the following three methods based on the past object position or the past candidate position.

[First method]
This is a method of distributing candidate positions P _{i and t} around the object position Q _{t-1 one time ago.} The candidate position setting means 52 sets the diffusion amount Δ0 based on a random number for each moving object, and adds the diffusion amount Δ0 to the _{object position Q t-1 to obtain the candidate positions P i and t} (P _{i). , T} = Q _t-1 + Δ0) is performed as many times as the number of candidate positions.

[Second method]
This is a method of distributing candidate positions P _{i and t around a predicted value q t} that predicts the current object position. The candidate position setting means 52 first applies a motion model such as a constant velocity motion model or a prediction filter such as a Kalman filter to a plurality of past object positions for each moving object to predict a predicted value q _t , and then converts it into a random number. Based on this, the diffusion amount Δ1 is set, and the process of adding the diffusion amount Δ1 to the _{predicted value q t to obtain the candidate positions P i, t (P i, t} = q _t + Δ1) is performed as many times as the number of candidate positions. ..

[Third method]
This is a method of predicting the current candidate position from the past candidate position. First, the candidate position setting means 52 selects the same number of hypotheses as the number of candidate positions for each moving object based on random numbers from the hypotheses one hour ago, allowing duplication. The winning probability of each hypothesis one hour ago is, for example, the ratio of the likelihood of the hypothesis to the sum of the likelihoods of the hypotheses one hour ago of the moving object. Next, the candidate position setting means 52 predicts each hypothesis selected for each moving object by applying the motion model or the prediction filter to the candidate positions of the hypothesis and a plurality of past hypotheses corresponding to the hypothesis. The value q _{i, t} is calculated and the diffusion amount Δ2 is set based on the random number, and the diffusion amount Δ2 is added to _{the predicted value q i, t to obtain the candidate position P i, t} (P _{i, t} = q _{i). , T} + Δ2).

In the present embodiment, the candidate position is calculated by the first method (2 hours to 5 hours including the time registered as a new moving object by the object position determination means 53) until the speed can be calculated, and the speed can be calculated. After that, the candidate position is calculated by the third method.

Further, the smaller the degree of congestion, the wider the range for setting the candidate position of the moving object may be. That is, the candidate position setting means 52 sets the diffusion amount Δ0 to Δ2 described in the first to third methods as a larger amount as the degree of congestion is smaller. For example, the diffusion amount Δ0 to Δ2 is set in the range of 0 cm to 60 cm for a person in a highly congested area and in the range of 30 cm to 100 cm for a person in a less congested area. Will be done. The reason for doing this is as follows. That is, if the degree of congestion is small, the range in which the moving object can move becomes wide, so if the candidate positions are not set in a wide range, it is easy for the matching to fail. Therefore, by controlling the range in which the candidate position is set based on the degree of congestion, it is possible to prevent a failure in associating when the degree of congestion is small. In particular, since the tracking parameters can be controlled by the congestion degree estimated independently of the tracking, highly reliable control becomes possible.

The object position determining means 53 obtains the object position of the moving object at the current time based on a plurality of candidate positions in each moving object. That is, for each moving object being tracked, the feature amount of the moving object appears at the position on the captured image at the current time corresponding to the candidate position indicated by each of the plurality of hypotheses set at the current time for the moving object. The degree of this is calculated as the likelihood of the hypothesis, and the object position at the current time of the moving object is calculated by integrating the candidate positions indicated by the plurality of hypotheses according to the high likelihood of the hypothesis. However, the hypothesis located outside the captured image and the hypothesis located in the ultra-high congestion region are excluded from the calculation target in advance because the reliability of the likelihood is low.

Specifically, the object position determination means 53 reads out the current time hypothesis, the shape model, and the template of each moving object stored in the object information storage means 41, and for each moving object, the hypothesis of the moving object is obtained. A shape model is placed at the indicated candidate position, a partial image of the area overlapping the shape model is extracted from the captured image, the similarity between the image feature of the partial image and the image feature of the template of the moving object is calculated, and the similarity is calculated. Calculate the corresponding likelihood. Further, the object position determination means 53 additionally stores the likelihood in the object information storage means 41 in association with the hypothesis.

At that time, the object position determining means 53 further performs subtraction processing between the background image and the captured image to detect the background subtraction region, and for each moving object, the ratio of the background subtraction region to the partial image of the moving object is set. The likelihood may be corrected with a corresponding correction value. The background image is stored in the storage unit 4 for the detection process of a new moving object, which will be described later.

The object position determining means 53 calculates the object position of the moving object by integrating the candidate positions indicated by the hypothesis of the moving object for each moving object based on the likelihood of the hypothesis. Preferably, the weighted average value obtained by weighting the candidate positions indicated by the hypothesis of the moving object with the likelihood of the hypothesis and averaging them is used as the object position of the moving object. It is also possible to select a plurality of likelihoods having a threshold value TL or more set in advance, and determine the weighted average value obtained by weighting and averaging the candidate positions paired with the selected likelihoods as the object position. can. It is also possible to select the top predetermined number of likelihoods in descending order of likelihood, and determine the weighted average value obtained by weighting and averaging the candidate positions paired with the selected likelihood as the object position. can. In addition, the maximum likelihood is selected for each moving object, the likelihood is compared with the threshold TL, and if the maximum likelihood is equal to or higher than the threshold TL, the candidate position paired with the likelihood is set as the object position. May be determined.

The calculated likelihood is associated with the hypothesis and stored in the object information storage means 41, and the calculated object position is associated with the moving object and stored in the object information storage means 41.

Further, the object position determination means 53 updates the object position and the template of the moving object being tracked, determines the existence of the new moving object, registers the object information for the new moving object, and disappears. Performs processing on moving objects. Hereinafter, the processing for the moving object being tracked, the processing for the new moving object, and the processing for the disappearing moving object will be sequentially described.

[Moving object being tracked]
The object position determination means 53 additionally stores the determined object position in the object information storage means 41 for the moving object whose object position has been determined, and updates the template of the moving object according to the image feature at the current time. The update may replace the extracted image features with the stored image features, or the extracted image features and the stored image features may be weighted and averaged.

[New moving object]
The object position determining means 53 detects the background subtraction region by performing subtraction processing between the background image captured when there is no moving object in the monitoring space and the captured image, and creates a shape model for each object position at the current time. Place and extract background subtraction areas that do not overlap with any shape model. Then, the object position determination means 53 determines that a new moving object exists in the non-overlapping background subtraction region if the non-overlapping background subtraction region has an area effective as a moving object (person). The object position determining means 53 applies a shape model to a non-overlapping background subtraction region to determine an object position and region of a new moving object, and associates the object ID with an object ID to determine a template of the moving object and an object position of the moving object. It is stored in the object information storage means 41. Further, the object position determination means 53 stores the captured image when the moving object does not exist as a background image in the storage unit 4, and updates the background image with the captured image in the region where the background subtraction region is not detected. When the background subtraction region is detected when calculating the likelihood, it may be diverted without detecting it again.

[Disappearing moving object]
The object position determination means 53 determines that a moving object whose likelihood of all candidate positions is equal to or less than the threshold value TL is a disappearing moving object, and deletes the object information of the moving object. A moving object that is entirely concealed by a shield, a moving object that has moved outside the captured image, or a moving object that has moved to an ultra-highly congested area is determined to be a disappearing moving object. Then, it is stored in the ultra-high congestion area storage means (not shown) as a group of moving objects not to be tracked. The moving object is tracked again as a new moving object when it escapes from the ultra-highly congested area.

The tracking result output means 31 generates, for example, a locus image in which a series of object positions for each moving object being tracked is plotted, and a color corresponding to the degree of congestion is determined in advance to correspond to each pixel of the degree of congestion map. A congestion degree image in which pixel values of colors corresponding to the congestion degree of the pixel are set for the pixel is generated, and an image obtained by transparently synthesizing the locus image and the congestion degree image is output to the display unit 6. Further, the captured image at the current time may be superimposed.

[Operation example]
Hereinafter, the operation of the moving object tracking device 1 will be described. FIG. 5 is an overall flow chart of the operation of the moving object tracking device 1. When the operation of the moving object tracking device 1 is started, the photographing unit 2 acquires the photographed image at a predetermined frame rate, and outputs the photographed image to the image processing unit 5. The image processing unit 5 repeats a series of processes of steps S2 to S11 each time a captured image is input (step S1).

The image processing unit 5 outputs a congestion degree map to the photographed image acquired by the photographing unit 2 by the congestion degree estimating means 50. Further, the congestion degree of each moving object of the tracking target recorded in the object information storage means 41 is obtained by referring to the value of the congestion degree map corresponding to the predicted position of the moving object of the moving object at the current time of the tracking target. In addition, an ultra-highly congested area that is difficult to track is extracted (step S2).

The image processing unit 5 allocates the number of candidate positions for each person based on the degree of congestion of each person to be tracked by the candidate position number allocating means 51 (step S3).

The image processing unit 5 performs tracking processing on the person to be tracked and estimates the current object position (steps S4 to S9). The image processing unit 5 sequentially sets the person recorded in the object information storage means 41 as the person of interest and performs the tracking process of the person of interest. When the tracking process is completed for all the persons, the image processing unit 5 performs the tracking process. The process proceeds to step S10, while the tracking process is continued if there is an unprocessed person (step S9).

Hereinafter, the tracking process of steps S4 to S9 will be described in more detail. The image processing unit 5 functions as the candidate position setting means 52, and sets a hypothesis for the person of interest based on the number of candidate positions assigned in step S3 (step S4). That is, the candidate position setting means 52 extrapolates the current object position to the past object position of the person of interest, sets the same number of candidate positions as the assigned number in the vicinity thereof, and includes each candidate position. Set a hypothesis. After that, the hypothesis that the candidate position is outside the captured image or within the ultra-highly congested region extracted in step S2 is deleted.

The image processing unit 5 functions as the object position determination means 53, and the object position determination means 53 calculates the likelihood for each hypothesis set in step S4 (step S5).

After that, the object position determining means 53 determines whether or not the object of interest is a disappearing moving object (step S6), and if it is determined to be a disappearing moving object, the tracking end process is performed (step S7). That is, in the hypothesis that remains without being deleted in step S4, if the likelihood of all the hypotheses is less than the threshold value TL, the object position determining means 53 determines that the person of interest is a disappearing moving object, and the object information storage means 41 relates to the person of interest. Delete the information.

When it is determined in step S6 that the tracking can be continued, the object position determination means 53 determines the object of the tracking person based on the candidate position of the hypothesis group set in step S4 and the likelihood calculated in step S5. The position is estimated (step S8).

When the above-mentioned tracking processes S4 to S9 are performed on all the persons registered in the object information storage means 41, the image processing unit 5 advances the process to step S10 as described above, and the object position determination means 53 advances the process. , A person who has not yet been tracked is detected in the captured image, and if it is detected, it is added as a new tracked person (step S10).

When the tracking of the person is completed by the above-mentioned processes S2 to S10 for the captured image input in step S1, the image processing unit 5 outputs the tracking result to the display unit 6 (step S11). For example, the image processing unit 5 causes the display device or the like of the display unit 6 to display the estimated positions of all the persons as the tracking result.

[Modification example]
(1) In the above embodiment, an example in which the congestion degree estimating means 50 uses an estimator that outputs a continuous value is shown, but an estimator that outputs a discrete degree of congestion can also be used.

For example, the estimator is modeled with a multi-class SVM (Support Vector Machine), and depending on the degree of congestion, "background (unmanned)", "low congestion", "medium congestion", "high congestion", " The model is trained using the learning images classified into 5 classes of "ultra-high congestion" and labeled. Then, the congestion degree estimation means 50 sets a window centered on each pixel of the captured image, inputs the feature amount of the image in the window into the estimator, and identifies the class of each pixel. In this way, when the congestion degree estimation means 50 estimates the discrete congestion degree, the candidate position number allocation means 51 calculates the number of candidate positions by, for example, the function shown in FIG. FIG. 6 assumes a case where the degree of congestion is estimated discretely in five stages, and the number of candidate positions also changes discretely according to each stage of the degree of congestion.

In addition to the multi-class SVM, the estimator can also be realized by various multi-class discriminators learned by the decision tree type random forest method, the multi-class AdaBoost method, the multi-class logistic regression method, or the like. Alternatively, an estimator can be realized by an identification type CNN (in the case of CNN, window scanning is unnecessary). Further, even when a class-classified learning image is used, it is possible to realize an estimator that outputs a continuous value of the congestion degree by using a regression type model that returns the congestion degree from the feature amount. In that case, the parameters of the regression function for obtaining the degree of congestion from the features are learned by the ridge regression method, the support vector regression method, the random forest method of the regression tree type, or the Gaussian process regression. Alternatively, it can be an estimator using a regression type CNN (in the case of CNN, window scanning is unnecessary).

(2) In each of the above-described embodiments and modifications thereof, an example in which the object position determining means 53 detects a new moving object based on background subtraction processing is shown, but instead, an image of the moving object to be tracked is shown. A new moving object may be detected using a trained model obtained by machine learning an unspecified number of people (for example, deep learning of images of an unspecified number of people). In that case, the object position determination means 53 inputs the captured image into the trained model to detect the region of the moving object, and the region that does not overlap with any of the shape models is the region of the moving object having a size equal to or larger than the threshold value TD. It is determined that a new moving object exists.

(3) In the above-described embodiment and each modification thereof, tracking by one imaging unit is illustrated, but tracking can also be performed by using a plurality of imaging units having a shared field of view.

First, a three-dimensional coordinate system common to a plurality of shooting units is defined, the relationship between the position and orientation of each shooting unit in the common three-dimensional coordinate system is obtained, and the three-dimensional coordinate system is transferred to the image coordinate system of each shooting unit. Calculate the parameters required for coordinate transformation. The object position and the candidate position are a set of X-coordinate, Y-coordinate, and Z-coordinate in a three-dimensional coordinate system represented by the X-axis, the Y-axis, and the Z-axis in the vertical direction, which are orthogonal to each other on a horizontal plane such as the ground in the monitoring space. It is represented by (X, Y, Z).

The congestion degree estimation means 50 first estimates the congestion degree map for the captured images of all the photographing units. Then, the degree of congestion for moving objects in the common three-dimensional coordinate system is defined as the integrated degree of congestion, and is calculated based on the degree of congestion map of each photographing unit. Specifically, for each moving object, the object position in the common three-dimensional coordinate system is converted into the image coordinate system of each imaging unit, the position of each imaging unit on the congestion map is obtained, and the position of each imaging unit is obtained. Calculate the degree of congestion on the captured image. Then, the highest value among the congestion degrees of each photographing unit for each moving object is set as the total congestion degree of the moving object. The highest value of the degree of congestion for each shooting unit is set to the shooting part with the highest degree of congestion, which requires the largest number of candidate positions when allocating the number of candidate positions based on the total degree of congestion. This is because the number of candidate positions required to maintain the tracking accuracy can be satisfied for any of the photographing units by combining them.

Although the object position at the current time is unknown at the time of estimating the congestion degree, the congestion degree estimating means 50 predicts the object position at the current time from the past object position, speed, etc. for each moving object, and predicts the present. The total congestion degree of each moving object is obtained by using the object position at the time.

The candidate position number assigning means 51 allocates the number of candidate positions to the moving object by using the total congestion degree as in the above embodiment, and the candidate position setting means 52 assigns the candidate positions of the common three-dimensional coordinate system based on the assigned number. Set. Like the calculation of the total congestion degree, the likelihood of the candidate position is coordinate-converted for each shooting unit to the position on the captured image, and the template and the feature amount are compared at the calculated position on the captured image. By doing so, the likelihood of each imaging unit is calculated and averaged to obtain the likelihood of the final hypothesis (candidate position). The object position at the current time is calculated from the likelihood of each hypothesis (candidate position) in the same manner as in the embodiment.

(4) The present invention can be applied to moving objects other than humans, such as vehicles and animals, which can be in a congested state.

1 ... Moving object tracking device, 2 ... Shooting unit, 3 ... Communication unit, 4 ... Storage unit, 5 ... Image processing unit, 6 ... Display unit, 30 ... Image acquisition unit, 31 ... Tracking result output means, 40 ... Congestion degree Storage means, 41 ... Object information storage means, 50 ... Congestion degree estimation means, 51 ... Candidate position number allocation means, 52 ... Candidate position setting means, 53 ... Object position determination means

Claims (8)

A moving object tracking device that tracks one or more moving objects based on captured images taken sequentially.
Congestion degree estimation means for estimating the congestion degree of the moving object captured in the captured image, and
Candidate position number allocation means for allocating the number of candidate positions of each moving object according to the degree of congestion, and
Candidate position setting means for obtaining the candidate position of the moving object at the current time from the position of the moving object in the past by the same number as the number of candidate positions assigned to each moving object.
An object position determining means for obtaining an object position of the moving object at the current time based on the plurality of candidate positions of the moving object.
A moving object tracking device characterized by being equipped with. The congestion degree estimating means inputs the photographed image into an estimator learned in advance to output the congestion degree at an arbitrary position in the photographed image when the photographed image is input, and arbitrarily in the photographed image. Estimate the degree of congestion at the position of
The moving object according to claim 1, wherein the candidate position number allocation means allocates the number of candidate positions of each moving object according to the object position of the moving object on the captured image or the degree of congestion of the candidate position. Tracking device. The moving object tracking device according to claim 1 or 2, wherein the candidate position number allocation means allocates a larger number of candidate positions of the moving object as the degree of congestion increases. 3. The candidate position number allocation means is characterized in that the number of candidate positions of each moving object is compressed so that the sum of the candidate positions of the moving objects to be tracked at one time falls within the upper limit of the total number. The moving object tracking device described. The moving object tracking device according to any one of claims 1 to 4, wherein the candidate position setting means widens the range in which the candidate position of the moving object is set as the degree of congestion decreases. In the region where the degree of congestion is above a certain level, the processing by the candidate position number allocation means, the candidate position setting means, and the object position determination means is not performed.
The moving object tracking device according to claim 1, wherein the position of the moving object group on the captured images sequentially photographed is memorized. A moving object tracking method using a moving object tracking device that tracks one or more moving objects based on sequentially captured images.
The degree of congestion of the moving object captured in the captured image is estimated, and the degree of congestion is estimated.
The number of candidate positions for each moving object is assigned according to the degree of congestion.
The candidate positions of the moving object at the current time are obtained from the positions of the moving objects in the past by the same number as the number of candidate positions assigned to each moving object.
A moving object tracking method, characterized in that an object position of the moving object at the current time is obtained based on the plurality of candidate positions of the moving object. A moving object tracking program executed in a moving object tracking device that tracks one or more moving objects based on sequentially captured images.
A process of estimating the degree of congestion of the moving object captured in the captured image, and
The process of allocating the number of candidate positions for each moving object according to the degree of congestion,
A process of obtaining the candidate positions of the moving object at the current time from the past positions of the moving objects by the same number as the number of candidate positions assigned to each moving object.
A process of obtaining an object position of the moving object at the current time based on the plurality of candidate positions of the moving object.
A moving object tracking program characterized by running.

Images