JP2009151359A - Moving object tracking device, moving object tracking method, moving object tracking program, and recording medium recorded with moving object tracking program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To predict a moving prediction of a person in an actual environment, to restrain a repetition processing frequency required until a prediction result gets stable, and to make compatible high tracking precision and low computational cost. <P>SOLUTION: This moving object tracking device or the like stores three-dimensional structure of an actual world predicted preliminarily, an internal/external parameter of an image sensor such as a camera arranged in the actual world, and an action history of the person, into a three-dimensional environmental information database 21, and stores a probability distribution of a tracking object state estimated at each time, and the tracking object state having the maximum probability, into an object state distribution storage means 22. The moving object tracking device or the like introduces an action history distribution to expresses probably the sequential moving of the person at the next time from an optional position in the environment, using the action history of the person obtained by the tracking. A moving route according to an environmental factor is predicted therein by processing statistically the action histories, and the prediction precision is enhanced of course and is updated sequentially without requiring the presetting of the environmental factor. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a moving object tracking device, a moving object tracking method, and a moving object tracking program for tracking the position of a moving object such as an object, an animal, or a person using one or a plurality of image input devices (for example, a camera). In addition, the present invention relates to a recording medium on which a moving object tracking program is recorded.

  In the field of computer vision, many studies on tracking of a moving target object such as a person using information from a plurality of cameras have been conducted (for example, see Non-Patent Document 1).

Further, a method has been proposed in which a three-dimensional position of a person is directly predicted using a probability statistical method based on the MCMC method, and a plurality of persons are tracked by comparing a prediction result with an actual surveillance camera image ( For example, refer nonpatent literature 2.). This method can obtain a stable tracking result by avoiding the inverse problem of three-dimensional restoration.
Tatsuya Osawa, U Minoru, Kaori Wakabayashi, Hideki Koike, “Person Tracking Technology for Video Monitoring” NTT Technical Journal August issue (Special Issue: Image Processing Technology Supporting Video Monitoring Service), Aug. 2007 T.A. Osawa, X .; Wu, K .; Sudo, K .; Wakabayashi, H .; Arai and T.A. Yasuno, “MCMC based Multi-body Tracking Using Full 3D Model of Both Target and Environment,” in Proc. of IEEE Int'l Conf. on Advanced Video and Signal based Survey (AVSS'07), London, UK, Sep. 2007.

  In Non-Patent Document 2, by using an iterative process based on the MCMC method, a three-dimensional position of a person is predicted for each iteration, and finally the expected values of the plurality of prediction results are calculated and tracked. Is realized. However, it is well known that the prediction accuracy is poor at the initial stage of the iterative process, and the expected value is calculated after performing a process called “burn-in” that discards the unstable prediction result until the prediction result becomes stable. Is going.

  For this reason, in order to achieve both high tracking accuracy and low calculation cost (that is, a small number of iterations), it is necessary to improve the prediction accuracy at the initial iteration and stabilize the prediction result with a small number of iterations. In Reference 2, since the prediction is performed by adding Gaussian noise to simple linear prediction, there is a problem that the actual human motion predicts a large amount of movement to a position with a low probability of occurrence. is doing.

  In general, the movement path of a person is biased due to various environmental factors. For example, taking “passage” as an example, it is easily considered that the movement path along the direction of the path becomes dominant, and taking other movement paths is exceptional. If such environmental factors can be incorporated into human movement prediction, the prediction accuracy can be improved and the above-mentioned problems can be solved. However, the environmental factors are different for each position in the environment. , Difficult to set in advance. In addition, it is not easy because the meaning of the environmental factor itself may be altered by some factor.

  The object of the present invention is to solve the above-mentioned problems of the prior art, and by predicting the movement of an animal body that is likely to occur in an actual environment, until the prediction result is stabilized. In addition to reducing the number of iterations required to achieve high tracking accuracy and low computational cost, the target state representing the position and size of multiple moving objects in a three-dimensional space at each time can be estimated sequentially. It is another object of the present invention to provide a moving body tracking apparatus, a moving body tracking method, a moving body tracking program, and a recording medium recording the moving body tracking program.

  Therefore, in the present invention, using the behavior history of the moving object obtained by tracking, the behavior history that expresses how the moving object moves from the arbitrary position in the environment to the next time sequentially. Introduce distribution. In this case, by statistically processing the action history, it is possible to predict the travel route according to environmental factors, and it is possible to improve the prediction accuracy. Since the update is performed sequentially, the above problem can be solved.

  In order to achieve the above object, the invention according to claim 1 is an apparatus for tracking a state such as a three-dimensional position and size of a moving object using an image input device such as one or a plurality of cameras, Means for obtaining an image taken by the photographing means at the time, means for creating a silhouette image obtained by extracting only the area where the tracking target object is shown from the obtained input image, and a real-world tertiary previously measured A three-dimensional environment information database for storing the original structure and internal / external parameters of the photographing means arranged in the real world and the action history distribution of the target object in the environment, the information of the three-dimensional environment information database and the silhouette image Means for estimating the probability distribution of the target state at each time, and the target state probability distribution estimated by the estimating means at each time and the target having the maximum probability Target state distribution storage means for storing states, means for calculating a target state having the maximum probability at each time from the probability distribution of the target states stored in the target state distribution storage means, and stored in the three-dimensional environment information database Means for calculating an action history distribution from the target state calculated by the target state calculation means.

  The invention according to claim 2 is the invention according to claim 1, wherein the means for calculating the action history distribution corresponds to the means for determining the action history distribution to be updated and the action history distribution represented by a mixed normal distribution. Means for determining a normal distribution to be updated, means for updating normal distribution parameters, and means for updating the mixing ratio of the mixed normal distribution.

  According to a third aspect of the present invention, in the first or second aspect, the object state is represented by means of a set of ellipsoidal models, and the position of each object is represented by a position (x, y) on an arbitrary plane. And means for expressing the size of each object by the radius r and the height h of the ellipsoid model.

  According to a fourth aspect of the present invention, there is provided a method for tracking a state such as a three-dimensional position and size of a moving object using an image input device such as one or a plurality of cameras. Extracting only a region in which the tracking target object is shown from the input image, creating a silhouette image by the silhouette image creating means, and estimating the probabilistic distribution of the state of the tracking target object Calculating by means for calculating a target state having a maximum probability from the estimated target state distribution, and calculating by means for calculating an action history distribution from the calculated target state; It is characterized by having.

  The invention according to claim 5 is the invention according to claim 4, wherein the step of calculating the behavior history distribution corresponds to the step of determining the behavior history distribution to be updated from the behavior history distribution represented by a mixed normal distribution. Determining a normal distribution to be updated, updating a normal distribution parameter, and updating a mixing ratio of the mixed normal distribution.

  The invention according to claim 6 is the method according to claim 4 or 5, wherein the object state is represented by a set of ellipsoidal models, and the position of each object is represented by a position (x, y) on an arbitrary plane. And a step of expressing the size of each object by a radius r and a height h of an ellipsoid model.

  The invention according to claim 7 is characterized in that the moving object tracking device or moving object tracking method according to any one of claims 1 to 6 is described by a computer program and can be executed. It is a program to do.

  The invention according to claim 8 is characterized in that a program that records the execution of the moving object tracking device or moving object tracking method according to any one of claims 1 to 6 as being executable by a computer is recorded. It is a recording medium.

  According to the present invention, it is possible to stably track individual movements of a plurality of moving objects having different sizes by integrating images from a plurality of cameras. Moreover, by utilizing the action history distribution, the time required for convergence of the prediction is fast, and high-speed and high-precision tracking is possible.

  Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a diagram for explaining the configuration of a moving object tracking apparatus according to an embodiment of the present invention. In FIG. 1, the present invention relates to an image acquisition means 11, a silhouette image creation means 12, and a target state distribution estimation means. 13, a target state calculation unit 14, an action history distribution calculation unit 15, a three-dimensional environment information database 21, and a target state distribution storage unit 22.

  The image acquisition unit 11 is a unit that acquires image data taken at each time, and includes, for example, an image sensor such as a digital camera or a video camera.

  The silhouette image creating means 12 creates an image obtained by extracting only the region where the object is captured from the acquired image data, that is, a silhouette image.

  The target state distribution estimating unit 13 sets the target state distribution obtained at the previous time as an initial state, and uses the silhouette image created by the silhouette image creating unit 12 and the three-dimensional environment information database 21 to generate a probability distribution of the target state. And the target state distribution at the previous time stored in the target state distribution storage unit 22 is updated to the target state distribution at the current time.

  The target state calculation unit 14 uses the target state distribution storage unit 22 in which the target state distribution at the current time is stored, calculates the target state that takes the maximum probability, and updates the target state distribution storage unit 22.

  The action history distribution calculating unit 15 updates the action history distribution stored in the three-dimensional environment information database 21 using the target state calculated by the target state calculating unit 14.

  The three-dimensional environment information database 21 includes real-world three-dimensional structure information measured in advance, camera internal parameters (parameters such as focal length and image center) of the image acquisition means 11 such as a camera installed, camera, and the like. The external parameters (3D position and orientation information of the installed camera) and the behavior history distribution calculated by the behavior history distribution calculating means 15 are stored.

  The target state distribution storage unit 22 stores the target state that takes the probabilistic distribution of the target state estimated by the target state distribution estimation unit 13 and the maximum probability calculated by the target state calculation unit 14.

  Next, an example of an embodiment of the present invention will be described in detail. The present invention is to sequentially estimate a target state representing the position and size of each of a plurality of objects and persons existing in a three-dimensional space at each time, and track the objects and persons.

  In this embodiment, for example, as shown in FIG. 2, a plurality of persons H1, H2,... HK walking on a plane are set as tracking targets, and here, video cameras (cameras (1, 2, An example of tracking using C)) will be described. The following description will be made on the assumption that the cameras (1, 2,... C) acquire images synchronously at the same shooting interval.

  Usually, there is a non-target object B that is not a target to be tracked in the space, and the three-dimensional structure data of the non-target object B in advance is created manually by using CAD data, using a range finder, This structural information can be measured in advance using three-dimensional restoration or the like.

  This three-dimensional structure data is converted into a coordinate system in which the XY plane coincides with the floor surface and the height direction coincides with the Z axis. In addition, the same coordinate system as the three-dimensional structure data obtained by calibrating the parameters in the cameras of the arranged C units, all the cameras, and acquiring the three-dimensional position (x, y, z) and posture (Φ, θ, γ) Ask for. This is a well-known process called camera calibration, and various methods have already been proposed and can be easily realized.

  These three-dimensional structure data and the three-dimensional positions and postures of the C cameras are stored in the three-dimensional environment information database 21 in advance. Also, an action history distribution that predicts how to move at the next time based on the action history for each frame constituting the lattice by dividing the acquired three-dimensional structure data into a grid on the XY plane Save. Specifically, a probability distribution using a movement vector as a component is used. For example, a mixed normal distribution.

  In addition, although the grid-like partitioning method is completely arbitrary, for example, it can be considered that the grid is partitioned into a grid of 10 cm square in actual size as shown in FIG.

  In this embodiment, as an example, as shown in FIG. 4, an ellipsoid is considered as a model e approximating the shape of a person, the position of the person is represented by a two-dimensional coordinate value (x, y) on the floor, and the size of the person. The ellipsoid is represented by a radius r and a height h.

The state S representing the state of a plurality of persons is represented by a vector M (person model) representing the position and size of each person. The number of dimensions of the state S changes according to the number of people tracked. That is, since the vector M is a four-dimensional vector, it becomes a 4K-dimensional vector when tracking K people. State S _t at time t is expressed by Equation 1 below.

The model M _{i, t at} time t representing the position, size, and shape of the person i constituting the state S is a four-dimensional vector represented by the following Expression 2.

Here, x _i and y _i are two-dimensional coordinate values (x, y) indicating the position of the person i on the floor plane, and r _i and h _i are obtained when the person is approximated by an ellipsoid. It represents the parameter radius r and height h of the ellipsoid.

This embodiment is above-described to estimate the state S _t at each time t, sequentially estimating the state of the plurality of persons, and performs tracking.

FIG. 5 is an example of a flowchart showing the operation of the present embodiment, which will be described with reference to FIGS. First, the image acquisition means 11 acquires image data (Im ₁ , Im ₂ ,... Im _c ) synchronously captured by the camera (1, 2... C) (S301).

Next, the silhouette image creation means 12 applies the respective silhouette images (Sil ₁ , Sil ₂ ,...) To the image data (Im ₁ , Im ₂ ,... Im _c ) acquired from the cameras (1, 2... C). Sil _c) to create a (S302). The silhouette image is a binary image in which the luminance value of the area where the object of the photographed image is shown in FIG. 6 is “1” and the other area is “0”, 501 is the input image, 502 Is an example of a silhouette image created. Such an image can be easily generated by using a well-known method such as a background difference method or an inter-frame difference method.

From the state S _t-1 calculated for the previous time then estimates the probability distribution of the state S _t at the current time t by iterating (S303). This will be described below with reference to FIG.

Estimation of probability distribution over states S _t, a plurality of times is performed by (B + P) times the state generation process. It discarded resulting state by the first B iterations, to save the state obtained at the end of P times as the state distribution of S _t. We consider that all the P states can occur with equal probability (burn-in).

Here, the state generated by the nth iteration at time t is described as S ′ _{t, n} .

First, the process is started with the update count n = 0 (S ′ _{t, 0} = S _t−1 ). In S401, a change type is selected. The change type represents the type of processing for changing the state, and consists of the following four types.

  1: Add person, 2: Delete person, 3: Change person position, 4: Change person size. First, when no person is tracking a current state (S is a 0 vector), (1: Add person) is always selected. In other cases, the probability that each type is selected is set arbitrarily in advance, and the probability is selected from four types. For example, it is possible to set the probability that each type is selected such as type 1: 0.1, type 2: 0.1, type 3: 0.4, and type 4: 0.4.

Next, in S402, if the number of iterations is n, the temporary change state S ^* is calculated using the state S ′ _{t, n−1} generated at the “n−1” th time. The processing will be described below for each type.

1: When addition of a person is selected, a new model M _{n, t} is added to the state S ′ _{t, n−1} . For example, this can be set as follows.

  It is assumed that the range of possible positions in the person's space is known from the three-dimensional environment information database 21, and here, this range is

And Regarding the parameters r and h representing the size of an ellipsoid that approximates a person, a range of values that can be similarly taken from empirical prior knowledge about the size of the person (no person with a height exceeding 3 m, etc.) Can be limited. Here is this range

And

By generating uniform random numbers for the parameters x, y, r, and h under the above conditions, the model M _{n, t} can be generated. This represents that a person has newly entered the tracking range.

  In addition, if information such as a place to enter / exit in advance, for example, the location of a door, is obtained, it is also possible to make only the vicinity thereof an entrance / exit area.

2: When deletion of a person is selected, one model is randomly selected from the models M _{i, t} constituting the state S ′ _{t, n−1} , and the selected model M _{i, t is changed} to the state S 'Delete from _{t, n-1} . This indicates that the person being tracked has disappeared from the tracking range.

3: When the position change of the person is selected, one model is selected at random from the models M _{i, t} constituting the state S ′ _{t, n−1,} and the elements of the selected model M _{i, t} , The temporary position (x ′ _{i, t} , y ′ _{i, t} ) changed from the position (x _{i, t} , y _{i, t} ) of the person on the two-dimensional plane according to the following equation 4 Predict.

However, d _x, d _y is a value calculated by using the position _{(x i, t, y i} , t) the stored action history distributed grating sections containing, d by sampling in accordance with the action history distribution _x, obtain the value of d _y. This can be realized by using a method that realizes sampling according to an arbitrary probability distribution, such as the well-known Metropolis-Hasting method or the rejection sampling method.

Using the provisional position determined as described above, the model M ′ _{i, t} is _expressed by Equation 5

And

4: When a change in the size of a person is selected, one model is randomly selected from the models M _i constituting the state S ′ _{t, n−1,} and among the elements of the selected model M _i , A provisional shape and size parameter (r ′ _{i, t} ,) changed from (r _{i, t} , h _{i, t} ) representing the radius r and height h of an ellipsoid approximating a person according to the following equation (6): h ′ _{i, t} ) is calculated.

However, δ _r and δ _h are one-dimensional Gaussian noises, and their parameters can be experimentally determined to arbitrary values. Using this provisional shape and size parameters, the model M ′ _{i, t} is

And

The provisional state S ^* is obtained by changing the state S ′ _{t, n−1} generated at the “n−1” th time, assuming that the number of iterations is nth at the current time t. If type 1 is added, a new model M _{n, t} is added; if type 2 is selected, the selected model M _{i, t} is deleted; if type 3 and type 4, the selected model M _{i, t} is selected _{. The} model M ′ _{i, t} is generated by changing _t .

Next, an image simulating the silhouette image (Sil ₁ , Sil ₂ ,... Sil _c ) created in the creation of the silhouette image (S 302) using the temporarily changed provisional state S ^* , that is, a simulation image (Sim ₁ , Sim ₂ ,... Sim _c ) are created (S403).

First, a virtual scene simulating the real world is constructed by combining the three-dimensional environment information database 21 and the target state. The three-dimensional environment information database 21 stores real-world three-dimensional structure information as a three-dimensional point cloud or polygon data in a coordinate system in which the floor surface is the XY plane and the height direction is the Z axis. It is assumed that A plurality of three-dimensional models of ellipsoids based on the provisional target state S ^* calculated in S403 can be arranged in this three-dimensional structure.

  Next, using the internal parameters (focal length, image center, etc.) and external parameters (camera 3D position and orientation) stored in the 3D environment information database 21 in the real world, A simulation image is created by projecting a scene onto a virtual camera.

As an example, this projection processing can be performed uniquely using the method described in Non-Patent Document 2. At this time, a silhouette image (Sil ₁ , Sil ₂ ,... Sil _c ) is obtained by setting a binary image in which the value of the ellipsoidal region on the image is “1” and the value of the other region is “0”. Simulated simulation images (Sim ₁ , Sim ₂ ,... Sim _c ) can be created.
The above process is S403.

Next, in order to determine whether or not the provisional state S ^* is likely, the likelihood is calculated (S404). This silhouette image created in the determination and S302 of certainty of the subject state itself by prior knowledge _{(Sil 1, Sil 2, ...} Sil c) and generated in step S403 has been simulated image _{(Sim 1, Sim 2, ...} Sim c ).

There are two types of determination of the probability of a state based on prior knowledge.
One is a limitation due to the three-dimensional environment. Using the three-dimensional environment information database 21, the existence probability can be weighted according to the position (x, y) of the person state. In the present embodiment, it is assumed that the person exists on the floor plane, and it is assumed that the person does not climb up on a stationary object other than the object in the space. Therefore, for example, by preparing a penalty function P (S ^* ) represented by the following equation 8 that is inversely proportional to the height of the three-dimensional structure at the position (x, y) where the person exists, It is possible to add a weight.

However, h (M _{i, t} ) is the Z value of the three-dimensional structure at the position (x _{i, t} , y _{i, t} ) in the element of the model M _{i, t} , α is a constant, and is arbitrarily set experimentally Although it is possible, for example, when the Z value changes greatly depending on the position where the person stands because the floor plane is inclined, α needs to be increased.

  The other is a condition that the persons being tracked do not overlap in the three-dimensional space. Now, in this embodiment, since the person is approximated by an ellipsoid, the condition that the individual ellipsoids do not overlap, that is, the distance between the centers of two different ellipsoids is greater than the sum of the radii of the two ellipsoids. It is necessary to satisfy that.

In order to prevent two persons from overlapping each other in a three-dimensional space, a penalty function E corresponding to the distance R represented by the following Expression 9, that is, the distance between the centers of ellipsoids representing the two persons. Set (S ^* ).

Here, x _{n, t} and y _{n, t} indicate the position x coordinate and y coordinate of the person n.
The function E that gives a penalty when the distance between the ellipsoids becomes short can be set as shown in Equation 10 below, for example.

However, β is a constant and can be arbitrarily set experimentally, and an optimum parameter is selected according to the degree of congestion in the environment. For example, when the degree of congestion is high and people frequently contact each other, the likelihood does not become extremely small if β is reduced.

Next, a comparison between the created silhouette image (Sil ₁ , Sil ₂ ,... Sil _c ) and the simulation image (Sim ₁ , Sim ₂ ,... Sim _c ) by creating a silhouette image (S 302) will be described. This can be calculated by, for example, the following evaluation formula (11).

However, (k, l) indicates a k-row l-column component of the image.
In Expression 11, information of all the C cameras is integrated, and evaluation is performed using information from a plurality of viewpoints.

Finally, the likelihood L of the provisional state S ^* is expressed by Equation 12 and calculated by product operation.

The above process is S404.

In step S405, the likelihood calculated in step S404 is used to calculate whether to accept and update the provisional state S ^* or reject the provisional state S ^* . This is determined by calculating the acceptance rejection judgment probability A. The acceptance refusal determination probability A can be calculated, for example, by the following equation (13).

Here, if it is the nth iteration, L ′ represents the likelihood of S ′ _{t, n−1} updated at the “n−1” th time. That is, the current state is accepted if the likelihood is higher than the previous state, and if not, the provisional state S ^* is adopted with a probability of L / L ′.

Here, if the provisional state S ^* is adopted, S ′ _{t, n} = S ^* is set, and if the adoption is rejected, S ′ _{t, n} = S ′ _{t, n−1} is set and the condition determination step of S407 is performed. Proceed with

  In S407, it is determined whether the iterative process has been performed a prescribed number of times. If the number of iterations n exceeds B + P times represented by the sum of constants B and P that can be arbitrarily determined, the process is terminated.

If n exceeds B times, the obtained state S ′ _{t, n} is stored in the state distribution storage means 22. That is, P states are stored in the target state distribution storage unit 22, and it is considered that all the P states can occur with equal probability, and this is set as a state distribution.
The process described above is S303.

Next, in S304, the state having the maximum probability is calculated from the state distribution obtained in S303. Since the P number of states constituting the state distribution is assumed to occur at all equal probability, the state S _t with the maximum probability can be calculated by the following equation 14.

  Further, a 4K-dimensional change vector V is obtained by the following Expression 15.

However, when the initial time t = 0 and the number of tracking persons increases, there is no corresponding model, so this element is set to “0”. _St and V calculated in this way are stored in the target state distribution storage means 22.

  Next, in S305, an action history distribution is calculated. The action history distribution can be represented by an arbitrary probability distribution model. Here, for simplicity, an example of a mixed normal distribution will be described. At the initial time t = 0, environmental factors are not known, so it is necessary to set a prior distribution in all grid sections.

  As an example, all the nine normal distributions representing the eight directions and the nine directions of dwelling as shown in FIG. 8 are uniformly mixed in all lattice sections so that all the normal distributions have a mixing ratio of 1/9. The initial state may be set. By setting in this way, a good approximation of the distribution moving isotropically from the current position is obtained. Since the initial parameters (average value and variance) of each normal distribution largely depend on the realization environment (location, processing interval, etc.), they are determined experimentally.

  The behavior history distribution A represented by U normal distributions is expressed as follows.

Here, v _{i, t} is the change vector of the model M _{i, t} , U is the number of mixture distributions, w _u , μ _u , and σ _u are the mixtures of the distributions u (u = 1 to U) constituting the mixture distribution. Rate, average value, variance, I represents a 2 × 2 matrix unit.

  From the next time, environmental factors are learned by updating the movement history distribution. Hereinafter, a description will be given with reference to FIG.

First, i = 1 is started, and an update action history distribution is determined (S501). This is a process of determining whether to update the action history distribution of which grid sections, model M _i, models of time before that corresponds to _t M _{i, t-1} of the _{(x i, t-1,} y i, Let us select the action history distribution of the grid section including _t-1 ).

Next, in the determination of the corresponding normal distribution (S502), the corresponding normal distribution is determined from the mixed normal distributions. This determines the normal distribution having the average value closest to the x and y components of v _{i, t} as the corresponding normal distribution.

  Next, in the normal distribution parameter update (S503), the normal distribution parameters determined in S502 are updated by the following Expressions 17 and 18.

  γ is a value representing the speed of learning, and is determined empirically between 0 and 1 (considering changes in image frame rate and environment).

  Next, in the mixture ratio update S504, the mixture ratio is updated by the following equation 19.

Here, B _{u, t} is “1” in the case of a normal distribution with u selected, and “0” in other cases.

  Finally, in S505, it is confirmed whether or not the process is finished. If i exceeds K, that is, the number of tracked persons, the process ends. If not, i = i + 1, and the process returns to S501. As described above, the action history distribution is updated.

  Next, in S306, it is checked whether or not the tracking is finished. If the tracking is completed, the process is terminated. If the tracking is still continued, the process of acquiring an image (S301) is performed.

  As described above, it is possible to efficiently and sequentially estimate the states of a plurality of persons whose number of persons dynamically changes by repeating the processing described above.

  Furthermore, in the present embodiment, the image acquisition unit 11 is used for the comparison between the silhouette image and the simulation image as an example in the process of S405. However, an evaluation function that takes similarity between images, for example, a simple product operation is used. However, it can be realized similarly. In the present embodiment, in the processing of S405, as an example, “Equation 8” weighted in the height direction is used as the environmental penalty function. However, if the function can express the weight of the position of the person, Of course, any function can be used.

  In the present embodiment, in the process of S405, as an example, “Equation 10” is used as a penalty function related to the approach of a person. However, any function that can give a penalty when people overlap is used. It is natural that we can do it.

  1 may be realized by configuring a part or all of the functions of each unit in the apparatus shown in FIG. 1 with a computer program and executing the program using the computer. 5. The processing procedure shown in FIG. 7, FIG. 7 and FIG. 9 can be configured by a computer program, and it is needless to say that the program can be executed by the computer. The information can be recorded on a readable recording medium, for example, a flexible disk, MO, ROM, memory card, CD, DVD, removable disk, and can be stored or distributed. It is also possible to provide the above program through a network such as the Internet or electronic mail. In this way, the present invention can be implemented by installing a program provided by a recording medium or a network in a computer.

  Below, using the moving object tracking device described above, the images obtained from the two video cameras (camera 1 and camera 2) are processed to track two persons entering the tracking range one by one. The results will be described using the processing flow shown in FIG.

  First, when the process is started, a camera image is acquired (S301). FIG. 10 is a part of an image sequence acquired from two cameras 1 and 2, and the upper row is an image from the camera 1 (701 to 706), and the lower row is an image from the camera 2 (707 to 712). .

  Next, a silhouette image is created from the obtained camera image (S302). A silhouette image created using the background subtraction method is shown in FIG. As in FIG. 10, the upper row is a silhouette image (801 to 806) created from the camera 1, and the lower row is a silhouette image (807 to 812) created from the camera 2. Each silhouette image in FIG. Corresponds to the image shown at the same position.

  Next, the target state distribution is estimated in S303. Here, while the state is sequentially updated using the action history distribution, a simulation image simulating the state is created and evaluated.

  In FIG. 12, the upper part is a silhouette image, and the lower part is a part of a simulation image simulating the upper silhouette image. In this way, a simulation image is created while changing to various states, and a state distribution is created by calculation of likelihood and acceptance rejection calculation.

  In step S304, the target state is calculated from the target state distribution estimated in step S303, and the current state having the highest probability is obtained, and this is output as the state at the current time. At the same time, the state change vectors at the previous time and the current time are obtained. In S305, the behavior history distribution is updated from the calculated target state.

  FIG. 13 shows an image representing a state taking the maximum probability calculated. In FIG. 13, the upper row is an image (901 to 906) from the camera 1, the middle row is an image (907 to 912) simulating a state where the maximum probability is obtained, and the lower row is a camera installed on the ceiling as an overhead view. It is an image (913 to 918) simulating a state where the maximum probability is taken as if the person H1 and person H2 have entered the tracking range after a while and then pass each other and finally exit the tracking range again It is shown that it can be traced clearly. In addition, the number of people to be tracked changes dynamically as 0 → 1 → 2 → 0, but it can be confirmed that the person is tracked stably while the appearance and disappearance of the person.

  In S306, if the tracking is to be ended up to the current time, the process is ended, and if not, the process returns to the acquisition of the image in S301.

  By repeating the processing described above, in this embodiment, stable tracking can be performed even in a sequence in which two persons approach and pass each other.

It is a block diagram which shows one Embodiment of the moving body tracking device of this invention. It is explanatory drawing which shows the mode of the image data acquisition in one Embodiment of the moving body tracking device of this invention. It is explanatory drawing which shows an example of action history distribution. It is explanatory drawing which shows the ellipsoidal model which is a three-dimensional model of the object used with one Embodiment of the moving body tracking device of this invention. It is a flowchart which shows the process sequence of the moving body tracking method of this invention. It is explanatory drawing which shows the silhouette image corresponding to an input image. It is a flowchart which shows the process sequence of estimation of object state distribution (S303). It is explanatory drawing which shows an example of the initial state of action history distribution. It is a flowchart which shows the process sequence of calculation of action history distribution (S305). It is an image figure which shows a part of image sequence acquired from the camera. It is the silhouette image figure created from a part of image sequence acquired from the camera. It is an image figure which shows a part of image which simulated the silhouette image from the silhouette image and the prediction result. It is an image figure which shows the image which looked down and the image which simulated the object state calculated from a part of input image and the input image.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 ... Image acquisition means 12 ... Silhouette image creation means 13 ... Target state distribution estimation means 14 ... Target state calculation means 15 ... Action history distribution calculation means 21 ... Three-dimensional environment information database 22 ... Target state distribution storage means S301 S306, S401 to S407, S501 to S506 ... Steps 701 to 712, 901 to 906 ... Camera image 801 to 812 ... Silhouette image 907 to 918 ... Simulated image

Claims (8)

A device that tracks the state of a moving object such as a three-dimensional position and size using an image input device such as one or more cameras,
Means for acquiring images taken by the photographing means at each time;
Means for creating a silhouette image obtained by extracting only the region where the tracking target object is shown from the acquired input image;
A three-dimensional environment information database that stores the three-dimensional structure of the real world measured in advance and the internal and external parameters of the imaging means arranged in the real world and the action history distribution of the target object in the environment;
Means for estimating the probability distribution of the target state at each time using the information of the three-dimensional environment information database and the silhouette image;
A target state distribution storage unit that stores a probability distribution of the target state estimated by the estimation unit at each time and a target state having a maximum probability;
Means for calculating the target state having the maximum probability at each time from the probability distribution of the target state stored in the target state distribution storage means;
Means for calculating the behavior history distribution from the target state calculated by the target state calculation means for the behavior history distribution stored in the three-dimensional environment information database;
With
Update the target state distribution at the previous time stored in the target state distribution storage means to the target state distribution at the current time, and to the target state having the maximum probability calculated by the target state calculation means, The moving object tracking device, wherein the behavior history distribution stored in the three-dimensional environment information database is updated using the target state calculated by the behavior history distribution calculation means. The moving object tracking device according to claim 1,
The means for calculating the behavior history distribution includes:
Means for determining an action history distribution to be updated;
Means for determining a corresponding normal distribution from the behavior history distribution represented by a mixed normal distribution;
Means for updating normal distribution parameters;
Means for updating the mixing ratio of the mixed normal distribution;
A moving object tracking device characterized by comprising: The moving object tracking device according to claim 1 or 2,
The target state is
Means for representing a set of ellipsoidal models;
Means for representing the position of each object by a position (x, y) on an arbitrary plane;
Means for representing the size of each object by the radius r and height h of the ellipsoid model;
A moving object tracking device characterized by that. Means for acquiring images taken by the photographing means at each time;
Means for creating a silhouette image obtained by extracting only the region where the tracking target object is shown from the acquired input image;
A three-dimensional environment information database that stores the three-dimensional structure of the real world measured in advance and the internal and external parameters of the imaging means arranged in the real world and the action history distribution of the target object in the environment;
Means for estimating the probability distribution of the target state at each time using the information of the three-dimensional environment information database and the silhouette image;
A target state distribution storage unit that stores a probability distribution of the target state estimated by the estimation unit at each time and a target state having a maximum probability;
Means for calculating the target state having the maximum probability at each time from the probability distribution of the target state stored in the target state distribution storage means;
Means for calculating the behavior history distribution from the target state calculated by the target state calculation means for the behavior history distribution stored in the three-dimensional environment information database;
With
In a method for tracking a state such as a three-dimensional position and size of a moving object using an image input device such as one or a plurality of cameras,
Acquiring an image from an image input device such as a camera;
Extracting only the region in which the tracking target object is shown from the input image, and creating a silhouette image by the silhouette image creating means;
Estimating the probability distribution of the state of the tracking target object by the estimating means;
From the estimated target state distribution, calculating by means for calculating the target state having the maximum probability;
Calculating by means for calculating an action history distribution from the calculated target state;
An object tracking method comprising: The moving body tracking method according to claim 4,
Calculating the behavior history distribution includes determining a behavior history distribution to be updated;
Determining a corresponding normal distribution from the behavior history distribution represented by a mixed normal distribution;
Updating normal distribution parameters;
Updating the mixing ratio of the mixed normal distribution;
An object tracking method comprising: In the moving body tracking method according to claim 4 or 5,
The target state is
A step represented by a set of ellipsoidal models;
Expressing the position of each object by a position (x, y) on an arbitrary plane;
Expressing the size of each object by the radius r and height h of the ellipsoid model;
A moving body tracking method characterized by that. A moving object tracking program characterized in that the moving object tracking device or moving object tracking method according to any one of claims 1 to 6 is described in a computer program and is executable. A recording medium recording a moving body tracking program, wherein the moving body tracking device or the moving body tracking method according to any one of claims 1 to 6 is recorded by a computer.