JP2008134939A - Moving object tracking apparatus, moving object tracking method, moving object tracking program with the method described therein, and recording medium with the program stored therein - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To stably track respective moving objects by integrating images sent from one or more imaging apparatuses for imaging a plurality of moving objects. <P>SOLUTION: A plurality of moving objects are tracked by using: a means 11 for predicting the target states of the plurality of moving objects at each time on the basis of a plurality of image data obtained by photographing the moving objects by using one or more imaging apparatuses; a means 12 for acquiring the image data; a means 13 for generating silhouette images obtained by extracting areas including respective moving objects; a means 14 for estimating target state distribution on the basis of the silhouette images, the three-dimensional structure of a real world, and an internal parameter and an external parameter of the imaging apparatus, a change type, and probability distribution updating frequency; a means 15 for calculating change vectors based on a target state having maximum probability at present time and a target state having maximum probability at previous time; a means 21 for storing the three-dimensional structure of the real world, the internal parameter and the external parameter; a means 22 for storing the probability distribution of the target states; and a means 22 for storing the change vectors. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a technique for tracking a state such as a three-dimensional position and a size of a moving object (moving object including an object, an animal, and a human) using one or a plurality of image input devices (for example, a camera or the like). .

  In the field of computer vision, there are many studies on tracking moving objects such as people using information from multiple cameras. For example, the following research results have been realized.

  A silhouette image is prepared by observing a target from a plurality of viewpoints and extracting a target region (ie, silhouette) on the image. Further, the three-dimensional structure of the space to be tracked in advance (hereinafter referred to as three-dimensional environment information) and the internal parameters and external parameters (also referred to as camera parameters) of all the cameras used for tracking are obtained by measuring in advance. Keep it.

  The silhouette image is obtained by performing a background difference method or an inter-frame difference method on a captured image, and setting the luminance value of an area where the target of the captured image is “1” and the luminance values of other areas to “ This is a binary image expressed by “0” (that is, an actual difference image). A camera calibration method for calibrating internal parameters of a camera and obtaining a three-dimensional position and orientation is also widely known (for example, see Non-Patent Document 1).

  Then, the person model is arranged in the three-dimensional environment, and the person model is arranged by a virtual camera (for example, a virtual imaging device constructed by software) having the internal parameters and external parameters of the camera calculated previously. By capturing a scene, a silhouette image can be simulated.

In addition, a method of tracking a person under a general environment by using an ellipsoid for a human model and comparing the generated simulation image with a silhouette image (hereinafter referred to as an ellipsoid model tracking method) has been proposed. (For example, refer nonpatent literature 2).
Z. Zhang, “Aflexible new technology for camera calibration”, IEEE, Transactions on Pattern Analysis and Machine Intelligence, 2000 (2000), vol. 22, no. 11, p. 1330-1334. Hirokazu Kato, Atsushi Nakazawa, Seiji Iguchi, “Real-time human tracking using ellipsoidal model”, Journal of Information Processing Society of Japan, 1999 (Heisei 11), vol. 40, No11, p. 4087-4096.

  As described above, in the method of tracking a target based on information acquired by a camera, when a plurality of persons are tracked, the persons overlap each other on an image captured by the camera due to the approach of the persons. (Ie occlusion) is a problem.

  The occlusion problem can be reduced by integrating information obtained from multiple viewpoints. However, in the above ellipsoid model tracking method, since multiple cameras track independently, information integration of multiple cameras is not performed, and there is a problem that tracking is impossible when people approach each other. is doing.

  The ellipsoid model tracking method has a problem that the number of people to be tracked is fixed from the beginning, so that it cannot cope with the exit and appearance of a person from the tracking space.

  Furthermore, since a processing amount proportional to the number of people to be tracked is required, there is a problem that if the number of people to be tracked increases, the processing cost increases and the real-time performance of the tracking processing is lost.

  Since the ellipsoidal model used for generating the simulation image is given a fixed height and a fixed radius, it is impossible to deal with individual differences in the size of the person. In other words, when tracking a person having a body whose size is significantly different from the size of the ellipsoid model, there is a problem that a large error occurs in the measured value, and as a result, tracking also fails. Yes.

  The present invention has been made based on the above-mentioned problem, and from the images sent from one or a plurality of imaging devices that image a plurality of moving objects, the positions of the plurality of moving objects existing in the three-dimensional space at each time point, Provided are a moving body tracking device, a moving body tracking method, a moving body tracking method, a moving body tracking program describing the method, and a recording medium storing the program, which sequentially estimate a target state representing a size and stably track the movement of each moving body. There is.

  In order to solve the above-mentioned problem, the invention according to claim 1 is characterized in that the three-dimensional position and size of a moving object are regarded as the target state of the moving object, and the moving object is specified using one or a plurality of imaging devices. A moving object tracking device that images at a time interval and tracks the moving object based on a plurality of image data obtained by the image capturing, and predicts a plurality of moving object target states at each time The target state obtained by the prediction means, the image acquisition means for acquiring the image data, the silhouette image creation means for creating a silhouette image obtained by extracting the region where the moving object is copied from the image data, and the target state obtained by the target state prediction means Considering the initial state, the silhouette image created by the silhouette image creation means, the real world 3D structure saved in the 3D environment information management means, and saved in the 3D environment information management means The target state distribution is estimated using the internal parameter and the external parameter of the image pickup apparatus, and the target state distribution at the previous time stored in the target state distribution storage unit is updated to the target state distribution at the current time. Based on the target state distribution estimation means and the target state distribution at the current time stored in the target state distribution storage means, the target state having the maximum probability is calculated, and the target state having the maximum probability at the current time and the previous time A target state calculation unit that calculates a change vector based on a target state having a maximum probability and stores the calculated change vector in a target state storage unit; a three-dimensional structure of the real world measured in advance and the imaging 3D environment information management means for storing internal parameters and external parameters of the apparatus, and target state distribution storage for storing the probabilistic distribution of the target state estimated at each time And a target state storage means for storing the change vector at each time, wherein the target state distribution estimation means includes a first probability distribution update count B for the target state distribution storage means and a final probability for the target state. A means for comparing the number of distribution updates N and selecting a change type according to an arbitrary probability from a change type prepared in advance based on the comparison result; a means for selecting a moving object that changes the change type adaptively; Based on the selected change type and the comparison result between the probability distribution update count B and the probability distribution update count N, means for changing the target state of the selected moving object, and the same external parameters and internal parameters as the imaging device An image created by projecting the changed target state is simulated by simulating the silhouette image. A first condition likelihood of the target state based on the prior knowledge condition regarding the three-dimensional structure, and the probability of the target state based on the prior knowledge condition regarding the overlapping of moving objects The second condition likelihood and the third condition likelihood based on the comparison between the silhouette image and the simulation image, and the product of the first to third condition likelihoods is changed to the provisional target state. Means for determining the target state predicted for the nth time according to the result of whether or not the provisional target state is adopted, and the number of updates is the probability distribution calculation number B. If equal, means for calculating the state distribution based on the target state obtained by the B-th update and currently tracking based on the state distribution based on the target state obtained by the B-th update. Determine if the number of moving objects is equal to 0 If the number of moving objects is not equal to 0 and the number of updates exceeds the number of times of adding the probability distribution update number N to the probability distribution update number B, the last N updates are stored. It is considered that all the N target states that are generated can occur with equal probability, and the target state predicted for the nth time is regarded as a target state distribution.

  The invention according to claim 2 is the invention according to claim 1, wherein the object state of the moving object is regarded as a set of ellipsoidal models, and the position of each moving object is a position (x, y) on an arbitrary plane. The size of the moving object is represented by a radius r and a height h of the ellipsoid model, and a plurality of combinations of values of the parameters x, y, r, and h are stored.

  The invention described in claim 3 is a real-world three-dimensional structure measured in advance, a three-dimensional environment information management means for storing internal parameters and external parameters of the imaging device, and a target state estimated at each time. A target state distribution storage means for storing a stochastic distribution; and a target state storage means for storing the change vector at each time; and considering the three-dimensional position and size of the moving object as the target state of the moving object, A moving object tracking method used in an apparatus for capturing an image of a moving object at a specific time interval using one or a plurality of imaging devices and tracking the moving object based on a plurality of image data obtained by the imaging A target state predicting step for predicting a target state of a plurality of moving objects at each time, an image acquiring step for acquiring the image data, and a region in which the moving object is extracted from the image data. The silhouette image creation step for creating the et image and the target state obtained by the target state prediction means are regarded as the initial state, and the silhouette image created by the silhouette image creation means and the three-dimensional environment information management means are stored. The target state distribution is estimated using the real-world three-dimensional structure and the internal and external parameters of the imaging device stored in the three-dimensional environment information management unit, and stored in the target state distribution storage unit. The target state distribution estimation step for updating the target state distribution at the previous time to the target state distribution at the current time, and the target state having the maximum probability is calculated based on the target state distribution at the current time stored in the target state distribution storage means The change vector is calculated based on the target state having the maximum probability at the current time and the target state having the maximum probability at the previous time. A target state calculation step for storing the calculated change vector in the target state storage means, and the target state distribution estimation step includes the first probability distribution update count B for the target state distribution storage means and the last for the target state. Comparing the probability distribution update count N of the two, selecting a change type according to an arbitrary probability from change types prepared in advance based on the comparison result, and selecting a moving body that changes the change type adaptively And changing the target state of the selected moving object based on the selected change type and the comparison result between the probability distribution update count B and the probability distribution update count N, and the same external parameters as the imaging device, An image created by projecting the changed target state with a virtual imaging means having internal parameters, Based on a step of considering an image as a simulated image, a probabilistic first condition likelihood of a target state based on a prior knowledge condition regarding the three-dimensional structure, and a prior knowledge condition regarding overlapping of moving objects Calculating a probabilistic second condition likelihood of the target state and a third condition likelihood based on a comparison between the silhouette image and the simulation image, and calculating a product of the first to third condition likelihoods as the change A step of determining the likelihood of the provisional target state, a step of determining the target state predicted n times according to a result of whether or not the provisional target state is adopted, and the number of updates is the probability If the number of distribution calculations is equal to B, the step of calculating the state distribution based on the target state obtained by updating the B times, and the state distribution based on the target state obtained by updating the B times, Present A step of determining whether or not the number of moving objects being tracked is equal to 0; and if the number of moving objects is not equal to 0, and the update count adds the probability distribution update count N to the probability distribution update count B If all the N target states stored in the last N updates can occur with equal probability, the target state predicted for the nth time is regarded as a target state distribution. And having.

  The invention according to claim 4 is the invention according to claim 3, wherein the object state of the moving object is regarded as a set of ellipsoidal models, and the position of each moving object is a position (x, y) on an arbitrary plane. The size of the moving object is represented by a radius r and a height h of the ellipsoid model, and a plurality of combinations of values of the parameters x, y, r, and h are stored.

The invention according to claim 5 is a moving object tracking program, wherein the moving object tracking method according to claim 3 or 4 is described as a computer program executable by a computer.
Features.

  The invention described in claim 6 is a recording medium, which is described as a computer program executable by a computer and records the computer program.

  According to the first and third aspects of the present invention, it is possible to acquire a target state distribution of a plurality of moving objects, and to acquire a change vector of the moving object based on the target state distributions. The probabilistic likelihood of the target state based on prior knowledge regarding the three-dimensional structure, the probabilistic likelihood of the target state based on prior knowledge regarding overlapping of moving objects, and the likelihood by comparing the silhouette image and the simulation image can be acquired.

  According to the second and fourth aspects of the present invention, it is possible to store the position (x, y) on an arbitrary plane as the position of each moving object and the radius r and height h of the ellipsoid model as the size of the moving object.

  According to the invention of claim 5, the moving body tracking method according to claim 3 or 4 can be described as a computer program.

  According to the sixth aspect of the present invention, a computer program that implements the moving body tracking method according to the third or fourth aspect can be recorded on a recording medium.

  As described above, according to the first and third aspects of the present invention, a change vector of a moving object is calculated based on images acquired from one or a plurality of imaging devices, and a plurality of moving objects are calculated based on the change vector of the moving object. The movement can be tracked stably. Even in situations where moving objects are very close to each other using probabilistic likelihood, likelihood based on comparison of silhouette images and simulation images (that is, the result of integrating image information from a plurality of imaging devices) Can be tracked stably.

  According to the second and fourth aspects of the present invention, since x, y, r, and h can be stored, it is possible to perform tracking according to changes in the number of moving objects having different sizes and the number of moving objects that are dynamically tracked. Further, even if tracking is performed according to a change in the number of moving objects having different sizes or the number of moving objects to be tracked dynamically, the processing time can be kept constant and real-time performance can be ensured.

  According to invention of Claim 5, the computer program which mounted the moving body tracking method can be provided.

  According to invention of Claim 6, the recording medium which recorded the computer program which mounted the moving body tracking method can be provided.

  These can contribute to the computer vision field.

  The moving body tracking apparatus in this embodiment is demonstrated based on FIG.

  The moving object tracking device includes a target state prediction unit 11, an image acquisition unit 12, a silhouette image creation unit 13, a target state distribution estimation unit 14, a target state calculation unit 15, and a three-dimensional environment information management unit (for example, managing three-dimensional environment information). Database) 21, target state distribution storage means 22, and target state storage means 23.

  The target state prediction means 11 determines the target state at the current time using the state of the tracking target obtained at the previous time (hereinafter simply referred to as the target state; the three-dimensional position and size of the tracking target) and the change vector of the target state. Predict.

  The image acquisition means 12 is means for acquiring image data taken at each time, and includes, for example, an imaging device such as a digital camera or a video camera. The moving object tracking device in the present embodiment may include one or a plurality of imaging devices. Further, the image data may be acquired from information management means (for example, a database) that stores image data taken at each time.

  The silhouette image creating means 13 creates a silhouette image obtained by extracting only the region where the object is captured from the acquired image data.

  The target state distribution estimation unit 14 regards the target state obtained by the target state prediction unit 11 as an initial state, and uses the silhouette image created by the silhouette image creation unit 13 and the three-dimensional environment information management unit 21 to The probabilistic distribution of the state is estimated, and the target state distribution at the previous time stored in the target state distribution storage unit 22 is updated to the target state distribution at the current time.

  The target state calculation means 15 uses the target state distribution storage means 22 in which the target state distribution at the current time is stored, calculates the target state that has the maximum probability, and calculates a change vector from the target state at the previous time. The change vector from the target state stored in the target state storage unit 23 is updated.

  The three-dimensional environment information management means 21 includes real-world three-dimensional structure information measured in advance and internal parameters (for example, a digital camera, hereinafter simply referred to as a camera) of the installed image acquisition means 12 (for example, Parameters such as focal length and image center) and external parameters of the image acquisition means 12 (for example, three-dimensional position and orientation information of the camera itself) are stored. For example, the three-dimensional environment information management means 21 may include a database that stores three-dimensional structure information, internal parameters, and external parameters. The three-dimensional environment information management unit 21 may store three-dimensional structure information, internal parameters, and external parameters in a storage unit (for example, a hard disk device or a nonvolatile memory).

  The target state distribution storage unit 22 stores the probabilistic distribution of the target state estimated by the target state distribution estimation unit 14. For example, the probabilistic distribution of the estimated target state may be stored in a memory (or storage area) included in the target state distribution storage unit 22 or a hard disk device.

  The target state storage unit 23 stores a change vector between the target state having the maximum probability calculated by the target state calculation unit 15 and the target state at the previous time. For example, the change vector may be stored in a memory (or storage area) included in the target state storage unit 23 or a hard disk device.

  The conditions regarding the moving body tracking apparatus in this embodiment are demonstrated based on FIG. In the following description, the same reference numerals as those in FIG. 1 are omitted.

  FIG. 2 shows a plurality of persons walking on a plane (persons H1, H2,... Hk in FIG. 2) as tracking targets (that is, moving objects), and video cameras with fixed positions and postures of p units (that is, An example of tracking using cameras C1, C2,... Cp) will be described. In addition, the cameras C1 to Cp acquire images synchronously at the same shooting interval. Note that when acquiring images synchronously at the same shooting interval, for example, a synchronization device may be connected to or mounted on the cameras C1 to Cp.

  In general, there is a non-target object (non-target object B in FIG. 2) that is not a tracking target in space, and the three-dimensional structure data of the non-target object is CAD (Computer-Aided Design) by hand. It can be measured and acquired in advance using data creation, a range finder, or three-dimensional reconstruction from camera images. This three-dimensional structure data shall be converted into a coordinate system in which the XY plane coincides with the floor surface and the height direction coincides with the Z axis.

  Further, calibration of internal parameters of all (p) cameras arranged, and the three-dimensional position (x, y, z) and posture (φ, θ, γ) are the same coordinate system as the acquired three-dimensional structure data. (For example, refer nonpatent literature 1). These three-dimensional structure data and the three-dimensional positions and orientations of all the cameras are stored in advance in the three-dimensional environment information management means 21.

  A shape model of a person (moving body) in the present embodiment will be described with reference to FIG.

The time in the following description indicates the number of times (or the number of orders) sampled at a specific sampling interval Δt. The initial time t ₀ indicates the time when moving object tracking is started (that is, the time t is “0”). The previous time is the time between the initial time t ₀ and the current time (ie, the current time), and the time when the moving object was tracked immediately before the current time (ie, the time being processed; eg, t) (ie, t -1).

  In FIG. 3, the ellipsoid e is considered as a model approximating the shape of a person, the position of the person is a two-dimensional coordinate value (x, y) on the floor plane, and the size of the person is the radius r and height of the ellipsoid e. Represented by h. Further, the MCMC (Markov Chain Monte Carlo) method, which is a kind of the stochastic statistical state distribution estimation method, is used for the target state distribution estimation means 14.

A target state S representing the states of a plurality of persons is represented by a vector M (ie, a person model) representing the position and size of each person. The number of dimensions of the target state S is determined by the maximum number K of people that can be tracked. That is, since the vector M is a four-dimensional vector, it becomes a 4K-dimensional vector. For instance, the state S _t at time t is expressed by the following equation.

A person model M _i representing the position, size, and shape of the person i constituting the state distribution S is a four-dimensional vector represented by the following expression.

Here, x _i and y _i are two-dimensional coordinate values (x, y) indicating the position of the person i on the floor plane, and r _i and h _i are ellipsoids when the person is approximated by an ellipsoid. Represents the parameter radius r and height h. Note that x _i , y _i , r _i , and h _i may be stored and accessed in a storage unit (not shown, for example, a memory or an external storage device) provided in the moving body tracking device, for example.

As described above, the present embodiment, by estimating the target state at each time t that is present in three-dimensional space S _t (the position at each time t of the plurality of objects and people, subject state representing the magnitude) In addition, the state of a plurality of objects and persons is sequentially estimated and tracked.

  The processing procedure of the moving body tracking method of the present embodiment will be described based on the flowchart in FIG. In the following description, the same reference numerals as those in FIG. 1 are omitted.

First, the process is started at an initial time t ₀ (time t = 0), and at the start, the target state S and the change vector V of the target state are initialized to a zero vector (S300).

Next, the target state prediction unit 11 predicts the target state at the current time (S301). This is a linear prediction process performed using the target state calculated at the previous time and the change vector of the target state. Assuming that the target state obtained at the previous time is S _t−1 and the change vector is a 4K-dimensional vector V _t−1 , the predicted value S ′ _{t, 0} of the target state at the current time can be expressed by the following equation. it can.

However, t in the index of S ′ represents time t. The portion “0” indicates that the predicted value update count is “0”. In the estimation of the target state distribution (S304), which will be described later, the predicted value is regarded as S ′ _{t, 0} as the initial value, and then the target state distribution is obtained by performing a plurality of updates. That is, “0” in the index indicates an initial value whose update count is “0”.

It should be noted that immediately after the start of moving body tracking (initial time t ₀ ), since the target state and the change vector of the target state at the previous time are unknown, all elements of M _i are set to “0” and S ₀ is initialized. Do.

Next, using the image acquisition unit 12, the synchronous photography image data Im _1, Im _2, acquires ... Im _p from the camera C1, C2 ... Cp (S302) .

Next, the silhouette images Sil ₁ , Sil ₂ ,... Sil _p are obtained from the image data Im ₁ , Im ₂ ,... Im _p acquired from the cameras C 1, C ₂ ,. Create (S303). Note that the silhouette image is a binary image in which the luminance value of the area where the object of the image taken as shown in FIG. 5 is “1” and the other area is “0”. It is an example of a silhouette image in which an input image and an image 502 are created. Such an image can be easily generated by using a well-known method such as a background difference method or an inter-frame difference method.

Next, the target state S ′ _{t, 0} predicted in step S301 is set as the initial value of the target state, and S ′ is sequentially updated to estimate the probabilistic distribution of the target state (S304).

  Here, a processing procedure for estimating the probabilistic distribution of the target state will be described with reference to FIG.

  The estimation of the probability distribution of the target state is performed by a plurality of times (that is, (B + N) times) of state update processing (state update processing for the target state distribution storage means). In the first B update, the person's increase / decrease in the monitoring area and the probability distribution of the rough target state of each person's position are obtained. In the last N updates, the rough obtained by the first B update is obtained. A more detailed probability distribution of the target state is obtained with respect to the position and size of each person without increasing or decreasing the number of persons from the target state. By this two-stage estimation process, the state of a plurality of persons can be estimated robustly.

  The state obtained in the last N times is stored as a state distribution, and it is considered that all the N states can occur with equal probability. Since it is considered that the state obtained by the first B updates depends on the initial value, only the state obtained by the last N updates (a process generally called Burn-in) is adopted. Yes, the ratio of B and N can be freely set empirically.

  First, the update count n is initially set to “0” and the process is started.

  In step S401, a change type is selected. Note that the change type represents a type of processing to change the state, and four types (first type is person addition, second type is person deletion, third type is person position change, 4 types consist of person size change). The selection of each change type differs as follows depending on the number of updates.

When the number of updates n ≦ B, it is as follows.
First, if no person is being tracked as the current state (all M is a “0” vector), (first type: addition of person) is always selected. In other cases (at least one person is tracked), the probability that each type is selected is set arbitrarily in advance, and the probability is selected from four types.

When the number of updates n> B, it is as follows.
As will be described later, in the first update of B times, when it is determined that no one is tracking, the update process ends. When the number of persons to be tracked is not “0”, the change type (third type: person position change) or (fourth type: person size change) is selected with a predetermined probability.

Next, in step S402, an object to be changed is selected. Now, assuming that the number of updates is n, the temporary change state S ^* is calculated using the target state S ′ _{t, n−1} updated _n−1 .

At this time, the state of all the objects is not changed by one update process, but only the state of one object is changed by each update process. First, when the change type is addition of a person, a random selection is made from M _i that is currently a “0” vector (that is, M _i that is not tracked). M _i (i.e., M _i being tracked) not "0" vector otherwise performing randomly selected from a.

Next, in step S403, a temporary person model M ′ _i is calculated according to each of the first to fourth types of processing selected from the selected person model M _i . Hereinafter, processing by type will be described.

  In the first type (person addition), the following processing is performed.

Since the person model M _i is a “0” vector, it is necessary to newly set a value for the person model M _i . For example, the value can be set as follows.

  It is assumed that the range of positions that can be taken in the person's space is known by the three-dimensional environment information management means 21, and this range is defined as follows.

  The parameters r and h representing the size of an ellipsoid that approximates a person are based on empirical prior knowledge about the size of the person (for example, there is no person with a height exceeding 3 m (meters)). The range of possible values can be similarly limited. Here, this range is defined as follows.

Under the conditions as described above, M ′ _i can be generated by generating uniform random numbers for the respective parameters of x, y, r, and h regarding the ellipsoid.

  Note that the first type (person addition) processing indicates that a person has newly entered the tracking range. In addition, if information on a place to enter / exit (for example, the location of a door) is obtained in advance within the tracking range, it is also possible to consider only the vicinity as an entrance / exit area.

  In the second type (person deletion), the following processing is performed.

All elements are considered “0” and M ′ _i is a “0” vector. That is, the second type (person deletion) represents that the person being tracked has disappeared from the tracking range.

  In the third type (person position change), the following processing is performed.

Based on the position (x _i , y _i ) of the person on the two-dimensional plane among the selected elements of M _i , when the number of updates is n ≦ B, Expression 4 and the number of updates is n> B. In this case, the temporary position (x ′ _i , y ′ _i ) changed according to Equation 5 is calculated.

However, W _x and W _y are one-dimensional white noises, respectively, and parameters thereof can be determined to arbitrary values experimentally.

However, δ _x and δ _y are one-dimensional Gaussian noises, and δ _x and δ _y can be experimentally determined to arbitrary values. Using this provisional position (x ′ _i , y ′ _i ), M ′ _i is defined as in the following equation.

  In the fourth type (person size change), the following processing is performed.

Of the selected elements of M _i , if the number of updates is n ≦ B from (r _i , h _i ) representing the radius r and height h of an ellipsoid approximating a person, Equation 7 and the number of updates is n If> B, the temporary shape and size parameters (r ′ _i , h ′ _i ) changed according to Equation 8 are calculated.

However, W _r and W _h are one-dimensional white noises, respectively, and their parameters can be experimentally determined to arbitrary values.

However, δ _r and δ _h are one-dimensional Gaussian noises, and δ _r and δ _h can be experimentally determined to arbitrary values. Using the provisional shape and size parameters (r ′ _i , h ′ _i ) of this ellipsoid, M ′ _i is defined by the following equation.

Using the M ′ _i generated in each of the above types, the target state S ^* that is temporarily changed is generated. For example, S ^* changes the selected person model M _i out of the states S ′ _{t, n−1} obtained by the (n−1) th update if the number of updates is nth at the current time t. The generated M ′ _i is replaced.

Next, using the temporarily changed target state S ^* , an image simulating the silhouette images Sil ₁ , Sil ₂ ,... Sil _p created in the creation of the silhouette image (S 303) (ie, the simulation image Sim ₁ , Sim ₂ ,... Sim _p ) are created (S 404). The method of creating the simulation image (S404) is as follows.

First, a virtual scene simulating the real world is constructed by combining the three-dimensional environment information management means 21 and the target state. In the three-dimensional environment information management means 21, three-dimensional point information (or a three-dimensional point group having a coordinate system in which the floor surface is the XY plane and the height direction is the Z axis) (or Polygon data). A plurality of three-dimensional ellipsoidal models based on the provisional target state S ^* calculated in step S403 can be arranged in the three-dimensional structure.

  Next, the virtual scene is projected with the virtual camera using the internal parameters (eg, focal length, image center, etc.) and external parameters (camera three-dimensional position and orientation) of the camera placed in the real world. To create a simulation image. The internal parameters and external parameters are stored in the three-dimensional environment information management unit 21.

The projection process can be uniquely performed by using a method described in Non-Patent Document 2, for example. In the projection process, silhouette images Sil ₁ , Sil ₂ ,... Sil are obtained by setting a binary image in which the value of the ellipsoidal region on the image is “1” and the value of the other region is “0”. the simulation image Sim _1, Sim ₂ simulating the _p, can create a ... Sim _p.

  The above is the method for creating a simulation image.

Next, in order to determine how likely this provisional target state S ^* is, the likelihood is calculated (S405). Calculation of likelihood, prior knowledge silhouette image created in the determination and step S303 of certainty of the subject state itself by Sil _1, Sil _2, ... simulated generated by Sil _p and step S404 image Sim _1, Sim _2, ... calculated based on Sim _p comparison.

  The following two types of determination can be considered for the determination of the likelihood of the target state itself based on the prior knowledge conditions (that is, the calculation of the probabilistic likelihood (that is, the conditional likelihood) based on the condition).

The first determination is a determination using a restriction based on a three-dimensional environment. By using the three-dimensional environment information management means 21, the existence probability can be weighted according to the position (x, y) of the person state. In the present embodiment, it is assumed that the person exists on the floor plane, and the condition is that the person does not climb up on a stationary object that exists in the space other than the target. Assumes. Under the assumption, for example, by preparing a penalty function P (S ^* ) represented by the following equation that is inversely proportional to the height of the three-dimensional structure at the position (x, y) where the person exists, It is possible to weight the existence probability.

However, h (M _i ) is the Z value of the three-dimensional structure at the position (x _i , y _i ) in the element of M _i , and α is a constant that can be arbitrarily set experimentally.

  The second determination is a determination based on a condition that the persons being tracked do not overlap in the three-dimensional space. In other words, since the person is approximated by an ellipsoid, the condition that the individual ellipsoids do not overlap (that is, the distance between the centers of two different ellipsoids) satisfies that the sum of the radii of the two ellipsoids is greater. There is a need.

In order to prevent two persons from overlapping each other in a three-dimensional space, a penalty function E (S) according to the distance R expressed by the following equation and the distance between the centers of ellipsoids representing the two persons is represented. ^* ) Is set.

Here, x _n and y _n indicate the position x coordinate and y coordinate of the person n.

  The function E that gives a penalty when the distance between the ellipsoids becomes short can be set as the following equation, for example.

  However, β is a constant and can be arbitrarily set experimentally.

Then, silhouette images Sil ₁ created by the creation of the silhouette image _{(S303), Sil 2, ...} Sil p and simulation image Sim _1, Sim _2, ... comparison of Sim _p will be described. For example, it can be calculated by the following evaluation formula (that is, the conditional likelihood is calculated).

  Here, (k, l) indicates k rows and 1 column components (pixels) of the image.

  In the above formula, information of all p cameras is integrated, and evaluation is performed using information from a plurality of viewpoints.

Finally, the likelihood L of the provisional state S ^* is calculated by a product operation like the following equation.

  The above processing is the likelihood calculation (S405).

Next, in step S406, it is determined using the likelihood calculated in step S405 whether the provisional state S ^* is accepted and updated or rejected. This determination is made based on the calculation of the acceptance rejection determination probability A. The acceptance refusal judgment probability A can be calculated by the following equation, for example.

Here, assuming that the update is the nth time, the likelihood L ′ represents the likelihood of S ′ _{t, n−1} updated _{the n−} 1th time. That is, the current state is always accepted if the likelihood is higher than the previous state, and otherwise, the provisional state S ^* is adopted with the probability of L / L ′ (that is, the acceptance rejection determination probability A).

Further, if the provisional state S ^* is adopted, it is assumed that S ′ _{t, n} is equal to S ^* (S ′ _{t, n} = S ^* ), and if adoption is rejected, S ′ _{t, n} Is assumed to be equal to S ′ _{t, n−1} (S ′ _{t, n} = S ′ _{t, n−1} ), and the process proceeds to the condition determination step of step S407.

  In step S407, it is determined whether the number of updates n is equal to B. If the number of updates n is not equal to B, the process proceeds to the condition determination step in step S410. When the number of updates n is equal to B, the process proceeds to the state distribution calculation step in step S408.

  In step S408, the state distribution is calculated as a rough estimation based on the state obtained by the B updates so far. The obtained B states may have different information such as the number of persons tracked by all of them and whether or not the person tracked until the previous time is being tracked.

  The following K-dimensional vector k can be used to represent the number of persons to be tracked and the information of tracking succession.

Each I _i corresponds to the state M _{i of the} person model and takes one of the values “0”, “1”, and “2”. Each value of I _i is defined as follows:

If I _i is “0”, the corresponding M is a zero vector and no tracking is performed.

If I _i is “1”, the person who was tracked at the previous time has been taken over and tracked.

If I _i is “2”, a person has been newly added to the corresponding M in the current time update process.

  Count the state where the k vectors are the same (equal to the k vector), and consider that the k vector that has obtained the largest number of votes represents the current number of people tracked and the succession of tracking, A rough state distribution is calculated using a state that holds many k vectors.

For example, if the maximum number of votes is C and the state where the maximum number of votes is obtained is S ′ _{t, i} , the state distribution _{St ′} is calculated by the following equation. However, it is assumed that (i = 1, 2,..., C), and the index i is assigned in ascending order of the number of updates while the k vector having the largest number of votes is held.

Next, in condition determination step S409, it is determined whether or not the number of people currently tracked is “0” in the state distribution obtained in step S408. If the number of persons tracked is “0”, the state update process is terminated. If there is a person who is tracking even one person (the number of persons tracked is not “0”), the state updated at the nth time is set as the state distribution _{St ′} , and the process proceeds to the condition determination step S410.

In step S410, it is determined whether the update process has been performed a prescribed number of times. If the number n of updates exceeds the number (B + N times) represented by the sum of constants B and N that can be arbitrarily determined, the process is terminated. Further, the state S ′ _{t, n} is stored in the target state distribution storage unit 22. It is considered that all N states stored in the last N updates can occur with equal probability, and this is set as a target state distribution. If it does not exceed B + N times, the update number n is incremented by 1, and the process returns to step S401.

  The process described above is S304.

Next, in step S305, the target state having the maximum probability is calculated from the target state distribution obtained in step S304. However, if it is determined in step S304 that the number of persons to be tracked is “0”, the target state is a “0” vector, so this calculation is not performed. Since it is assumed that N pieces of object state constituting the target state distribution occurs at all equal probability, the target state S _t with the maximum probability it can be calculated by the following equation.

  Further, in order to predict the target state at the next time by the target state prediction means 11, a 4K-dimensional change vector V is obtained by the following equation.

However, since the above equation cannot be calculated at the initial time t ₀ , all elements of V are set to “0”. _St and V calculated in this way are stored in the target state storage means 23.

  In step S306, it is checked whether or not to end tracking. If the tracking is completed, the process is terminated. If the tracking is still continued, the process of predicting the state at the next time (S301) is performed. For example, it is checked whether or not the time t exceeds a predetermined value set in advance. If the time t exceeds the predetermined value, the tracking is ended. If it does not exceed the predetermined value, the process returns to step S301 to continue tracking. In addition, in order to check whether or not the tracking is to be ended, a tracking stop instruction means (for example, a keyboard device provided in the moving object tracking device) is provided, and when there is a tracking stop instruction from the tracking stop instruction means You may stop tracking.

  As described above, by repeating the processing in the present embodiment, it is possible to sequentially estimate the states of a plurality of persons whose number of persons dynamically changes.

  In other words, the present embodiment is made to solve the above-described problems of the prior art, and can integrate a plurality of camera information and perform tracking stably even if the objects approach each other. it can.

  Further, by performing two-stage probability distribution estimation corresponding to the dynamic increase / decrease in the number of people tracked, it is possible to stably track a person who has newly appeared in the tracking space or a person who has left the tracking space.

  Further, even if the number of people to be tracked increases at a time, the entire processing amount can be kept constant, and the real-time performance of the tracking process can be maintained.

  Moreover, tracking corresponding to individual differences can be performed by estimating the size of the object simultaneously with the position of the object.

  In the present embodiment, in the process of step S405, Expression 11 is used as an example for comparing the silhouette image and the simulation image, but an evaluation function (for example, a simple product operation) that takes similarity between images is used. However, it can be realized similarly.

  Further, in the present embodiment, in the process of step S405, as an example, the expression 10 weighted in the height direction is used as the environmental penalty function, but a function that can express the weight of the position of the person can be used separately. Is natural.

  In the present embodiment, in the process of step S405, Expression 11 is used as a penalty function related to the approach of a person as an example, but it is natural that any function that gives a penalty when the persons overlap can be used.

  It is to be noted that a part or all of the functions of each means in the apparatus shown in FIG. 1 can be configured by a computer program and the program can be executed using the computer to realize the present invention, FIG. 4 and FIG. It is needless to say that the processing procedure shown in FIG. 6 is configured by a computer program and the program can be executed by the computer, and the program for realizing the function by the computer can be read by a computer-readable recording medium, for example, FD (Floppy (registered trademark) Disk), MO (Magneto-Optical disk), ROM (Read Only Memory), memory card, CD (Compact Disk), DVD (Digital Versatile Disk), removable device And recorded in a disk, or stored, it is possible or distribute. It is also possible to provide the above program through a network such as the Internet or electronic mail.

  Further, a computer program describing a method related to the moving object tracking device may be mounted so as to access a memory or an external storage device that stores input / output data required for the method related to the moving object tracking device.

[Example]
In this embodiment, the moving object tracking device described above is used to process images obtained from two video cameras (camera C1 and camera C2), and two persons entering the tracking range one by one are detected. The traced result will be described based on the processing flow shown in FIG.

First, the process is started at an initial time t ₀ (time t = 0), and at the start, the target state S and the change vector V of the target state are initialized to a zero vector (S300).

  Next, the target state is linearly predicted (S301).

  Next, an image is acquired from the camera (S302). FIG. 7 is a part of an image sequence acquired from two cameras. The upper row is images 701 to 706 acquired from the camera C1, and the lower row is images 707 to 712 acquired from the camera C2.

  Next, a silhouette image is created from the obtained image (S303). A silhouette image created using the background subtraction method is shown in FIG. As in FIG. 7, the upper row is a silhouette image 801 to 806 created from an image acquired from the camera C1, and the lower row is a silhouette 807 to 812 created from an image acquired from the camera C2. Each silhouette image in FIG. 8 corresponds to each image shown in the same position in FIG.

  Next, the target state distribution is estimated in step S304. That is, in step S304, a simulation image simulating the state is created while sequentially updating the state, and the simulation image is evaluated.

  In FIG. 9, the upper part is a silhouette image 901, and the lower part is a part of a simulation image simulating the upper silhouette image (simulation images indicated by reference numerals 902 to 904). In this way, a simulation image is created while changing to various states, and a target state distribution is created by likelihood calculation and acceptance rejection calculation.

  In step S305, a target state that currently takes the maximum probability is obtained from the target state distribution calculated in step S304, and this is output as the target state at the current time. At the same time, a change vector of the target state at the previous time and the current time is obtained.

  FIG. 10 shows an image representing the target state taking the maximum probability calculated. FIG. 10 shows images 1001 to 1006 from the camera in the upper stage, images 1007 to 1012 in which the middle stage simulates a target state that takes the maximum probability, and takes the maximum probability assuming that the camera is installed on the ceiling as the overhead view. It is the images 1013 to 1018 simulating the target state.

  This shows that after the person H1 and the person H2 have entered the tracking range after a short time, they can clearly trace the situation of passing each other and finally leaving the tracking range. In addition, it can be confirmed that the number of people to be tracked is dynamically changed as 0 → 1 → 2 → 0, but the person is stably tracked while appearing and disappearing.

  In step S306, if the tracking is to be ended up to the current time, the process is ended, otherwise, the process returns to the prediction of the target state in step S301.

  By repeating the processing in the above embodiment, stable tracking can be performed even in a scene where two persons approach each other and pass each other. That is, this indicates that information of a plurality of cameras is integrated.

  In addition, although the number of people to be tracked is dynamically changing, it is shown that tracking can be performed in response to the appearance and disappearance of a person.

  Furthermore, even if the number of persons to be dynamically tracked changes as described above, there is no problem that the total processing amount increases, and real-time performance is ensured.

  Also, stable tracking is possible without giving parameters in advance to two persons with different body shapes.

  These features are because the information of a plurality of cameras can be integrated to estimate the size of the body.

  In the embodiment, information acquired from a plurality of cameras is targeted. However, the same result can be obtained even if information acquired from a single camera is targeted.

  Although the embodiments of the present invention have been described above, the present invention is not limited to the described embodiments, and various modifications can be made within the scope described in each claim.

For example, as a modification of the present embodiment, values that can be arbitrarily set experimentally (for example, the above-described B, N, W _x , W _y , W _r , W _h , δ _x , δ _y , δ _r , δ _h , Α, β values) may be provided, and the above-described processing related to moving object tracking may be performed based on the input value from the input unit. That is, a value is input to the moving object tracking device by input means including a keyboard device and the above-described processing related to moving object tracking is performed based on the input value.

The block diagram of the moving body tracking device in this embodiment. The figure which shows the conditions regarding the moving body tracking device in this embodiment. The three-dimensional model figure used with the moving body tracking device in this embodiment. The flowchart which shows the process sequence of the moving body tracking method in this embodiment. The figure which shows the silhouette image corresponding to the input image in this embodiment. The flowchart which shows the process sequence regarding estimation of object state distribution in this embodiment. The figure which showed a part of image sequence acquired from the camera in a present Example. The figure which showed the silhouette image created from a part of image sequence acquired from the camera in a present Example. The figure which showed a part of image which simulated the silhouette image based on the silhouette image and the prediction result in a present Example. The figure which showed the image which simulated the object state calculated from a part of input image and the input image in this Example, and the bird's-eye view of it.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 ... Target state prediction means 12 ... Image acquisition means 13 ... Silhouette image creation means 14 ... Target state distribution estimation means 15 ... Target state calculation means 21 ... Three-dimensional environment information management means 22 ... Target state distribution storage means 23 ... Target state storage Means 501 ... Input image 502, 801 to 812, 901 ... Silhouette image 701 to 712, 1001 to 1006 ... Image from camera 902 to 904 ... Simulation image 1007 to 1018 ... Image simulating the target state B ... Non-target object C1 , C2, Cp ... camera H1, H2, Hk ... person e ... ellipsoid

Claims (6)

Assuming the three-dimensional position and size of the moving object as the target state of the moving object, the moving object is imaged at a specific time interval using one or a plurality of imaging devices, and a plurality of images obtained by the imaging A moving object tracking device that tracks the moving object based on image data,
A target state prediction means for predicting a plurality of target states of the moving object at each time;
Image acquisition means for acquiring the image data;
Silhouette image creating means for creating a silhouette image obtained by extracting an area in which the moving object is copied from the image data;
Considering the target state obtained by the target state prediction means as the initial state, the silhouette image created by the silhouette image creation means, the three-dimensional structure of the real world stored in the three-dimensional environment information management means, and the three-dimensional The target state distribution is estimated using the internal parameters and external parameters of the imaging device stored in the environment information management unit, and the target state distribution at the previous time stored in the target state distribution storage unit is determined as the current time. Target state distribution estimating means for updating to the target state distribution;
Based on the target state distribution at the current time stored in the target state distribution storage means, the target state having the maximum probability is calculated, and the target state having the maximum probability at the current time and the target state having the maximum probability at the previous time are calculated. A target state calculation unit that calculates a change vector based on the calculated change vector and stores the calculated change vector in a target state storage unit;
Three-dimensional environment information management means for storing the real-world three-dimensional structure measured in advance and the internal and external parameters of the imaging device;
Object state distribution storage means for storing a probabilistic distribution of the object state estimated at each time;
Target state storage means for storing the change vector at each time;
With
The target state distribution estimating means is
The first probability distribution update number B for the target state distribution storage means is compared with the last probability distribution update number N for the target state, and based on the comparison result, a change type is selected according to an arbitrary probability from a change type prepared in advance. Means to choose;
Means for adaptively changing the moving type to change the moving type;
Means for changing the target state of the selected moving object based on the selected change type and the comparison result of the probability distribution update count B and the probability distribution update count N;
A virtual imaging unit having the same external parameters and internal parameters as the imaging device, a unit that regards an image created by projecting the changed target state as a simulation image simulating the silhouette image;
A probabilistic first condition likelihood of a target state based on a prior knowledge condition relating to the three-dimensional structure, and a probabilistic second condition likelihood of the target state based on a prior knowledge condition relating to overlapping of moving objects;
Means for calculating a third condition likelihood based on a comparison between the silhouette image and the simulation image, and considering a product of the first to third condition likelihoods as the likelihood of the changed provisional target state; ,
Means for determining a target state predicted n times according to a result of whether or not the provisional target state is adopted;
When the number of updates is equal to the probability distribution calculation number B, means for calculating the state distribution based on the target state obtained by the B times of update,
Means for determining whether or not the number of moving objects currently being tracked is equal to 0 based on the state distribution based on the target state obtained in the B-th update;
If the number of moving objects is not equal to 0 and if the update count exceeds the probability distribution update count B plus the probability distribution update count N, the N stored in the last N updates Means that all target states can occur with equal probability, and the target state predicted for the nth time is regarded as a target state distribution;
Having
A moving body tracking device characterized by that. The moving body tracking device according to claim 1,
The target state of the moving object is regarded as a set of ellipsoid models, the position of each moving object is a position (x, y) on an arbitrary plane, the size of the moving object is the radius r and height h of the ellipsoid model, Represented by
Storing a plurality of combinations of values of these parameters x, y, r, h,
A moving body tracking device characterized by that. Three-dimensional environment information management means for storing the real-world three-dimensional structure measured in advance and the internal and external parameters of the imaging device;
Object state distribution storage means for storing a probabilistic distribution of the object state estimated at each time;
Target state storage means for storing the change vector at each time;
With
Assuming the three-dimensional position and size of the moving object as the target state of the moving object, the moving object is imaged at a specific time interval using one or a plurality of imaging devices, and a plurality of images obtained by the imaging A moving object tracking method used in an apparatus for tracking a moving object based on image data,
A target state prediction step of predicting a plurality of target states of the moving object at each time; and
An image acquisition step of acquiring the image data;
A silhouette image creating step for creating a silhouette image obtained by extracting an area in which the moving object is copied from the image data;
Considering the target state obtained by the target state prediction means as the initial state, the silhouette image created by the silhouette image creation means, the three-dimensional structure of the real world stored in the three-dimensional environment information management means, and the three-dimensional Using the internal parameters and external parameters of the imaging device stored in the environmental information management means, the target state distribution is estimated,
A target state distribution estimating step of updating the target state distribution at the previous time stored in the target state distribution storage means to the target state distribution at the current time;
Based on the target state distribution having the maximum probability based on the target state distribution at the current time stored in the target state distribution storage means, and based on the target state having the maximum probability at the current time and the target state having the maximum probability at the previous time A target state calculating step of calculating a change vector and storing the calculated change vector in a target state storage means;
Have
The target state distribution estimation step includes
The first probability distribution update number B for the target state distribution storage means is compared with the last probability distribution update number N for the target state, and based on the comparison result, a change type is selected according to an arbitrary probability from a change type prepared in advance. A step to choose;
Selecting a moving body to adapt and change the change type;
Changing the target state of the selected moving object based on the selected change type and the comparison result of the probability distribution update count B and the probability distribution update count N;
A virtual imaging means having the same external parameters and internal parameters as the imaging device, and considering an image created by projecting the changed target state as a simulation image simulating the silhouette image;
A probabilistic first condition likelihood of a target state based on a prior knowledge condition relating to the three-dimensional structure, and a probabilistic second condition likelihood of the target state based on a prior knowledge condition relating to overlapping of moving objects;
Calculating a third condition likelihood based on a comparison between the silhouette image and the simulation image, and considering a product of the first to third condition likelihoods as the likelihood of the changed provisional target state; ,
Determining a target state predicted n times according to a result of whether or not the provisional target state is adopted;
When the number of updates is equal to the probability distribution calculation number B, calculating a state distribution based on the target state obtained by the B times of updating,
Determining whether the number of moving objects currently being tracked is equal to 0 based on the state distribution based on the target state obtained by the B-th update;
If the number of moving objects is not equal to 0 and if the update count exceeds the probability distribution update count B plus the probability distribution update count N, the N stored in the last N updates Considering that all target states can occur with equal probability, and assuming the target state predicted n times as a target state distribution;
Having
The moving body tracking method characterized by this. The moving body tracking method according to claim 3,
The target state of the moving object is regarded as a set of ellipsoid models, the position of each moving object is a position (x, y) on an arbitrary plane, the size of the moving object is the radius r and height h of the ellipsoid model, Represented by
Storing a plurality of combinations of values of these parameters x, y, r, h,
The moving body tracking method characterized by this.   5. A moving body tracking program according to claim 3, wherein the moving body tracking method is described as a computer program executable by a computer.   5. A recording medium in which the moving body tracking method according to claim 3 is described as a computer program executable by a computer and the computer program is recorded.