JP4759447B2 - Moving object tracking device, moving object tracking method, and recording medium storing a program describing the method - Google Patents

Description

  The present invention relates to a technique for tracking a state such as a three-dimensional position and a size of a moving object (moving object including an object, an animal, and a human) using one or a plurality of image input devices (for example, a camera or the like). .

  In the field of computer vision, there are many studies on tracking moving objects such as people using information from multiple cameras. For example, the following research results have been realized.

  A silhouette image is prepared by observing a target from a plurality of viewpoints and extracting a target region (ie, silhouette) on the image. Further, the three-dimensional structure of the space to be tracked in advance (hereinafter referred to as three-dimensional environment information) and the internal parameters and external parameters (also referred to as camera parameters) of all the cameras used for tracking are obtained by measuring in advance. Keep it.

  The silhouette image is obtained by performing a background difference method or an inter-frame difference method on a captured image, and setting the luminance value of an area where the target of the captured image is “1” and the luminance values of other areas to “ This is a binary image expressed by “0” (that is, an actual difference image). A camera calibration method for calibrating internal parameters of a camera and obtaining a three-dimensional position and orientation is also widely known (for example, see Non-Patent Document 1).

  Then, the person model is arranged in the three-dimensional environment, and the person model is arranged by a virtual camera (for example, a virtual imaging device constructed by software) having the internal parameters and external parameters of the camera calculated previously. By capturing a scene, a silhouette image can be simulated.

In addition, a method of tracking a person under a general environment by using an ellipsoid for a human model and comparing the generated simulation image with a silhouette image (hereinafter referred to as an ellipsoid model tracking method) has been proposed. (For example, refer nonpatent literature 2).
Z. Zhang, “Aflexible new technology for camera calibration”, IEEE, Transactions on Pattern Analysis and Machine Intelligence, 2000 (2000), vol. 22, no. 11, p. 1330-1334. Hirokazu Kato, Atsushi Nakazawa, Seiji Iguchi, “Real-time human tracking using an ellipsoid model”, Journal of Information Processing Society of Japan, 1999 (November 1999), vol. 40, No11, p. 4087-4096.

  As described above, in the method of tracking a target based on information acquired by a camera, when a plurality of persons are tracked, the persons overlap each other on an image captured by the camera due to the approach of the persons. (Ie occlusion) is a problem.

  The occlusion problem can be reduced by integrating information obtained from multiple viewpoints. However, in the above ellipsoid model tracking method, since multiple cameras track independently, information integration of multiple cameras is not performed, and there is a problem that tracking is impossible when people approach each other. is doing.

  The ellipsoid model tracking method has a problem that the number of people to be tracked is fixed from the beginning, so that it cannot cope with the exit and appearance of a person from the tracking space.

  Furthermore, since a processing amount proportional to the number of people to be tracked is required, there is a problem that if the number of people to be tracked increases, the processing cost increases and the real-time performance of the tracking processing is lost.

  Since the ellipsoidal model used for generating the simulation image is given a fixed height and a fixed radius, it is impossible to deal with individual differences in the size of the person. In other words, when tracking a person having a body whose size is significantly different from the size of the ellipsoid model, there is a problem that a large error occurs in the measured value, and as a result, tracking also fails. Yes.

  The present invention has been made to solve the above-described problems of the prior art, and a plurality of images existing in a three-dimensional space from images sent from a plurality of imaging devices that capture a plurality of moving objects. A moving body tracking apparatus, a moving body tracking method for stably estimating a moving state of each moving body by sequentially estimating a target state representing the position and size of each moving body at each time, and a recording medium storing a program describing the method. It is to provide.

In order to solve the above-mentioned problems, the invention according to claim 1 is based on the assumption that the three-dimensional position and size of a moving object is the target state of the moving object, and uses one or a plurality of imaging devices. A moving object tracking device that images a moving object at a specific time interval and tracks the moving object based on a plurality of image data obtained by the imaging, and obtains the moving object at a previous time with respect to the moving object. A target state prediction unit that predicts a current target state using the target state and a change vector of the target state, an image acquisition unit that acquires the image data, and an area in which the moving object is copied from the image data The silhouette image created by the silhouette image creating means for creating the silhouette image and the target state obtained by the target state predicting means as the initial state, and the three-dimensional silhouette image created by the silhouette image creating means And real world three-dimensional structure stored in the boundary information management means, imaging relates to a three-dimensional position and orientation information of the internal parameters and an imaging device itself of the image pickup apparatus relating to the stored focal length, image center on the three-dimensional environment information managing means A target state distribution estimating unit that estimates a target state distribution using an external parameter of the apparatus, and updates the target state distribution at the previous time stored in the target state distribution storage unit to the target state distribution at the current time; Based on the target state distribution at the current time stored in the target state distribution storage means, the target state having the maximum probability is calculated, and the target state having the maximum probability at the current time and the target state having the maximum probability at the previous time are calculated. Based on the target state calculation means for calculating the change vector based on the result and storing the calculated change vector in the target state storage means; Three-dimensional environment information management means for storing the dimensional structure and internal parameters and external parameters of the imaging apparatus; target state distribution storage means for storing a probabilistic distribution of the target state estimated at each time; and the change at each time Object state storage means for storing a vector , wherein the object state distribution estimation means selects a change type according to an arbitrary probability from a change type prepared in advance, and a moving object that adaptively changes the change type And a virtual imaging means having the same external parameters and internal parameters as the imaging device, the means for changing the target state of the selected moving object based on the selected change type Means for considering an image created by projecting a target state as a simulation image simulating the silhouette image; A probabilistic first condition likelihood of a target state based on a prior knowledge condition regarding a dimensional structure, a probabilistic second condition likelihood of a target state based on a prior knowledge condition regarding overlapping of moving objects, and the silhouette A third condition likelihood based on a comparison between the image and the simulation image, and calculating the product of the first to third condition likelihoods as the likelihood of the changed target state; And means for estimating a target state distribution according to the number of times the simulation image has been updated to the provisional state using the likelihood of the target state .

According to a second aspect of the invention, in the moving object tracking device of claim 1 Symbol placement, the target state of said moving object is regarded as a set of ellipsoidal model, the position on the position of any plane of the body (x, y ), The size of the moving object is represented by the radius r and the height h of the ellipsoid model, and a plurality of combinations of the values of the parameters x, y, r, h are stored.

The invention according to claim 3 considers the three-dimensional position and size of the moving object as the target state of the moving object, and images the moving object at a specific time interval using one or a plurality of imaging devices. A moving body tracking method for tracking a moving object based on a plurality of image data obtained by the imaging , using a target state and a change vector of the target state obtained at a previous time for the plurality of moving objects A target state prediction step for predicting a target state at the present time, an image acquisition step for acquiring the image data,
A silhouette image creating step for creating a silhouette image obtained by extracting an area in which the moving object is copied from the image data, and a target state obtained by the target state predicting unit is regarded as an initial state, and is created by the silhouette image creating unit. Silhouette image, real-world 3D structure stored in 3D environment information management means, internal parameters of the imaging device related to focal length and image center stored in 3D environment information management means, and 3D of the imaging apparatus itself The target state distribution is estimated using the external parameters of the imaging device related to the position and orientation information , and the target state distribution at the previous time stored in the target state distribution storage unit is updated to the target state distribution at the current time An object state with the maximum probability based on the state distribution estimation step and the object state distribution at the current time stored in the object state distribution storage means The target state calculation that calculates the change vector based on the target state having the maximum probability at the current time and the target state having the maximum probability at the previous time, and stores the calculated change vector in the target state storage means A step, a pre-measured real-world three-dimensional structure, a three-dimensional environment information management step for storing internal parameters and external parameters of the imaging device, and a stochastic distribution of the target state estimated at each time select the target state distribution storing step, the target state storage step of storing the change vectors for each time has the target state distribution estimating step, the variation type in accordance with any probability from a prepared variation type of Selecting a moving object to be changed by adapting the change type, and based on the selected change type. And changing the target state of the selected moving object, and an image created by projecting the changed target state by the virtual imaging means having the same external parameters and internal parameters as the imaging device, the silhouette A simulation image simulating an image, a probabilistic first condition likelihood of a target state based on prior knowledge about the three-dimensional structure, and a target state based on prior knowledge about overlapping of moving objects A probabilistic second condition likelihood and a third condition likelihood by comparing the silhouette image and the simulation image are calculated, and a product of the first to third condition likelihoods is calculated for the changed target state. The target state distribution corresponding to the number of times of updating to the provisional state of the simulation image using the likelihood of the target state changed and the likelihood of the changed target state is estimated. And a step of determining .

According to a fourth aspect of the present invention, in the moving object tracking method according to the third aspect , the target state of the moving object is regarded as a set of ellipsoidal models, and the position of each moving object is a position (x, y) on an arbitrary plane. The size of the moving object is represented by the radius r and the height h of the ellipsoid model, and a plurality of combinations of the values of the parameters x, y, r, h are stored.

The invention according to claim 5 is a recording medium, wherein the moving object tracking method according to claim 3 or 4 is described as a computer program executable by a computer, and the computer program is recorded. And

According to the first and third aspects of the present invention, it is possible to acquire a target state distribution of a plurality of moving objects, and to acquire a change vector of the moving object based on the target state distributions.

Moreover, obtaining a three-dimensional stochastic likelihood of states of interest based on prior knowledge about the structure, probabilistic likelihoods of states of interest based on a prior knowledge of the overlap of the moving object between the likelihood by comparison of the silhouette image and the simulated image it can.

According to the invention of the claim 2 and 4, the position of each moving object position on an arbitrary plane (x, y), the size of the moving object can be stored radius r and height h of the ellipse body model.

According to the invention of claim 5 , the moving body tracking method according to claim 3 or 4 can be described as a computer program.

As described above, according to the first and third aspects of the present invention, the change vector of the moving object is calculated based on the images acquired from the plurality of imaging devices, and the movement of the plurality of moving objects is calculated based on the change vector of the moving object. It can be tracked stably.

Further , even in situations where moving objects are very close to each other by using the probability (that is, the result of integrating image information from a plurality of imaging devices) obtained by comparing the stochastic likelihood, the silhouette image, and the simulation image. , Moving body can be tracked stably.

According to the second and fourth aspects of the present invention, since x, y, r, and h can be stored, it is possible to perform tracking according to changes in the number of moving objects having different sizes and the number of moving objects that are dynamically tracked. Further, even if tracking is performed according to a change in the number of moving objects having different sizes or the number of moving objects to be dynamically tracked, the processing time can be kept constant and real time performance can be ensured.

According to invention of Claim 5 , the recording medium which recorded the computer program which mounted the moving body tracking method can be provided.

  These can contribute to the computer vision field.

  The moving body tracking apparatus in this embodiment is demonstrated based on FIG.

  The moving object tracking device includes a target state prediction unit 11, an image acquisition unit 12, a silhouette image creation unit 13, a target state distribution estimation unit 14, a target state calculation unit 15, and a three-dimensional environment information management unit (for example, managing three-dimensional environment information). Database) 21, target state distribution storage means 22, and target state storage means 23.

  The target state prediction means 11 determines the target state at the current time using the state of the tracking target obtained at the previous time (hereinafter simply referred to as the target state; the three-dimensional position and size of the tracking target) and the change vector of the target state. Predict.

  The image acquisition means 12 is means for acquiring image data taken at each time, and includes, for example, an imaging device such as a digital camera or a video camera. The moving object tracking device in the present embodiment may include one or a plurality of imaging devices. Further, the image data may be acquired from information management means (for example, a database) that stores image data taken at each time.

  The silhouette image creating means 13 creates a silhouette image obtained by extracting only the region where the object is copied from the acquired image data.

  The target state distribution estimation unit 14 regards the target state obtained by the target state prediction unit 11 as an initial state, and uses the silhouette image created by the silhouette image creation unit 13 and the three-dimensional environment information management unit 21 to The probabilistic distribution of the state is estimated, and the target state distribution at the previous time stored in the target state distribution storage unit 22 is updated to the target state distribution at the current time.

  The target state calculation means 15 uses the target state distribution storage means 22 in which the target state distribution at the current time is stored, calculates the target state that has the maximum probability, and calculates a change vector from the target state at the previous time. The change vector from the target state stored in the target state storage unit 23 is updated.

  The three-dimensional environment information management means 21 includes real-world three-dimensional structure information measured in advance and internal parameters (for example, a digital camera, hereinafter simply referred to as a camera) of the installed image acquisition means 12 (for example, Parameters such as focal length and image center) and external parameters of the image acquisition means 11 (for example, the three-dimensional position and orientation information of the camera itself) are stored. For example, the three-dimensional environment information management means 21 may include a database that stores three-dimensional structure information, internal parameters, and external parameters. The three-dimensional environment information management unit 21 may store three-dimensional structure information, internal parameters, and external parameters in a storage unit (for example, a hard disk device or a nonvolatile memory).

  The target state distribution storage unit 22 stores the probabilistic distribution of the target state estimated by the target state distribution estimation unit 14. For example, the probabilistic distribution of the target state estimated in the memory (or storage area) included in the target state distribution storage unit 22 or the hard disk device may be stored.

  The target state storage unit 23 stores a change vector between the target state having the maximum probability calculated by the target state calculation unit 15 and the target state at the previous time. For example, the change vector may be stored in a memory (or storage area) included in the target state storage unit 23 or a hard disk device.

  The conditions regarding the moving body tracking apparatus in this embodiment are demonstrated based on FIG. In the following description, the same reference numerals as those in FIG. 1 are omitted.

  FIG. 2 illustrates a plurality of persons walking on a plane (in FIG. 2, persons H1, H2,... Hk; k is a natural number) to be tracked (that is, a moving object), and positions and postures of p (p is a natural number) units. Is shown for an example in which tracking is performed using a fixed video camera (that is, cameras C1, C2,... Cp). In addition, the cameras C1 to Cp acquire images synchronously at the same shooting interval. Note that when acquiring images synchronously at the same shooting interval, for example, a synchronization device may be connected to or mounted on the cameras C1 to Cp.

  In general, there is a non-target object (non-target object B in FIG. 2) that is not a tracking target in space, and the three-dimensional structure data of the non-target object is CAD (Computer-Aided Design) by hand. It can be measured and acquired in advance using data creation, a range finder, or three-dimensional reconstruction from camera images. This three-dimensional structure data shall be converted into a coordinate system in which the XY plane coincides with the floor surface and the height direction coincides with the Z axis.

  Further, calibration of internal parameters of all (p) cameras arranged, and the three-dimensional position (x, y, z) and posture (φ, θ, γ) are the same coordinate system as the acquired three-dimensional structure data. (For example, refer nonpatent literature 1). These three-dimensional structure data and the three-dimensional positions and orientations of all the cameras are stored in advance in the three-dimensional environment information management means 21.

  A shape model of a person (moving body) in the present embodiment will be described with reference to FIG.

The time in the following description indicates the number of times (or the number of orders) sampled at a specific sampling interval Δt. The initial time t ₀ indicates the time when moving object tracking is started (that is, the time t is “0”). The previous time is the time between the initial time t ₀ and the current time (ie, the current time), and the time when the moving object was tracked immediately before the current time (ie, the time being processed; eg, t) (ie, t -1).

  In FIG. 3, the ellipsoid e is considered as a model approximating the shape of a person, the position of the person is a two-dimensional coordinate value (x, y) on the floor plane, and the size of the person is the radius r and height of the ellipsoid e. Represented by h. Further, the MCMC (Markov Chain Monte Carlo) method, which is a kind of the stochastic statistical state distribution estimation method, is used for the target state distribution estimation means 14.

A target state S representing the states of a plurality of persons is represented by a vector M (ie, a person model) representing the position and size of each person. The number of dimensions of the target state S is determined by the maximum number K of people that can be tracked. That is, since the vector M is a four-dimensional vector, it becomes a 4K-dimensional vector. For example, state S _t at time t is expressed by the following equation.

A person model M _i representing the position, size, and shape of the person i constituting the state distribution S is a four-dimensional vector represented by the following expression.

Here, x _i and y _i are two-dimensional coordinate values (x, y) indicating the position of the person i on the floor plane, and r _i and h _i are ellipsoids when the person is approximated by an ellipsoid. Represents the parameter radius r and height h. Note that x _i , y _i , r _i , and h _i may be stored and accessed in, for example, a storage unit (for example, a memory or an external storage device) provided in the moving object tracking device.

As described above, the present embodiment, by estimating the target state S _t at each time t, sequentially estimating the state of the plurality of persons, and performs tracking.

  The processing procedure of the moving body tracking method of the present embodiment will be described based on the flowchart in FIG. In the following description, the same reference numerals as those in FIG. 1 are omitted.

First, when the process is started, the target state prediction unit 11 predicts the target state at the current time (S301). This is a linear prediction process performed using the target state calculated at the previous time and the change vector of the target state. Assuming that the target state obtained at the previous time is S _t−1 and the change vector is a 4K-dimensional vector V _t−1 , the predicted value S ′ _{t, 0} of the target state at the current time can be expressed by the following equation. it can.

However, t in the index of S ′ represents time t. The portion “0” indicates that the predicted value update count is “0”. In the estimation of the target state distribution (S304), which will be described later, the predicted value is regarded as S ′ _{t, 0} as the initial value, and then the target state distribution is obtained by performing a plurality of updates.
That is, “0” in the index indicates an initial value whose update count is “0”.

It should be noted that immediately after the start of moving body tracking (initial time t ₀ ), since the target state and the change vector of the target state at the previous time are unknown, all elements of M _i are set to “0” and S ₀ is initialized. Do.

Next, using the image acquisition unit 12, the synchronous photography image data Im _1, Im _2, acquires ... Im _p from the camera C1, C2 ... Cp (S302) .

Next, the silhouette images Sil ₁ , Sil ₂ ,... Sil _p are obtained from the image data Im ₁ , Im ₂ ,... Im _p acquired from the cameras C 1, C ₂ ,. Create (S303). Note that the silhouette image is a binary image in which the luminance value of the area where the object of the image taken as shown in FIG. 5 is “1” and the other area is “0”. It is an example of a silhouette image in which an input image and an image 502 are created. Such an image can be easily generated by using a well-known method such as a background difference method or an inter-frame difference method.

Next, the target state S ′ _{t, 0} predicted in step S301 is set as the initial value of the target state, and S ′ is sequentially updated to estimate the probabilistic distribution of the target state (S304).

  Here, a processing procedure for estimating the probabilistic distribution of the target state will be described with reference to FIG.

  The estimation of the probability distribution of the target state is performed by the state update process a plurality of times (that is, (B + N) times). The target state obtained by the first B updates is discarded, and the target state obtained in the last N times is stored as a state distribution. We consider that all these N states can occur with equal probability.

  First, the update count n is initially set to “0” and the process is started.

  In step S401, a change type is selected. Note that the change type represents a type of processing to change the state, and four types (first type is person addition, second type is person deletion, third type is person position change, 4 types consist of person size change).

  The method of selecting the change type is as follows. First, when no person is tracked as the current state (all M is a “0” vector), the first type is always selected. In other cases, the probability that each type is selected is set arbitrarily in advance, and the four types are selected probabilistically.

Next, in step S402, an object to be changed is selected. Now, assuming that the number of updates is n, the temporary change state S ^* is calculated using the target state S ′ _{t, n−1} updated _n−1 .

At this time, the state of all the objects is not changed by one update process, but only the state of one object is changed by each update process. First, when the change type is addition of a person, a random selection is made from M _i that is currently a “0” vector (that is, M _i that is not tracked). M _i (i.e., M _i being tracked) not "0" vector otherwise performing randomly selected from a.

Next, in step S403, a temporary person model M ′ _i is calculated according to each of the first to fourth types of processing selected from the selected person model M _i . Hereinafter, processing by type will be described.

  In the first type (person addition), the following processing is performed.

Since the person model M _i is a “0” vector, it is necessary to newly set a value for the person model M _i . For example, the value can be set as follows.

  It is assumed that the range of positions that can be taken in the person's space is known by the three-dimensional environment information management means 21, and this range is defined as follows.

  The parameters r and h representing the size of an ellipsoid that approximates a person are based on empirical prior knowledge about the size of the person (for example, there is no person with a height exceeding 3 m (meters)). The range of possible values can be similarly limited. Here, this range is defined as follows.

Under the conditions as described above, M ′ _i can be generated by generating uniform random numbers for the respective parameters of x, y, r, and h regarding the ellipsoid.

  Note that the first type (person addition) processing indicates that a person has newly entered the tracking range. In addition, if information on a place to enter / exit (for example, the location of a door) is obtained in advance within the tracking range, it is also possible to consider only the vicinity as an entrance / exit area.

  In the second type (person deletion), the following processing is performed.

All elements are considered “0” and M ′ _i is a “0” vector. That is, the second type (person deletion) represents that the person being tracked has disappeared from the tracking range.

  In the third type (person position change), the following processing is performed.

Of the selected M _i elements, a temporary position (x ′ _i , y ′ _i ) changed from the position (x _i , y _i ) of the person on the two-dimensional plane according to the following equation is calculated. .

However, δ _x and δ _y are one-dimensional Gaussian noises, and δ _x and δ _y can be experimentally determined to arbitrary values. Using this provisional position (x ′ _i , y ′ _i ), M ′ _i is defined as in the following equation.

  In the fourth type (person size change), the following processing is performed.

Among the selected elements of M _i , the provisional shape and size of the ellipsoid changed from (r _i , h _i ) representing the radius r and height h of the ellipsoid approximating the person according to the following equation: The parameters (r ′ _i , h ′ _i ) are calculated.

However, δ _r and δ _h are one-dimensional Gaussian noises, and δ _r and δ _h can be experimentally determined to arbitrary values. Using the provisional shape and size parameters (r ′ _i , h ′ _i ) of this ellipsoid, M ′ _i is defined by the following equation.

Using the M ′ _i generated in each of the above types, the target state S ^* that is temporarily changed is generated. For example, S ^* changes the selected person model M _i out of the states S ′ _{t, n−1} obtained by the (n−1) th update if the number of updates is nth at the current time t. The generated M ′ _i is replaced.

Next, using the temporarily changed target state S ^* , an image simulating the silhouette images Sil ₁ , Sil ₂ ,... Sil _p created in the creation of the silhouette image (S 303) (ie, the simulation image Sim ₁ , Sim ₂ ,... Sim _p ) are created (S 404). The method of creating the simulation image (S404) is as follows.

First, a virtual scene simulating the real world is constructed by combining the three-dimensional environment information management means 21 and the target state. In the three-dimensional environment information management means 21, three-dimensional point information (or a three-dimensional point group having a coordinate system in which the floor surface is the XY plane and the height direction is the Z axis) (or Polygon data). A plurality of three-dimensional ellipsoidal models based on the provisional target state S ^* calculated in step S403 can be arranged in the three-dimensional structure.

  Next, a virtual scene is projected by a virtual camera using internal parameters (for example, focal length, image center, etc.) and external parameters (camera three-dimensional position and orientation) arranged in the real world (for example, The non-patent literature 2) creates a simulation image. The internal parameters and external parameters are stored in the three-dimensional environment information management unit 21.

During projection, a silhouette image Sil ₁ , Sil ₂ ,... Sil _{p is} obtained by setting a binary image in which the value of the ellipsoidal region on the image is “1” and the value of the other region is “0”. Sim ₁ , Sim ₂ ,... Sim _p can be created.

  The above is the method for creating a simulation image.

Next, in order to determine how likely this provisional target state S ^* is, the likelihood is calculated (S405). Calculation of likelihood, prior knowledge silhouette image created in the determination and step S303 of certainty of the subject state itself by Sil _1, Sil _2, ... simulated generated by Sil _p and step S404 image Sim _1, Sim _2, ... calculated based on Sim _p comparison.

  The following two types of determination can be considered for the determination of the likelihood of the target state itself based on the prior knowledge conditions (that is, the calculation of the probabilistic likelihood (that is, the conditional likelihood) based on the condition).

The first determination is a determination using a restriction based on a three-dimensional environment. By using the three-dimensional environment information management means 21, the existence probability can be weighted according to the position (x, y) of the person state. In the present embodiment, it is assumed that the person exists on the floor plane, and the condition is that the person does not climb up on a stationary object that exists in the space other than the target. Assumes. Under the assumption, for example, by preparing a penalty function P (S ^* ) represented by the following equation that is inversely proportional to the height of the three-dimensional structure at the position (x, y) where the person exists, It is possible to weight the existence probability.

However, h (M _i ) is the Z value of the three-dimensional structure at the position (x _i , y _i ) in the element of M _i , and α is a constant that can be arbitrarily set experimentally.

  The second determination is a determination based on a condition that the persons being tracked do not overlap in the three-dimensional space. In other words, since the person is approximated by an ellipsoid, the condition that the individual ellipsoids do not overlap (that is, the distance between the centers of two different ellipsoids) satisfies that the sum of the radii of the two ellipsoids is greater. There is a need.

In order to prevent two persons from overlapping each other in a three-dimensional space, a penalty function E (S) according to the distance R expressed by the following equation and the distance between the centers of ellipsoids representing the two persons is represented. ^* ) Is set.

Here, x _n and y _n indicate the position x coordinate and y coordinate of the person n.

  The function E that gives a penalty when the distance between the ellipsoids becomes short can be set as the following equation, for example.

  However, β is a constant and can be arbitrarily set experimentally.

Then, silhouette images Sil ₁ created by the creation of the silhouette image _{(S303), Sil 2, ...} Sil p and simulation image Sim _1, Sim _2, ... comparison of Sim _p will be described. For example, it can be calculated by the following evaluation formula (that is, the conditional likelihood is calculated).

  Here, (k, l) indicates k rows and 1 column components (pixels) of the image.

  In the above formula, information of all p cameras is integrated, and evaluation is performed using information from a plurality of viewpoints.

Finally, the likelihood L of the provisional state S ^* is calculated by a product operation like the following equation.

  The above processing is the likelihood calculation (S405).

Next, in step S406, it is determined using the likelihood calculated in step S405 whether the provisional state S ^* is accepted and updated or rejected. This determination is made based on the calculation of the acceptance rejection determination probability A. The acceptance refusal judgment probability A can be calculated by the following equation, for example.

Here, assuming that the update is the nth time, the likelihood L ′ represents the likelihood of S ′ _{t, n−1} updated _{the n−} 1th time. That is, the current state is always accepted if the likelihood is higher than the previous state, and otherwise, the provisional state S ^* is adopted with the probability of L / L ′ (that is, the acceptance rejection determination probability A).

Further, if the provisional state S ^* is adopted, it is assumed that S ′ _{t, n} is equal to S ^* (S ′ _{t, n} = S ^* ), and if adoption is rejected, S ′ _{t, n} Is assumed to be equal to S ′ _{t, n−1} (S ′ _{t, n} = S ′ _{t, n−1} ), and the process proceeds to the condition determination step of step S407.

  In step S407, it is determined whether the update process has been performed a prescribed number of times. If the number of updates n exceeds B + N times represented by the sum of constants B and N that can be arbitrarily determined, the process is terminated.

If n exceeds B times, the obtained state S ′ _{t, n} is stored in the target state distribution storage unit 22. That is, N states are stored in the target state distribution storage unit 22, and it is considered that all the N states can occur with equal probability, and these N states are set as a target state distribution.

  The process described above is step S304.

Next, in step S305, the target state having the maximum probability is calculated from the target state distribution obtained in step S304. Since it is assumed that N pieces of object state constituting the target state distribution occurs at all equal probability, the target state S _t with the maximum probability it can be calculated by the following equation.

  Further, in order to predict the target state at the next time by the target state prediction means 11, a 4K-dimensional change vector V is obtained by the following equation.

However, since the above equation cannot be calculated at the initial time t ₀ , all elements of V are set to “0”. _St and V calculated in this way are stored in the target state storage means 23.

  In step S306, it is checked whether or not to end tracking. If the tracking is completed, the process is terminated. If the tracking is still continued, the process of predicting the state at the next time (S301) is performed. For example, in order to check whether or not to end the tracking, when tracking stop instruction means (for example, a keyboard device provided in the moving object tracking device) is provided and there is a tracking stop instruction from the tracking stop instruction means You may stop tracking.

  As described above, by repeating the processing in the present embodiment, it is possible to sequentially estimate the states of a plurality of persons whose number of persons dynamically changes.

  It is to be noted that a part or all of the functions of each means in the apparatus shown in FIG. 1 can be configured by a computer program and the program can be executed using the computer to realize the present invention, FIG. 4 and FIG. It is needless to say that the processing procedure shown in FIG. 6 is configured by a computer program and the program can be executed by the computer, and the program for realizing the function by the computer can be read by a computer-readable recording medium, for example, FD (Floppy (registered trademark) Disk), MO (Magneto-Optical disk), ROM (Read Only Memory), memory card, CD (Compact Disk), DVD (Digital Versatile Disk), removable device And recorded in a disk, or stored, it is possible or distribute. It is also possible to provide the above program through a network such as the Internet or electronic mail.

  Further, a computer program describing a method related to the moving object tracking device may be mounted so as to access a memory or an external storage device that stores input / output data required for the method related to the moving object tracking device.

  In the present embodiment, in the process of step S405, the target state prediction unit 11 is used as an example for comparison between a silhouette image and a simulation image. However, an evaluation function (for example, a simple product operation) that takes similarity between images. The same can be achieved using.

  Further, in the present embodiment, in the process of step S405, as an example, the expression 8 weighted in the height direction is used as the environmental penalty function, but a function that can express the weight of the position of the person can be used separately. Is natural.

  In this embodiment, in the process of step S405, Expression 10 is used as a penalty function related to the approach of a person as an example, but it is natural that a function that gives a penalty when people overlap can be used separately.

[Example]
In this embodiment, the moving object tracking device described above is used to process images obtained from two video cameras (camera C1 and camera C2), and two persons entering the tracking range one by one are detected. The traced result will be described based on the processing flow shown in FIG.

  First, when the process is started, the target state is linearly predicted (S301).

  Next, an image is acquired from the camera (S302). FIG. 7 is a part of an image sequence acquired from two cameras. The upper row is images 701 to 706 acquired from the camera C1, and the lower row is images 707 to 712 acquired from the camera C2.

  Next, a silhouette image is created from the obtained image (S303). A silhouette image created using the background subtraction method is shown in FIG. As in FIG. 7, the upper row is a silhouette image 801 to 806 created from an image acquired from the camera C1, and the lower row is a silhouette 807 to 812 created from an image acquired from the camera C2. Each silhouette image in FIG. 8 corresponds to each image shown in the same position in FIG.

  Next, the target state distribution is estimated in step S304. That is, in step S304, a simulation image simulating the state is created while sequentially updating the state, and the simulation image is evaluated.

  In FIG. 9, the upper part is a silhouette image 901, and the lower part is a part of a simulation image simulating the upper silhouette image (simulation images indicated by reference numerals 902 to 904). In this way, a simulation image is created while changing to various states, and a target state distribution is created by likelihood calculation and acceptance rejection calculation.

  In step S305, a target state that currently takes the maximum probability is obtained from the target state distribution calculated in step S304, and this is output as the target state at the current time. At the same time, a change vector of the target state at the previous time and the current time is obtained.

  FIG. 10 shows an image representing the target state taking the maximum probability calculated. FIG. 10 shows images 1001 to 1006 from the camera in the upper stage, images 1007 to 1012 in which the middle stage simulates a target state that takes the maximum probability, and takes the maximum probability assuming that the camera is installed on the ceiling as the overhead view. It is the images 1013 to 1018 simulating the target state.

  This shows that after the person H1 and the person H2 have entered the tracking range after a short time, they can clearly trace the situation of passing each other and finally leaving the tracking range. In addition, it can be confirmed that the number of people to be tracked is dynamically changed as 0 → 1 → 2 → 0, but the person is stably tracked while appearing and disappearing.

  In step S306, if the tracking is to be ended up to the current time, the process is ended, otherwise, the process returns to the prediction of the target state in step S301.

  By repeating the processing in the above embodiment, stable tracking can be performed even in a scene where two persons approach each other and pass each other. That is, this indicates that information of a plurality of cameras is integrated.

  In addition, although the number of people to be tracked is dynamically changing, it is shown that tracking can be performed in response to the appearance and disappearance of a person.

  Furthermore, even if the number of persons to be dynamically tracked changes as described above, there is no problem that the total processing amount increases, and real-time performance is ensured.

  Also, stable tracking is possible without giving parameters in advance to two persons with different body shapes.

  These features are because the information of a plurality of cameras can be integrated to estimate the size of the body.

  Although the present invention has been described in detail only for the specific examples described above, it is obvious to those skilled in the art that various changes and modifications are possible within the scope of the technical idea of the present invention. Such variations and modifications are naturally within the scope of the claims.

For example, as a modification of the present embodiment, an input unit that can input values that can be arbitrarily set experimentally (for example, the values of δ _x , δ _y , δ _r , δ _h , α, and β described above) is provided. Based on the input value from the input means, the above-described processing related to moving object tracking may be performed. That is, a value is input to the moving object tracking device by input means including a keyboard device and the above-described processing related to moving object tracking is performed based on the input value.

The block diagram of the moving body tracking device in this embodiment. The figure which shows the conditions regarding the moving body tracking device in this embodiment. The three-dimensional model figure used with the moving body tracking device in this embodiment. The flowchart which shows the process sequence of the moving body tracking method in this embodiment. The figure which shows the silhouette image corresponding to the input image in this embodiment. The flowchart which shows the process sequence regarding estimation of object state distribution in this embodiment. The figure which showed a part of image sequence acquired from the camera in a present Example. The figure which showed the silhouette image created from a part of image sequence acquired from the camera in a present Example. The figure which showed a part of image which simulated the silhouette image based on the silhouette image and the prediction result in a present Example. The figure which showed the image which simulated the object state calculated from a part of input image and the input image in this Example, and the bird's-eye view of it.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 ... Target state prediction means 12 ... Image acquisition means 13 ... Silhouette image creation means 14 ... Target state distribution estimation means 15 ... Target state calculation means 21 ... Three-dimensional environment information management means 22 ... Target state distribution storage means 23 ... Target state storage Means 501 ... Input image 502, 801 to 812, 901 ... Silhouette image 701 to 712, 1001 to 1006 ... Image from camera 902 to 904 ... Simulation image 1007 to 1018 ... Image simulating the target state B ... Non-target object C1 , C2, Cp ... camera H1, H2, Hk ... person e ... ellipsoid

Claims (5)

Assuming the three-dimensional position and size of the moving object as the target state of the moving object, the moving object is imaged at a specific time interval using one or a plurality of imaging devices, and a plurality of images obtained by the imaging A moving object tracking device that tracks the moving object based on image data,
Target state prediction means for predicting a target state at the present time using a target state obtained at a previous time and a change vector of the target state for the plurality of moving objects;
Image acquisition means for acquiring the image data;
Silhouette image creating means for creating a silhouette image obtained by extracting an area in which the moving object is copied from the image data;
Considering the target state obtained by the target state prediction means as the initial state, the silhouette image created by the silhouette image creation means, the three-dimensional structure of the real world stored in the three-dimensional environment information management means, and the three-dimensional Estimate the target state distribution using the internal parameters of the imaging device related to the focal length and image center stored in the environmental information management means and the external parameters of the imaging device related to the 3D position and orientation information of the imaging device itself, and the target Target state distribution estimating means for updating the target state distribution at the previous time stored in the state distribution storage means to the target state distribution at the current time;
Based on the target state distribution at the current time stored in the target state distribution storage means, the target state having the maximum probability is calculated, and the target state having the maximum probability at the current time and the target state having the maximum probability at the previous time are calculated. A target state calculation unit that calculates a change vector based on the calculated change vector and stores the calculated change vector in a target state storage unit;
Three-dimensional environment information management means for storing the real-world three-dimensional structure measured in advance and the internal and external parameters of the imaging device;
Object state distribution storage means for storing a probabilistic distribution of the object state estimated at each time;
Object state storage means for storing the change vector at each time ,
Means for selecting a change type according to an arbitrary probability from a change type prepared in advance, the target state distribution estimating means;
Means for adaptively changing the moving type to change the moving type;
Means for changing the target state of the selected moving object based on the selected change type;
A virtual imaging unit having the same external parameters and internal parameters as the imaging device, a unit that regards an image created by projecting the changed target state as a simulation image simulating the silhouette image;
A probabilistic first condition likelihood of a target state based on a prior knowledge condition relating to the three-dimensional structure, and a probabilistic second condition likelihood of the target state based on a prior knowledge condition relating to overlapping of moving objects;
Means for calculating a third condition likelihood by comparing the silhouette image and the simulation image, and considering a product of the first to third condition likelihoods as the likelihood of the changed target state;
Means for estimating a target state distribution according to the number of times the simulation image is updated to the provisional state using the likelihood of the changed target state ;
A moving body tracking device comprising: In the moving object tracking device of claim 1 Symbol placement,
The target state of the moving object is regarded as a set of ellipsoid models, the position of each moving object is a position (x, y) on an arbitrary plane, the size of the moving object is the radius r and height h of the ellipsoid model, Represented by
Storing a plurality of combinations of values of these parameters x, y, r, h,
A moving body tracking device characterized by that. Assuming the three-dimensional position and size of the moving object as the target state of the moving object, the moving object is imaged at a specific time interval using one or a plurality of imaging devices, and a plurality of images obtained by the imaging A moving body tracking method for tracking a moving body based on image data,
A target state prediction step of predicting a current target state using a target state and a change vector of the target state obtained at a previous time for the plurality of moving objects;
An image acquisition step of acquiring the image data;
A silhouette image creating step for creating a silhouette image obtained by extracting an area in which the moving object is copied from the image data;
Considering the target state obtained by the target state prediction means as the initial state, the silhouette image created by the silhouette image creation means, the three-dimensional structure of the real world stored in the three-dimensional environment information management means, and the three-dimensional Estimate the target state distribution using the internal parameters of the imaging device related to the focal length / image center stored in the environmental information management means and the external parameters of the imaging device related to the 3D position and orientation information of the imaging device itself, and A target state distribution estimating step of updating the target state distribution at the previous time stored in the state distribution storage means to the target state distribution at the current time;
Based on the target state distribution having the maximum probability based on the target state distribution at the current time stored in the target state distribution storage means, and based on the target state having the maximum probability at the current time and the target state having the maximum probability at the previous time A target state calculating step of calculating a change vector and storing the calculated change vector in a target state storage means;
A three-dimensional environment information management step for storing the real-world three-dimensional structure measured in advance and the internal and external parameters of the imaging device;
A target state distribution storage step for storing a probabilistic distribution of the target state estimated at each time;
A target state storage step for storing the change vector at each time ,
The target state distribution estimating step selects a change type according to an arbitrary probability from previously prepared change types;
Selecting a moving body to adapt and change the change type;
Changing the target state of the selected moving object based on the selected change type;
A virtual imaging means having the same external parameters and internal parameters as the imaging device, and considering an image created by projecting the changed target state as a simulation image simulating the silhouette image;
A probabilistic first condition likelihood of the target state based on prior knowledge regarding the three-dimensional structure, and a probabilistic second condition likelihood of the target state based on prior knowledge regarding overlapping of moving objects,
Calculating a third condition likelihood by comparing the silhouette image and the simulation image, and considering a product of the first to third condition likelihoods as the likelihood of the changed target state;
Estimating a target state distribution according to the number of times of updating to the provisional state of the simulation image using the likelihood of the changed target state ;
A moving body tracking method characterized by comprising: The moving body tracking method according to claim 3 ,
The target state of the moving object is regarded as a set of ellipsoid models, the position of each moving object is a position (x, y) on an arbitrary plane, the size of the moving object is the radius r and height h of the ellipsoid model, Represented by
Storing a plurality of combinations of values of these parameters x, y, r, h,
The moving body tracking method characterized by this. 5. A recording medium in which the moving body tracking method according to claim 3 is described as a computer program executable by a computer, and the computer program is recorded.