JP5340228B2 - State estimation device, state estimation method, and program - Google Patents

Description

  The present invention relates to a state estimation device, a state estimation method, and a program for estimating a future state change based on a past history of a state change of an observation target.

  There is a particle filter as a method for estimating a future state change based on a state change of an observation target. In typical particle filter processing, for example, (1) the prior probability distribution of the state at the next time is estimated based on information indicating the state of the past observation target up to the current time, and (2) the estimated prior The posterior probability distribution is obtained by comparing the estimated state of the observation target indicated by the probability distribution with the actual state of the observation target observed at the next time. By repeating these two processes, it is possible to sequentially estimate the state of the observation object that dynamically changes.

Dan MIKAMI, Kazuhiro OTSUKA, Junji YAMATO: Memory-based Particle Filter for Face Pose Tracking Robust under Complex Dynamics, pp.999-1006, Proc.IEEE CVPR, 2009.

Mikami, Kazuhiro Otsuka, Koji Yamato, “Particle Filter Based on Memory of State History for Robust Face Posture Tracking” MIRU2009 OS3-1

However, in the particle filter described in Non-Patent Documents 1 and 2, a template is created using a front face as a template to be observed at the time of initialization. The front face template is used when estimating the posterior probability distribution. A front face template is deformed by rotation / translation specified by particles representing a prior probability distribution and an image observed at the next time, and a posterior probability distribution is created based on the comparison result.
For this reason, for example, when the posture of the observation target changes greatly, the portion that has been seen until now gradually disappears, and the portion that has not been seen until then appears, so that the appearance of the observation target changes greatly. As a result, there is a problem that the matching with the front face template may not be sufficiently matched with the actual situation of the observation target.

  The present invention solves the above-described problem, and is a state estimation device and a state estimation method capable of performing highly accurate state estimation corresponding to a change in posture even when the posture of an observation target changes greatly And to provide a program.

  In view of the above-described problems, the state estimation device according to the present invention includes state information indicating the position and orientation of an observation target at a past time, appearance information indicating an appearance form of the observation target at the time, and the time at the time. Based on the history storage device that stores a table for associating temporal reproduction probabilities indicating the probability that the observation target is in the state indicated by the state information, the past state information and the temporal reproduction probability, A state prior probability distribution representing a state indicating the position and orientation of the object by a plurality of particles, and an appearance prior probability representing an appearance form of the observation object in the future based on the appearance information associated with the state information Observations related to the observation target measured by an external observation device and a prior probability distribution prediction unit that creates a distribution Information is input, the observation information is collated with the appearance prior probability distribution, the observation likelihood is calculated based on the collation error, and the state prior probability distribution is weighted according to the observation likelihood and the posterior probability is calculated. A posterior probability distribution estimation unit that creates a distribution, and a history storage unit that updates and stores the state information indicated by the posterior probability distribution in the history storage device.

  Further, the state estimation device described above is characterized in that the temporal reproduction probability is calculated based on a change in the state of the observation target over time in the past and the similarity of the state information immediately before. .

  In the above state estimation device, the prior probability distribution prediction unit selects the state information similar to the particles representing the state prior probability distribution, and based on the appearance information associated with the state information, The appearance prior probability distribution is created.

  Further, in view of the above-described problems, the state estimation method according to the present invention includes state information indicating the position and orientation of an observation target at a past time, appearance information indicating the appearance form of the observation target at the time, and the time A state estimation method in a state estimation device including a history storage device that stores a table for associating temporal reproduction probabilities indicating the probability that the observation target is in the state indicated by the state information, the past state information and Based on the temporal reproduction probability, creating a state prior probability distribution representing a state indicating the position and orientation of the future observation target by a plurality of particles, and based on the appearance information associated with the state information, Prior probability distribution predictions for creating an appearance prior probability distribution that represents the appearance of the observation object in the future And observation information on the observation target measured by an external observation device, and the observation information and the appearance prior probability distribution are collated to calculate an observation likelihood based on the collation error, and the observation A posteriori probability distribution estimating step for creating a posteriori probability distribution by performing weighting according to likelihood on the condition prior probability distribution, and history storage for updating and storing the state information indicated by the posteriori probability distribution in the history storage device And a step.

  Further, in view of the above-described problems, the program according to the present invention causes a computer to store state information indicating the position and orientation of an observation target at a past time, appearance information indicating an appearance form of the observation target at the time, Based on the history storage means for storing a table for associating each of the temporal reproduction probabilities indicating the probability that the observation target becomes the state indicated by the state information at the time, the past state information and the temporal reproduction probability, A state prior probability distribution that represents a state indicating the position and orientation of the observation target by a plurality of particles is created, and an appearance that represents the appearance of the observation target in the future based on the appearance information associated with the state information Prior probability distribution prediction means for creating a prior probability distribution, the view measured by an external observation device Input observation information about the object, collate the observation information with the appearance prior probability distribution, calculate an observation likelihood based on the collation error, and weight the state prior probability distribution according to the observation likelihood. A program for functioning as a posterior probability distribution estimating means for creating a posterior probability distribution, and a history accumulating means for updating and accumulating the state information indicated by the posterior probability distribution in the history storage means, To do.

  According to the present invention, it is possible to predict the simultaneous prior probability distribution of the posture and appearance even when the posture of the observation target changes greatly and the appearance changes accordingly.

It is a block diagram which shows an example of a structure of the state estimation apparatus concerning 1st Embodiment of this invention. It is a figure which shows an example of the log | history table concerning 1st Embodiment of this invention. It is a figure which shows an example of the time reproduction probability table concerning 1st Embodiment of this invention. It is a flowchart which shows an example of the state estimation method concerning 1st Embodiment of this invention. It is a flowchart which shows an example of the prior distribution estimation method concerning 1st Embodiment of this invention. It is a flowchart which shows the other example of the prior distribution estimation method concerning 1st Embodiment of this invention. It is a block diagram which shows an example of a structure of the state estimation apparatus concerning 2nd Embodiment of this invention. It is a flowchart which shows an example of the state estimation method concerning 2nd Embodiment of this invention.

[First Embodiment]
The state estimation device 1 according to the present embodiment is a device that functions as a particle filter, holds a history of past state estimation results for predicting a prior probability distribution, and a probability that this past history appears again ( The prior probability distribution is estimated based on the temporal reproduction probability. In addition, the state estimation device 1 according to the present embodiment accumulates, as history information, a set of a state estimation value to be observed and an observation value (hereinafter referred to as an appearance value) indicating an appearance form observed by a camera or the like, Based on the accumulated history information, by obtaining the simultaneous prior probability distribution of the state estimation value and appearance value, it is possible to perform robust tracking against a large change in appearance (appearance form of observation target) due to a large change in posture To do.
The state estimation device 1 according to the present embodiment represents the prior probability distribution and the posterior probability distribution by a set of particles in the framework of a particle filter. Many particles are generated at positions having a high probability distribution in the state space.

Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing an example of a state estimation device 1 according to the first embodiment of the present invention.
As shown in FIG. 1, the state estimation device 1 includes a prior probability distribution prediction unit 11, a posterior probability distribution estimation unit 12, an estimation result calculation unit 13, a tracking stability determination unit 14, a history storage device 15, a shape model storage device 16, A history storage unit 17, an estimation target time setting unit 18, and an end determination unit 19 are included.

The history storage device 15 holds a history table D1 in which a state estimation value indicating a past state estimation result from the tracking start time 1 to the current time T and an appearance value to be described later are associated for each time.
The history table D1 is the time t, the state estimate vector x _t is the past state estimation value, the appearance values A _t corresponding to the time t, a table associating the time, figure an example It is shown in 2. Incidentally, referred to as history information _{X t} in combination with state estimate vector _{x t} and appearance values _{A t,} where the historical information _{X t = (x t, A} t) shown as.
Here, the state estimate vector x _t, a state estimation value at a certain time t, the state estimate vector x _T, a state estimation value at the current time T. The state estimated value vector _xt is a state estimated value calculated by the posterior probability distribution estimating unit 12 described later.
Appearance values A _t is information indicating the appearance of a tracked object (exterior form), for example, can be used a set of pixel values in the region of the tracking target as appearance.

Further, the history storage device 15 stores a temporal reproduction probability table D2 in which the probability that the past history appears again (temporal reproduction probability) _pt and the time are associated with each other. This temporal reproducibility (TRP) indicates the probability that the posture and position (state) of the past time t will appear again, and is defined as a function on the time axis. In other words, the temporal reproduction probability is a model of the probability that a past state will occur again in the future.
The temporal reproduction probability table D2 includes, for example, information that is a set of a time t and a temporal reproduction probability p _{t at} which the posture at the time t appears again as shown in FIG.

  The shape model storage device 16 stores shape model information D3 indicating the approximate shape of the tracking target. This shape model information D3 is information that can determine the appearance of the target area on the current input image and determine the appearance, for example, by rotating or translating by a predetermined amount calculated based on the estimated state value. is there.

The prior probability distribution prediction unit 11 reads out history information X _t = (x _t , A _t ) from the past history table D1 stored in the history storage device 15 and calculates a temporal reproduction probability p _t . It should be noted that the temporal reproduction probability p _t is the time when the current state is known (for example, even if the current state indicated by the image input to the posterior probability distribution estimation unit 12 is input to the prior probability distribution prediction unit 11). It is calculated by the prior probability distribution prediction unit 11. The prior probability distribution prediction unit 11 estimates the prior probability distribution based on the history information X _t = (x _t , A _t ) and the temporal reproduction probability p _t . In the particle filter, the prior probability distribution is represented by a set of particles. That is, the prior probability distribution prediction unit 11 regards the state at the next time that can occur from the current state as a large number of particles (particles), and represents the prior probability distribution by a set of particles.

  For example, for the particle generation method in the prior probability distribution estimation unit 11, it is assumed that Non-Patent Documents 1 and 2 can be used, and that past states repeatedly appear based on past history. Prior probability distribution estimation unit 11 generates particles representing the prior distribution based on sampling according to the temporal reproduction probability.

  In the present invention, the prior probability distribution is expressed by the following equation (1).

Here, X ^ _{1: T} = {(x ^ ₁ , A ^ ₁ ), ..., (x ^ _T , A ^ _T )} is a state estimated value vector x ^ ₁ and an appearance value A ^ _t. Represents the history information from time 1 to time T. A ^ _{1: T} = {A ^ ₁ ,..., A ^ _T } represents history information from appearance time 1 to time T of the appearance value. Further, the function π () represents that the simultaneous prior distribution of the posture and appearance in the future from the current time T by Δt time is predicted based on the past history of the posture and appearance, and Δt.
Here, z _t is an observation vector at time t, and z _{1: T} = {z ₁ ,..., Z _T } represents a sequence from time 1 to time T of the observation vector. Further, x _t represents a history state estimated value vector at time t, and state estimated value vector x _{1: T} = {x ₁ ,..., X _T } is from time 1 of the history state estimated value vector. A series up to time T is represented. Furthermore, _{X t} denotes the history information at the time t, the history information _{X 1} including a state estimate vector _{x t} and appearance values _{A t: T = {X 1} , ···, X T} is the history information A series from time 1 to time T is represented.
That is, the prior distribution after Δt time of the current time T is defined as being obtained from the history information X _{1: T} up to the present and the predicted time ahead Δt.

Further, the above equation (1) shows the simultaneous prior distribution of the state estimated value (information indicating the position and orientation of the object) and the appearance value (information indicating the appearance indicating the appearance form of the object), and the future position and orientation. Conditional probability of appearance under the condition given state estimated value vector _{xT + Δt} , history estimated value vector history x ^ _{1: T} indicating past position and posture, and past appearance value A ^ _{1: T} Is defined as follows.

The first half of the above equation (2) corresponds to a prior probability distribution (state prior probability distribution) of position and orientation. Here, it is assumed that the state estimated value vector x _{T + Δt} indicating the position and posture after the time _Δt does not depend on the appearance value A ^ _{1: T} , that is, the posture dynamics does not depend on the appearance value. The latter half corresponds to an appearance conditional probability distribution (appearance prior probability distribution) under a condition given a posture prior distribution.
Here, the appearance value A _{T + Δt} after the time _Δt is the state estimated value vector x _{T + Δt} indicating the position and posture after the time _Δt , the history x ^ _{1: T of the} state estimated value vector indicating the past position and posture, and the appearance. It is assumed that it depends on the value A _{1: T} and not on the predicted time ahead Δt.

Also, in order to determine the conditional probability of appearance in the latter half of the equation (2), it is assumed that the change in appearance value is mainly caused by the change in posture. However, there is no one-to-one deterministic relationship between posture and appearance, and it is assumed that these relationships include uncertainty. This is due to the following observations.
First, due to the rotation of the object, the part of the object that has been visible until now disappears and the part that has not been visible appears, and the appearance changes greatly. However, the appearance also changes due to illumination changes and non-rigid deformation. In addition, it is difficult to model the appearance change itself. This is because the appearance is very high-dimensional and changes in a complex manner.
Based on the above assumptions and observations, this embodiment adopts an approach that captures the relationship between posture and appearance as a kind of probability distribution. Here, when a certain posture is given, the appearance corresponding to the posture has a distribution having a certain width, and when given a certain appearance, the posture corresponding to the appearance has a distribution having a certain width. In this embodiment, such an uncertainty can be handled by a Bayesian filter framework.

Next, the prior probability distribution estimation unit 11 will be described in more detail. The prior probability distribution estimation unit 11 is a particle set {(x ^{* (1)} , A ^{* (1)} ),..., (X ^{* (N)} , A ^{* (N)} )} is generated using those past histories. N is the number of particles.
Here, the simultaneous prior distribution of the above equation (2) is subject to the prior distribution π ( _{xT + Δt} | x ^ _{1: T} , Δt) of the position and orientation of the object and the condition of the appearance under which it is given Focusing on the fact that it is represented by the product of the probability distribution π (A _{T + Δt} | x _{T + Δt} , x _{1: T} , A _{1: T} ), the prior probability distribution estimation unit 11 makes particles representing the prior distribution of position and orientation. And a two-stage configuration in which an appearance is obtained for each particle. Here, it is assumed that the current time is T and a state after Δt time from the present is predicted. Further, it is assumed that posture history x ^ _{1: T} and appearance history A ^ _{1: T} from time 1 to T are already held.

<Step 1 Generation of prior probability distribution of position and orientation>
As described above, the prior probability distribution estimation unit 11 calculates the prior probability distribution of the position and posture at the time T + Δt using the history of the point estimated value and the past posture estimated value (state estimated value) of the posterior probability distribution at the current time. A particle set {x ^{* (1)} ,..., X ^{* (N)} } to represent is obtained. This corresponds to the first half of equation (2).

<Step 2 Generation of appearance prior probability distribution>
The prior probability distribution estimating unit 11 generates conditional particles π (A _{T + Δt} | x _{T + Δt} , x ^ _{1: T} , A ^ in the latter half of the equation (2) in order to generate particles representing the prior probability distribution of appearance. _1: Random sampling from _T ). As a result, the appearance {A ^{* (1)} ,... Corresponding to each of the particles {x ^{* (1)} ,..., X ^{* (N)} } representing the prior probability distribution of the position and orientation obtained in step 1. ., A ^{* (N)} } is generated.
Here, what is required is an appearance distribution p (A | x ^{* (i)} ) under the condition where the position and orientation x ^{* (i)} are given. Prior probability distribution estimation unit 11 calculates the appearance distribution by mixing appearance values A ^ _t in past postures x ^ _t similar to x ^{* (i)} .
As a result, the prior probability distribution estimation unit 11 acquires the posture x ^ _t similar to x ^{* (i)} from the history, and calculates the difference in appearance due to the difference between x ^{* (i)} and the past similar posture x ^ _t. In order to fill, geometric deformation is applied to the past appearance value A ^ _t . Here, the appearance prior distribution p (A | x ^{* (i)} ) is defined as the following equation (3).

Here, f (•) is a geometric deformation, δ (•) is a Dirac delta function, and w (t) is a weight determined by x ^{* (i)} and x ^ _t . Α is

とするための正規化項である。

Is a normalization term for

Then, the prior probability distribution estimation unit 11 generates a particle set representing the prior probability distribution of appearance by random sampling of past appearances weighted by w (t) based on the equation (3).
Here, w (t) is assumed to be a history selection probability. This is defined as a function on the time axis, and is higher as the posture x ^{* (i) of} the appearance estimation target is similar to the posture x ^ _t (t = 1,..., T) included in the past history. Indicates the value.
Here, the uncertainties in the relationship between the state representing the position and posture and the appearance are expressed using a set of past histories, and the random sampling from the history using the history selection probability corresponds to any posture. We expect to see a distribution of appearances. The specific procedure of step 2 is shown below.

<Step 2-1 History sampling>
Prior probability distribution estimation unit 11 samples a history of postures similar to particles x ^{* (i)} . For example, the prior probability distribution estimation unit 11 samples one past posture x ^ _t , _t to w (t) for x ^{* (i)} based on the history selection probability. This is because a large number of samples x ^{* (i)} , (i = 1,..., N) are used, so that a variety of appearances can be expressed sufficiently even with one appearance for each particle. is there.
Here, as the history selection probability w (t), a function proportional to the reciprocal of the Euclidean distance between the posture x ^{* (i} ) of the appearance estimation target and the posture x ^ _t (t <T) in the history at each time t is used. .

  Where β is

とするための正規化項である．

Is a normalization term for.

<Step 2-2 Estimation of appearance>
In consideration of the difference between the sampled posture x ^ _t and the posture x ^{* (i)} of the appearance estimation target particle, A ^* _t is subjected to the geometrical transformation of Equation (5) to obtain A ^{*. (I)} is determined.

  Here, it is assumed that a minute change in appearance due to a minute posture change can be predicted well by geometric deformation.

  For example, the prior probability distribution estimation unit 11 (1) obtains a temporal reproduction probability, (2) performs temporal sampling from the temporal reproduction probability, and (3) refers to a state estimation value at a sampled past time. In addition, a prior distribution is obtained by superimposing kernel distributions taking uncertainty into consideration. The prior probability distribution estimation unit 11 obtains a set of particles representing the prior probability distribution by performing spatial sampling from the prior probability distribution.

The posterior probability distribution estimation unit 12 uses the prior probability distribution estimated by the prior probability distribution estimation 11 and its observation likelihood to estimate the posterior probability distribution. This posterior probability distribution estimation unit 12 represents the posterior probability distribution by a set of particles weighted by the observation likelihood in the framework of the particle filter.
Therefore, the posterior probability distribution estimation unit 12 obtains a matching error for each set of particles representing the prior probability distribution, and calculates an observation likelihood based on the matching error. That is, the posterior probability distribution estimation unit 12 obtains the observation likelihood using the appearance value A calculated by the prior probability distribution estimation unit 11.

Then, tracking is performed while predicting the weighted average based on the observation likelihood of all particles as the next state. Therefore, a weight is obtained from the observation likelihood for each particle to obtain a weighted particle set.
This posterior probability distribution can be expressed by the following equation (6).

Here, _{k t} is a normalization term. Moreover, p (z _t | x _t ) is an observation likelihood.

Note that the observation likelihood represents whether or not the prior probability distribution calculated by the prior probability distribution prediction unit 11 is close to the position estimated from the observation result. For example, it is possible to determine how likely the state of each particle is by comparing image data obtained by capturing the tracking target with a camera and a template that is deformed (rotated / translated) according to the state of each particle. . The posterior probability distribution estimation unit 12 calculates an observation likelihood based on observation information (such as an input image) related to the tracking target and an a priori probability distribution. The posterior probability distribution estimation unit 12 obtains a matching error for each set of particles representing the prior probability distribution, and calculates an observation likelihood based on the matching error. Here, _xt is an estimated posture value (state parameter) representing a position or posture parameter at time t. p _t is a temporal reproduction probability indicating the probability that the posture at the past time t appears again.
Further, the posterior probability distribution estimation unit 12 calculates a set of particles weighted with a score corresponding to the calculated observation likelihood, and outputs this to the estimation result calculation unit 13 as a posterior probability distribution.

  The estimation result calculation unit 13 calculates the estimation result information D4 using the weighted particle set representing the posterior probability distribution calculated by the posterior probability distribution estimation unit 12. The estimation result calculation unit 13 obtains the estimation result information D4 based on, for example, a weighted average of particles representing the posterior probability distribution or the posture of a particle having the maximum observation likelihood. In other words, for example, the estimation result calculation unit 13 obtains, as the estimation result information D4, an average of the particle states weighted by the weight of the observation likelihood.

In addition, the estimation result calculation unit 13 reads shape model information D3 indicating the approximate shape of the tracking target from the shape model storage device 16, and also stores the state estimation value vector x _t from the history estimation device history table D1 from the history storage device 15. reading, on the basis of the shape model information D3 and state estimate vector x _t, and calculates the appearance values a _t corresponding to the time t indicating the estimation result of the appearance.
The estimation result calculation unit 13, for example, the subject of the geometric model information D3, the shape model is rotated, the translation by a variation amount according to the state estimation value vector x _t in the history table D1, on the current input image Identify the area of the and determine the appearance. Here, the appearance is information representing the appearance of the object. For example, a set of pixel values in the region to be tracked can be used as an appearance.

  The method of obtaining the appearance by the estimation result calculation unit 13 depends on the method. In the technique in which a set of pixels in the tracking target area is used as the appearance, the estimation result calculation unit 13 sets the pixel value in the area as the appearance. In addition, in the method of selecting an attention point (feature point) by the process of extracting a feature point indicating a predetermined feature from the region to be tracked and making it an appearance, the estimation result calculation unit 13 causes the specified tracking target to be identified. The same processing is performed on the area to obtain the appearance.

The tracking stability determination unit 14 determines the reliability of the current estimation result information D4. When it is determined that the current estimation result information D4 is reliable, the tracking stability determination unit 14 moves the process to the estimation target time setting unit 18. On the other hand, if the tracking stability determination unit 14 determines that the current state estimation result information D2 is unreliable, the tracking stability determination unit 14 deletes the current state estimation result information D2 without storing it in the history storage device 15. That is, when it is determined that the tracking stability determination unit 14 is reliable, the tracking stability determination unit 14 instructs the estimation target time setting unit 18 to set the target time.
The reliability determination by the tracking stability determination unit 14 can be obtained by performing threshold processing on the weight of the largest weighted particle. When the weight is larger than the predetermined threshold value R, the tracking stability determination unit 14 determines that the state estimation result information D2 is reliable.

The history storage unit 17 stores the estimation result information D4 output from the tracking stability determination unit 14 in the history storage device 15 in association with the time of the history table D1.
The estimation target time setting unit 18 increments the tracking target time.
The end determination unit 19 determines whether or not to continue state estimation. When continuing, the end determination unit 19 instructs the prior probability distribution prediction unit 11 to perform state estimation again.

Next, an example of the state estimation method according to the present embodiment will be described with reference to FIG. FIG. 4 is a diagram for explaining an example of the state estimation method according to the present embodiment. In the present invention, as the state of the estimation target, in addition to the position and orientation, the prior probability distribution of both including the appearance is estimated based on the past history.
As shown in FIG. 4, the state estimation method of the present invention includes a prior probability distribution prediction step ST10 for predicting a prior probability distribution, a posterior probability distribution estimation step ST20 for estimating a posterior probability distribution, and an estimation result calculating step for calculating an estimation result. ST30, tracking stability determination step ST40 for determining tracking stability, history storage step ST50 for storing history, target time setting step ST60 for setting state estimation target time, and end for determining whether or not the processing has ended Determination step ST70.

In the prior probability distribution prediction step ST10, the prior probability distribution prediction unit 11 calculates the prior probability distribution with reference to the past history table D1 and the temporal reproduction probability table D2 stored in the history storage device 15.
The prior probability distribution prediction unit 11 predicts the prior probability distribution of the state. Here, the state includes a position, posture, and appearance. In the particle filter framework, the probability distribution is represented by a set of particles. Therefore, the prior probability distribution prediction unit 11 obtains a set of particles representing the simultaneous prior distribution of position, posture, and appearance.

  In the posterior probability distribution estimation step ST20, the posterior probability distribution estimation unit 12 calculates the posterior probability distribution using the prior probability distribution estimated by the prior probability distribution prediction unit 11 in the prior probability distribution estimation step ST10 and its observation likelihood. calculate. For example, the posterior probability distribution estimation unit 12 calculates an illuminance error for each set of particles represented by the prior probability distribution, calculates an observation likelihood based on the illuminance error, and based on a weight corresponding to the observation likelihood for each particle. The weighted particle set is obtained as a posterior probability distribution.

In the estimation result calculation step ST30, the estimation result calculation unit 13 calculates a posture estimation result (state estimation result value D5) using a weighted particle set representing the posterior probability distribution calculated by the posterior probability distribution estimation unit 12. At the same time, an appearance estimation result (appearance estimation result value D6) is calculated. The estimation result information D4 described above includes a state estimation result value D5 and an appearance estimation result value D6.
Specifically, the estimation result calculation unit 13 calculates a posture estimation result (state estimation result value D3) based on a weighted average of particles representing the posterior probability distribution or the posture of particles having the maximum likelihood. . Further, the estimation result calculation unit 13 calculates an appearance estimation result (appearance estimation result value D6) from the rough shape of the tracking target and the posture estimation value stored in the shape model storage device 16.

  In the tracking stability determination step ST40, the tracking stability determination unit 14 determines the reliability of the current state estimation result. When it is determined that the current state estimation result is reliable, the tracking stability determination unit 14 moves the process to the state estimation target time setting step ST60. On the other hand, if it is determined that it is not reliable, the tracking stability determination unit 14 moves the process to the history accumulation step ST50.

In the history storage step ST50, the state estimation result value D5 and appearance estimation result value D6 outputted from the tracking stability determination unit 14, for each time t, respectively, the state estimation value of the history table D1 vector x _t and appearance values A _t is stored in the history storage device 15.

In the estimation target time setting step ST60, the estimation target time setting unit 18 increments the tracking target time.
In the end determination step ST70, the end determination unit 19 determines whether or not to continue the state estimation. When continuing, it transfers to step ST10.

<Advance Probability Distribution Prediction Method 1>
Next, an example of the prior probability distribution prediction step ST10 shown in FIG. 4 will be described in more detail with reference to FIG. FIG. 5 is a flowchart for explaining the steps included in the prior probability distribution prediction step ST10.
In step ST101, the prior probability distribution prediction unit 11 obtains one particle representing the prior probability distribution of the posture by random sampling from the prior probability distribution of the posture. Note that the method of acquiring the prior probability distribution of the posture by the prior probability distribution prediction unit 11 can use a random walk from the immediately preceding posterior probability distribution, which is frequently used in the framework of a particle filter. However, the method is not limited to this, and a method using the temporal reproduction probability described in Non-Patent Documents 1 and 2 may be used.

In step ST102, the prior probability distribution prediction unit 11 obtains an appearance using the past history for one particle representing the prior probability distribution of the posture obtained in step ST101. For example, a method of using an appearance of a similar past posture is conceivable.
In step ST103, the prior probability distribution prediction unit 11 adds the set of the posture obtained in step ST101 and the appearance estimated in step ST102 to the set of particles.
Prior probability distribution prediction unit 11 repeats steps ST101 to ST103 until a predetermined number of particles is reached.

<Advance Probability Distribution Prediction Method 2>
Next, another example of the prior probability distribution prediction step ST10 shown in FIG. 4 will be described in more detail with reference to FIG. FIG. 6 is a flowchart for explaining the steps included in the prior probability distribution prediction step ST10.
In step ST121, the prior probability distribution prediction unit 11 acquires one particle representing the prior probability distribution of the posture by random sampling from the prior probability distribution of the posture, and adds the random noise to the prior probability distribution of the posture. Let the particles represent.
The prior probability distribution prediction unit 11 can obtain the prior probability distribution of the posture by a random walk from the immediately preceding posterior probability distribution, as is often used in the framework of particle filters. Further, the prior probability distribution prediction unit 11 may calculate the prior probability distribution of the posture on the assumption that the past history appears again according to the temporal reproduction probability as in Non-Patent Documents 1 and 2.

  In the case of Non-Patent Document 1, (1) one past history is selected according to the temporal reproduction probability, and (2) Gaussian noise is added to the selected past history posture to obtain the particle posture. However, the temporal reproduction probability is calculated based on the similarity between the predicted time ahead (information indicating what time ahead is predicted), the past posture history, and the previous posture estimated value. It is assumed that the history following the posture similar to the current state is likely to appear again, and the temporal reproduction probability takes a large value.

In step ST122, the prior probability distribution prediction unit 11 selects a posture history similar to the particle posture.
In step ST123, the prior probability distribution prediction unit 11 geometrically deforms the appearance of the history based on the posture of the particle and the selected posture of the history, and estimates the appearance of the particle. The geometric deformation may be a three-dimensional rotation.
In step ST124, the prior probability distribution prediction unit 11 adds the set of the posture obtained in step ST121 and the appearance obtained in step ST123 to the particle set representing the prior distribution.

  Note that in the prior distribution prediction procedure step ST122, the prior probability distribution prediction unit 11 probabilistically selects a posture similar to the particle posture. For example, the prior probability distribution prediction unit 11 can follow a probability proportional to the inverse of the difference between the particle posture and the past history posture. Alternatively, the prior probability distribution prediction unit 11 may set the difference between the particle posture and the past history posture as d and the probability according to the following equation.

  Here, σ is a predetermined constant and indicates the variance of the selected probability. When the variance σ is large, the probability that a past history having a large difference from the particle posture is selected increases.

  Further, according to the present invention, the prior probability distribution estimation unit 11 estimates the appearance of all particles in addition to the position and orientation in step ST10. Therefore, in step ST20, the posterior probability distribution estimation unit 12 collates the estimated appearance with the input image to obtain a collation error, calculates an observation likelihood based on the collation error, and calculates it for each particle. Use weights. This set of weighted particles represents the posterior probability distribution.

Here, as an example of the appearance, a set of pixel values in the tracking target region, or a set of coordinates of the attention point of the tracking target and the pixel values thereof can be considered.
In addition, as a method for calculating a matching error, a sum of absolute errors of pixel values, a g (e) = e / (e + 1) obtained by applying a robust function to an absolute error e of pixel values, and the like can be used. The present invention is not limited to this.

In step ST30, the estimation result calculation unit 13 obtains an estimation result based on the posterior probability distribution. First, the estimation result calculation unit 13 obtains a posture estimation result by weighted average using the posture and weight held by each particle.
In calculating the appearance estimation result, the estimation result calculation unit 13 obtains the position of the target on the screen by rotating and translating the shape model of the tracking target according to the posture estimation result, and estimates the appearance of the region. As a result appearance. The appearance of the object changes as the posture changes.
In the present invention, a known shape model stored in the shape model storage device 16 is rotated and translated in accordance with the posture estimation result, thereby specifying an appearance (appearance form) that can be observed at present. Obtain an appearance from the identified tracked area.

  As an example of appearance, a set of pixel values of a tracking target region, a set of coordinate values and pixel values as a result of detecting a certain point of interest from the tracking target region, and the like can be considered.

In step ST40, the tracking stability determination unit 14 determines whether tracking is stable. This can be determined by the maximum likelihood of the particle and the like. If the maximum likelihood is larger than the threshold value, it is considered that the tracking is stable. Other indexes such as particle dispersion, kurtosis, and skewness may be used. If the tracking is stable, the process proceeds to step ST50. Otherwise, the process proceeds to step ST60.
In step ST50, the history storage unit 17 stores the history. The history in the present invention is a set of past posture estimated values (state estimated values) and appearance values obtained in step ST30. This is stored in the history storage device 15.

[Second Embodiment]
The state estimation device 2 according to the present embodiment tracks the face posture of a person shown in a video shot by a camera. As shown in FIG. 7, the state estimation device 2 is configured to further include a camera 21 and an image processing unit 22 in the state estimation device 1 described above. Here, the details of each component are denoted by the same reference numerals as those in the first embodiment, and the description thereof is omitted.
The tracking of the face state is to estimate the position and orientation (posture) of the face on the screen. In the present invention, the position and orientation x of the tracking target (a vector including the center (m _x , m _y ) of the face on the screen and the three-axis center rotation (r _x , r _y , r _z )) and the appearance value A The information indicates the state of the estimation target.

  In step ST901, the state estimation device 1 performs initialization. In the initialization step, the image processing unit 22 detects the front face by the front face detection method based on the tracking symmetrical face image data captured by the camera 21. The face at the time of initialization is assumed to be position (0, 0) and posture (0, 0, 0).

  In step ST902, the image processing unit 22 reads a frame image based on the image data output from the camera 21. The image processing unit 22 reads the frame image of the video file when performing the tracking process on the video file. Note that the image processing unit 22 reads one frame of the camera when performing the tracking process on the video captured by the camera 21.

  In step ST903, the prior probability distribution prediction unit 11 predicts (calculates) the prior probability distribution. In this embodiment, as a first stage of prior distribution prediction, the prior probability distribution prediction unit 11 obtains a prior probability distribution of posture. The prior probability distribution of the posture can be obtained from various methods such as a random walk from the immediately preceding state or a method using the past history of Non-Patent Document 1. A set of particles representing the prior distribution of posture can be obtained by random sampling from the prior distribution.

Then, as the second stage of the prior distribution prediction, the prior probability distribution prediction unit 11 estimates the appearance of each particle.
(2.1) In the second stage, first, the prior probability distribution prediction unit 11 selects a past history of postures similar to the postures of the appearance estimation target particles. This selection is performed stochastically based on the history selection probability. The history selection probability is defined on the past time axis based on the similarity of posture. In the present embodiment, it is assumed that the value is proportional to the inverse of the difference between the particle posture and the history posture at each time.

(2.2) Next, the prior probability distribution prediction unit 11 estimates (calculates) the particle posture by geometric deformation of the appearance of the history in consideration of the difference between the posture of the particle and the posture of the selected history. To do. In this embodiment, three-dimensional rotation is used as the geometric deformation.
By the above two-stage processing, the prior probability distribution prediction unit 11 can obtain a set of particles representing the simultaneous prior distribution of posture and appearance.

  In step ST904, the posterior probability distribution estimation unit 12 estimates the posterior probability distribution. In the present invention, each particle holds an estimated appearance value. Therefore, in the posterior probability distribution estimation, a collation error is obtained by collating the appearance of each particle with the input image. An observation likelihood is obtained based on the matching error, and a set of particles weighted by the observation likelihood represents a posterior probability distribution.

  The posterior probability distribution estimation unit 12 collates the appearance with the input image using, for example, a value based on the difference in pixel values at the appearance attention point (feature point). Specifically, the posterior probability distribution estimation unit 12 calculates according to the following formula.

ただし、推定されたアピアランスｖ_ｋ（ｘ_ｉ，ｙ_ｉ）は、パーティクルｋにおける特徴点（ｘ_ｉ，ｙ_ｉ）の画素値を表し、ｚ（ｘ_ｉ，ｙ_ｉ）は入力画像の座標（ｘ_ｉ，ｙ_ｉ）での画素値を表す。Ｎはアピアランスの注目点（画素）の総数である。

However, the estimated appearance v _k (x _i , y _i ) represents the pixel value of the feature point (x _i , y _i ) in the particle k, and z (x _i , y _i ) represents the coordinates (x _i , y _i ) represents the pixel value. N is the total number of appearance attention points (pixels).

In step ST905, the estimation result calculation unit 13 calculates an estimation result. In the present embodiment, the estimated value of the posture is obtained by a weighted average of the posture. The appearance is obtained from the estimated posture value and the human face average model. First, the face area is specified by rotating and translating the human face average model according to the estimated posture value. Then, the appearance is acquired by performing the attention point extraction process on the identified face area.
In this embodiment, about 250 points of interest are selected from the face region from the vicinity of the edge and the maximum and minimum points of the pixel value, and the coordinates of the point of interest and the set of the pixel values are set as appearances.

In step ST906, the tracking stability determination unit 14 determines whether tracking is performed stably. If stable, the process proceeds to step ST907. Otherwise, the process proceeds to step ST908.
In step ST907, the history storage unit 17 stores the history. That is, the set of posture estimated values (state estimated values) and appearances obtained in step ST905 are accumulated as a history.
In step ST907, the estimation target time setting unit 18 increases the processing target time by one.
In step ST909, the end determination unit 19 determines whether the video has ended. If not completed, the process returns to step ST902.

  As described above, by estimating the prior probability distribution of the state at the next time based on the past history of the state of the tracking target, symmetry can be tracked even if the object moves vigorously. At this time, in order to estimate the prior probability distribution, the appearance (appearance, appearance, appearance form) observed by the camera should be included in the parameters in addition to the state information indicating the position and orientation of the tracking target Thus, the object can be tracked robustly against changes in appearance due to large changes in position and posture.

On the other hand, if not according to the present invention, changes in the appearance of the tracking target are not taken into consideration, the template constructed at the time of initialization is held as an appearance model, and this template is rotated and translated according to the state of each particle to input images. And the observation likelihood is obtained from the matching error. When the posture of the tracking target changes greatly, a simple rotation of the template cannot appropriately represent the change in the appearance of the target. This is because the front face included in the template cannot be observed gradually due to the rotation. As a result, there has been a problem that it is difficult to obtain the observation likelihood appropriately, and tracking is often performed. In the face posture tracking using the M-PF described in the above-mentioned Non-Patent Document 1, tracking up to about 50 degrees is the limit in the swing direction.
However, according to the present invention, it is possible to expand the range of postures that can be tracked while maintaining the features of M-PF, by extending the prediction of the simultaneous prior probability distribution of posture and appearance.

  In the first and second embodiments described above, a program for realizing the function of the state estimation device is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read by a computer system, You may control by performing. Here, the “computer system” may include an OS and hardware such as peripheral devices. The “computer-readable recording medium” means a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, etc. This is a storage device.

  Further, the “computer-readable recording medium” means a volatile memory (for example, DRAM (Dynamic DRAM) in a computer system that becomes a server or a client when a program is transmitted through a network such as the Internet or a communication line such as a telephone line. Random Access Memory)), etc., which hold programs for a certain period of time. The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

  DESCRIPTION OF SYMBOLS 11 ... Prior probability distribution prediction part, 12 ... A posteriori probability distribution estimation part, 13 ... Estimation result calculation part, 14 ... Tracking stability determination part, 15 ... History storage apparatus, 16 ... Shape model storage device, 17 ... history storage unit, 18 ... estimation target time setting unit, 19 ... end determination unit

Claims (5)

State information indicating the position and orientation of the observation target at the past time, appearance information indicating the appearance form of the observation target at the time, and time indicating the probability that the observation target is in the state indicated by the state information at the time A history accumulator that stores a table for associating the respective reproduction probability with each other,
Based on the past state information and the temporal reproduction probability, a state prior probability distribution representing a state indicating the position and orientation of the future observation target by a plurality of particles is created and associated with the state information. Based on the appearance information, a prior probability distribution prediction unit that creates an appearance prior probability distribution representing an appearance form of the observation target in the future;
Input observation information on the observation target measured by an external observation device, collate the observation information with the appearance prior probability distribution, calculate the observation likelihood based on the collation error, and respond to the observation likelihood A posterior probability distribution estimator that performs weighting on the state prior probability distribution to create a posterior probability distribution;
A state estimation device comprising: a history storage unit that updates and stores the state information indicated by the posterior probability distribution in the history storage device. The time reproduction probability is
The state estimation apparatus according to claim 1, wherein the state estimation apparatus is calculated based on a change in the state of the observation target over time in the past and similarity of the state information immediately before. The prior probability distribution prediction unit
The state information similar to the plurality of particles representing the state prior probability distribution is selected, and the appearance prior probability distribution is created based on the appearance information associated with the state information. Item 3. The state estimation device according to Item 1 or 2. State information indicating the position and orientation of the observation target at the past time, appearance information indicating the appearance form of the observation target at the time, and time indicating the probability that the observation target is in the state indicated by the state information at the time A state estimation method in a state estimation device including a history storage device that stores a table for associating each of the typical reproduction probabilities,
Based on the past state information and the temporal reproduction probability, a state prior probability distribution representing a state indicating the position and orientation of the future observation target by a plurality of particles is created and associated with the state information. A prior probability distribution prediction step of creating an appearance prior probability distribution representing an appearance form of the observation object in the future based on the appearance information;
Input observation information on the observation target measured by an external observation device, collate the observation information with the appearance prior probability distribution, calculate the observation likelihood based on the collation error, and respond to the observation likelihood A posteriori probability distribution estimating step of creating a posteriori probability distribution by performing weighting on the state prior probability distribution;
A history storage step of updating and storing the state information indicated by the posterior probability distribution in the history storage device;
A state estimation method comprising: On the computer,
State information indicating the position and orientation of the observation target at the past time, appearance information indicating the appearance form of the observation target at the time, and time indicating the probability that the observation target is in the state indicated by the state information at the time History storage means for storing a table for associating the respective reproduction probabilities with each other,
Based on the past state information and the temporal reproduction probability, a state prior probability distribution representing a state indicating the position and orientation of the future observation target by a plurality of particles is created and associated with the state information. A prior probability distribution predicting means for creating an appearance prior probability distribution representing an appearance form of the observation object in the future based on the appearance information;
Input observation information on the observation target measured by an external observation device, collate the observation information with the appearance prior probability distribution, calculate the observation likelihood based on the collation error, and respond to the observation likelihood A posteriori probability distribution estimating means for creating a posteriori probability distribution by performing weighting on the state prior probability distribution,
History storage means for updating and storing the state information indicated by the posterior probability distribution in the history storage means;
Program to function as.

Images