JP2009217523A - Dynamic image processing method, dynamic image processing device and dynamic image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To further accurately perform tracking for a plurality of moving objects. <P>SOLUTION: A dynamic image processing device 1 can further accurately perform tracking for a plurality of moving objects by repetitively executing each processing of estimation of a plurality of hidden variables and respective weights of the hidden variables, estimation of hidden state quantities, resampling based on the weight distribution of particles, and update of the hyper parameter of dynamics. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  In the present invention, when time-series data composed of information output from a plurality of hidden state quantities (estimated values of true values) is given as input, such as a moving image having a plurality of moving objects. The present invention relates to a technique for estimating the number of dynamic patterns that control the temporal development of multiple objects (the value of a physical quantity changes from moment to moment) and the characteristics of each pattern, and accurately estimating the hidden state quantities of unknown tracking targets.

  Tracking multiple moving objects is one of the important problems in the study of moving image processing. The tracking is to continuously estimate the correct position and state quantity of the tracking target detected from the moving image by modeling the temporal change. As a result, a spatiotemporal trajectory of the position (state) of each object can be obtained from the moving image. If a plurality of objects in a moving image can be tracked at the same time, it can be used for various purposes such as recognition of behavior patterns, automatic monitoring of situations, and understanding of contents of moving images.

  Conventional multi-object tracking algorithms (see, for example, Non-Patent Documents 1 and 2) can estimate the state trajectory of each target even when the number of tracking targets to be tracked is unknown or temporally fluctuates. . In particular, according to the technique of Non-Patent Document 2, a change in the number of tracking targets is also modeled stochastically, and tracking processing can be performed not only for moving images but also for many time-series data.

  In most existing multi-object tracking methods (Non-Patent Documents 1, 2, etc.), the time evolution of the tracking object is modeled by a single dynamics model. That is, it is assumed that all objects in the moving image move with the same dynamics pattern (motion pattern). However, when many moving objects are observed in the scene, the assumption that there are a plurality of different dynamics depending on the characteristics of each target is expected to be a more natural and correct modeling. In this specification, the dynamics refers to a “time evolution function of a hidden state quantity” and an “observation function for calculating an observation quantity”, and details will be described later.

Designing such a tracking model requires prior knowledge of i) the number of dynamics patterns and ii) the characteristics of each pattern. Since it is difficult to estimate this information manually, a solution by machine learning is required. The problem i) can be solved by using a time-series data clustering method (for example, Non-Patent Documents 3 and 4). The problem ii) is solved as a model parameter estimation problem in time series data (for example, Non-Patent Documents 5 and 6).
B. Leibe, N. Cornelis, K. Cornelis and L. Van Gool, “Dynamic 3D Scene Analysis from a Moving Vehicle”, in Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, 2007. S. Saerkkae, A. Vehtari and J. Lampinen, “Rao-Blackwellized Particle Filter for Multiple Target Tracking”, Information Fusion, Vol. 8, N0. 1, pp 2-15, 2007. M. Ramoni and P. Sebastiani and P. Cohen, “Bayesian Clustering by Dynamics”, Machine Learning, Vol. 47, No. 1, pp. 91-121, 2002. P. Smyth, “Clustering Sequences with Hidden Markov Models”, in Advances in Neural Information Processing Systems 9, pp. 648-654, 1997. F. Caron, M. Davy, A. Doucet, E. Duflos and P. Vanheeghe, “Bayesian Inference for Linear Dynamic Models with DirichletProcess Mixtures”, IEEE Transactions on Signal Processing, To appear, 2007. MJ Cassidy and WD Penny, “Bayesian Nonstationary Autoregressive Models for Biomedical Signal Analysis”, IEEE Transactions on Biomedical Engineering, Vol. 49, No. 10, PP. 1142-1152, 2002.

  However, existing clustering or parameter estimation methods can only be used for observation series (time series) data separated for each object. That is, it is impossible to learn a pattern using these conventional methods for observation series data in which a plurality of tracking targets are mixed.

  Accordingly, the present invention has been made to solve the above-described problem, and an object thereof is to perform tracking with respect to a plurality of moving objects with higher accuracy.

  In order to solve the above-described problem, the present invention provides a true value of a plurality of moving objects using a probabilistic generation model when observation data related to a moving image in which a plurality of moving objects exist is given as an input. Estimating the number of hidden states of the multiple moving objects by estimating the number of patterns and each feature of the dynamics indicating the time evolution function of the hidden state amount indicating the estimation amount and the observation function for calculating the observation data A moving image processing method by a moving image processing device, wherein the moving image processing device includes the observation data, a hidden state quantity of each of the moving objects, and between the observed data and each of the hidden state quantities of each of the moving objects. Storage means for storing a plurality of hidden variables indicating the relationship of each, a weight of each of the hidden variables, and a hyperparameter of the dynamics, and an operation The hidden variable set estimation unit estimates the weights of the plurality of hidden variables and the hidden variables based on the observation data; and hidden state estimation. Is configured to calculate the hidden state quantity based on the observation data, the hidden state quantity at a time immediately before the currently calculated time, and the plurality of hidden variables estimated by the hidden variable set estimation unit. The estimation step, and the resampling unit, based on the plurality of hidden variables estimated by the hidden variable set estimation unit and the hidden state quantity estimated by the hidden state estimation unit, one estimation result of the hidden variable A step of performing resampling by estimating a portion with a high posterior distribution from the distribution of the weights of the particles indicating the estimation result of the hidden state amount accompanying with the hyperparameter updating unit And repeatedly executing the step of updating the hyperparameters of the dynamics based on the resampling result by the resampling unit, thereby estimating the hidden state quantities of the plurality of moving targets by the hidden state estimating unit. It is characterized by that.

  According to this invention, the respective processes of estimation of a plurality of hidden variables and the respective weights of the hidden variables, estimation of the hidden state quantity, resampling based on the distribution of the weights of the particles, and update of the hyperparameters of the dynamics are repeatedly executed. Thus, it is possible to perform tracking with respect to a plurality of moving objects with higher accuracy.

  Further, according to the present invention, in the step in which the hidden variable set estimation unit estimates the weights of the plurality of hidden variables and the hidden variables based on the observation data, the hidden variable determination unit determines the previous one. The plurality of hidden variables for each particle based on the weight of the hidden state at the time of the first time, the plurality of hidden variables at the previous time, and the weight of the hidden variable at the previous time. And a step of estimating a weight of each hidden variable for each particle based on the plurality of hidden variables estimated by the hidden variable determination unit by a hidden variable weight calculation unit. It is desirable.

  According to this invention, the hidden variable determination unit estimates a plurality of hidden variables for each particle, and the hidden variable weight calculation unit estimates the weight of each hidden variable for each particle, so that the estimations differ by the number of particles. You can have hypotheses (and each weight). Therefore, even when the moving object suddenly (instantaneously) changes the direction of movement, a plurality of moving objects can be tracked with higher accuracy by using the corresponding estimation hypothesis.

  Further, according to the present invention, the hidden variable determination unit includes the hidden state quantity at the previous time, the plurality of hidden variables at the previous time, and the hidden variable at the previous time. In the step of estimating the plurality of hidden variables for each particle based on the respective weights, the hidden state quantity at the previous time, the plurality of hidden variables at the previous time, and Receives input of the observation data at the currently calculated time, and indicates whether each of the moving objects exists in the moving image at a certain time among the plurality of hidden variables by the moving object number determination unit Determining the hidden variable; determining the hidden variable representing an identifier of the movement pattern of the movement target among the plurality of hidden variables by the movement pattern number determination unit; A step of determining a parameter of the operation pattern by a pattern parameter determination unit; and a determination of the hidden variable representing the correspondence between the observation data and the movement target among the plurality of hidden variables by the movement target correspondence determination unit It is desirable to carry out the following steps.

  According to this invention, it is possible to cope with increase / decrease in the number of movement targets by determining the hidden variable representing the existence of the movement target by the movement target number determination unit. In addition, since the motion pattern number determination unit determines the hidden variable that represents the identifier of the motion pattern to be moved, it is possible to cope with a situation in which the movement target acts while switching the motion pattern. Furthermore, since the motion pattern parameter determination unit determines the motion pattern parameters, it is possible to avoid a situation in which the tracking accuracy is deteriorated due to an inappropriate initial parameter of the motion pattern. In addition, the correspondence relationship between the observation data and the movement target can be grasped by the movement target correspondence determination unit determining the hidden variable representing the correspondence between the observation data and the movement target.

  A moving image processing program according to the present invention causes a computer to function as a moving image processing apparatus. With such a configuration, a computer in which this program is installed can realize each function based on this program.

  According to the present invention, tracking for a plurality of moving objects can be performed with higher accuracy.

  The best mode for carrying out the present invention (hereinafter referred to as “embodiment”) will be described in detail below with reference to the drawings (refer to drawings other than the referenced drawings as appropriate). In the present embodiment, a method of simultaneously tracking a plurality of objects (also referred to as moving objects or tracking objects) existing in a moving image will be described. As described above, the dynamics is calculated by calculating the time evolution function of the hidden state quantity (x (t) described later) (formula (20) described later) and the observed quantity (y (t) described later). It refers to the observation function (formula (21) to be described later) for The learning of dynamics is to estimate parameters (ξ, ψ described later) necessary for Expressions (20) and (21).

  The conventional method and the method according to the present embodiment (hereinafter referred to as the present method) will be described again. In the conventional method, it is necessary to determine a dynamics model that represents a change in the state of an object prior to tracking a plurality of objects. In general, the number of target dynamics and the values of the parameters of each dynamics are unknown, and it is difficult to automatically estimate and set them. On the other hand, there are methods for estimating the dynamics of objects, but these methods are not applicable to data with multiple objects. In the present embodiment, a method will be described in which the number of dynamics patterns and parameters are estimated using a probabilistic generation model for data having a plurality of targets, and at the same time, tracking of an unknown number of targets can be performed.

(Configuration of moving image processing apparatus)
First, the configuration of the moving image processing apparatus of the present embodiment will be described. FIG. 1 is a functional block diagram schematically showing the configuration of the moving image processing apparatus according to this embodiment. The moving image processing device 1 is a computer device including, for example, a central processing unit (CPU), a random access memory (RAM), a read only memory (ROM), a hard disk drive (HDD), and an input / output interface. As shown in FIG. 1, the moving image processing apparatus 1 includes a computing unit 2 (for example, CPU and RAM), a storage unit 3 (for example, ROM and HDD), an input unit 4 such as a keyboard and a mouse, and a liquid crystal display. Output means 5 is provided, and they are connected by a bus line 6. Here, an outline of the configuration of the moving image processing apparatus 1 will be described, and details will be described in FIG.

  The storage unit 3 includes an observation amount (also referred to as observation data or an observation value) 31 relating to a moving image, a plurality of hidden state quantities 32 indicating an estimated value of a true value relating to each moving object in the moving image, and an observation relating to each moving object. A plurality of hidden variables 33 indicating the relationship between data and a hidden state quantity (also referred to as a hidden state), weights 34 of the hidden variables, and various parameters 35 including hyper parameters of dynamics are stored (for details, see FIG. (Postscript). Note that the storage means 3 also stores an operation program of the calculation means 2 and the like although not particularly shown. In the following, when the calculation unit 2 reads / writes each data from / to the storage unit 3, the description “to the storage unit 3” is omitted.

  The computing means 2 includes a hidden variable set estimation unit 21, a hidden state estimation unit 22, a resampling unit 23, and a hyper parameter update unit 24.

  The hidden variable set estimation unit 21 estimates a plurality of hidden variables and the weight of each hidden variable based on the observation data, and includes a hidden variable determination unit 211 and a hidden variable weight calculation unit 212. Yes.

  The hidden variable determination unit 211 estimates the hidden variable for each particle based on the hidden variable at the previous time, the plurality of hidden state quantities, and the weight of each hidden variable. A determination unit 2111, an operation pattern number determination unit 2112, an operation pattern parameter determination unit 2113, and a movement target correspondence determination unit 2114 are provided.

  The movement target number determination unit 2111 determines a hidden variable indicating whether or not each movement target exists in the moving image at a certain time. The motion pattern number determination unit 2112 determines a hidden variable that represents an identifier (such as an identification number) of the motion pattern to be moved.

  The operation pattern parameter determination unit 2113 determines an operation pattern parameter. The movement target correspondence determining unit 2114 determines a hidden variable representing the correspondence between the observation data and the movement target.

  The hidden variable weight calculation unit 212 estimates the weight of each hidden variable for each particle (details will be described later) based on the hidden variable estimated by the hidden variable determination unit 211.

  The hidden state estimation unit 22 estimates the hidden state amount based on the observation data, the hidden state amount at the previous time, and the plurality of hidden variables estimated by the hidden variable set estimation unit 21.

  Based on the plurality of hidden variables estimated by the hidden variable set estimation unit 21 and the hidden state quantity estimated by the hidden state estimation unit 22, the resampling unit 23 estimates one hidden variable and the accompanying hidden state quantity. Re-sampling is performed by inferring a portion with a high posterior distribution from the distribution of particle weights indicating the estimation result, and sampling a plurality of times.

  The hyper parameter update unit 24 updates dynamics parameters based on the resampling result by the resampling unit 23.

(Outline and purpose of calculation process)
Next, the outline of the calculation process of this method and its purpose will be described. FIG. 2A is an image diagram of multi-object tracking. Some of the observation data may be noise that has nothing to do with the object to be tracked. In this method, a plurality of state vectors x (t) (for example, two-dimensional position vectors) to be tracked are observed time series data (observed data) Y (t) = {y (t)}, t = 1. The goal is to estimate from.

  x (t) is a hidden state quantity (ie, direct observation is impossible), and in order to simplify the calculation, x (t) is Markovian (the probability of an event occurring in the future is only the current state) It is assumed that it is determined from the above and does not depend on the past state). Note that x (t) is not directly observable because even if some data is observed, the data may be deviated from the true value due to noise and is not necessarily accurate data. It means that. That is, the hidden state quantity refers to this true value (or its estimated value). The true value cannot be observed, but can be estimated with high accuracy according to this method.

  Furthermore, a hidden variable φ (t) is defined to express the relationship between the observation data and the hidden state quantity. In actual calculation, in order to estimate the hidden state quantity x (t) at each time t, the observation data Y (t), the hidden variable φ (t-1) and the hidden state quantity x (t at time t−1 are calculated. -1) is used to probabilistically generate values for the hidden variable φ (t) and estimate the hidden state quantity x (t) under each candidate value. The estimation of the probability distribution of the hidden state quantity at each time is repeated by weighting the combination candidates of the plurality of hidden variables and the hidden state quantity with the certainty.

  An example will be described. Now, assuming that the purpose is person tracking in a video image, the hidden state quantity is the position of a person at each time or a velocity vector. The observation amount is moving image data or time-series data of feature amount vectors extracted therefrom. The hidden variable is information such as the number of persons in the scene (moving image) at each time, the movement pattern of each person, and the correspondence between the observation amount and the person (which person a certain pixel or feature vector corresponds to). From the observation information obtained at each time t and information such as the number of persons and the positions of each person at the time t−1, a plurality of hidden variables and candidate values for the state of each person at the time t are estimated, and the weight of the probability The estimation of the hidden state quantity distribution is repeated.

(Generation model)
Next, the generation model used for the calculation of this method will be described. FIG. 2B is a graphical model of the generation model used for the calculation of this method. FIG. 2B represents the dependency between random variables and the process of generating time-series data from these variables. The generated data is x (t) representing the state (position) of the tracking target and y (t), which is an observation amount in which noise is superimposed on the state amount. This generation model can represent action data of multiple objects according to a mixed model of dynamics parameters.

Since there are a plurality of tracking targets, the hidden state quantity x (t) has an index i, and the observation quantity y (t) has an index m. The i th target (hidden state quantity) is x _i (t), and the m th observation quantity is y _m (t). N _t represents the number of tracking objects at time t, and M _t represents the number of observations. When there are multiple tracking targets (hidden state quantities) and observed quantities, resolve the correspondence of which target outputs which observation quantity (corresponds) to estimate the hidden state quantity for each target. Must. This problem is called Data Association and is an important issue for accurate estimation of hidden state quantities. In this generation model, hidden variables c and j are introduced to solve the Data Association problem.

Each hidden variable will be described. First, c _i (t) is a variable representing the birth (addition) and death (deletion) of the tracking target i. That is, explain how many objects currently appear or disappear. When c _i (t) = 1, this indicates that the i-th object is alive, ie, exists in the scene, and conversely, when c _i (t) = 0, this indicates that the object exists outside the scene. When a new tracking target is generated at time t, a new index i ^ (in this specification, the symbol "^" is a symbol positioned on the immediately preceding character) is introduced. Let c _{i ^} (t) = 1. j _m (t) is a data association variable that associates the observation amount y _m (t) with the target x _i (t) that is the output source. j _m (t) = i indicates that the i-th tracking target x _i (t) outputs the m-th observation amount y _m (t).

The parameter is a set {ξ _k }, {ψ _k } indexed by the subscript k. x (t) (y (t)) is generated according to the distribution characterized by the parameter ξ _k (ψ _k ). In this model, the Kalman filter (Kalman filter. For example, “Genjiro Kitagawa,“ Introduction to Time Series Analysis ”, Iwanami Shoten, pp. 125-141, 209-222, 2005.”) (hereinafter referred to as “Non-Patent Document 7”) ) _K ) and ξ _k and ψ _k are their normal distribution parameters. Although the Kalman filter is used here, the present invention is not limited to this, and a model that is generally characterized by a normal distribution can be used.

G ₀ and H ₀ are Normal Inverse Wishart distributions (NIW) (see, for example, “Watanabe Hiroshi,“ Introduction to Bayesian Statistics ”, Fukumura Publishing, 1999.), and their parameters (hyper parameters) are θ ^ξ and θ ^ψ , respectively. To do. Here, the NIW distribution is used because it is a conjugate distribution with respect to the normal distribution, and the calculation can be simplified. Other distributions can also be used.

At each time t, first, an index z _i (t) of the dynamics pattern that dominates the i-th tracking target is generated according to the method of Non-Patent Document 5. This process is the Chinese Restaurant Process (CRP) with parameter γ (eg “D. Blackwell and J. MacQueen,“ Ferguson Distributions via Polyaurnschemes ”, The Annals of Statistics, Vol. 1, pp. 353-355, 1973.”) “Non-Patent Document 8”) and “C. Kemp, JB Tenenbaum, TL Griffiths, T. Yamada and N. Ueda,“ Learning Systems of Concepts With An Infinite Relational Model ”, in Proceedings of 21st National Conference on Artificial Intelligence, 2006.)) and Bernoulli's trial parameterized by π.

CRP is one of the implementation examples of Dirichlet Process Mixture (DPM) and gives a prior distribution of clustering. By using CRP, dynamics patterning (clustering) can be modeled without fixing the number of dynamics patterns (number of clusters) in advance. When z _i (t) = k is generated, a hidden state quantity and an observed quantity are generated using parameters ξ _k and ψ _k corresponding thereto. A feature of this model is that DPM is used to represent a parameterized distribution. DPM is one of non-parametric Bayes models and can be understood as a mixture model with an infinite number of mixtures. The advantage of using DPM is that not only the properties (parameters) of each mixing component can be estimated as in the conventional mixing model estimation, but also the number of mixtures can be estimated simultaneously using a Bayesian framework.

Thus, all the hidden variables necessary for generating x _i (t) and y _m (t) are prepared. The hidden state quantity x _i (t) is generated only when the target i exists in the scene (c _i (t) = 1). The generation process is characterized by ξ _k and ψ _k determined by the index z _i (t) = k. Subsequently, in order to generate the observation amount y _m (t), a data association variable j _m (t) is generated. The m-th observed quantity y _m (t) is generated by x _i (t) and the observation distribution parameter ψ _{zi (t)} when j _m (t) = i. By repeating this process for all t, it is possible to generate general multi-target behavior time series data composed of a superposition of a plurality of target behavior trajectories controlled by a plurality of different dynamics.

(Calculation algorithm)
Next, the calculation algorithm of this method will be described. The final goal is to calculate the posterior distribution p (x (t) | Y (t)) at each time, and can be calculated by the following equation (1), for example.

  FIG. 3 is a flowchart showing the overall processing of this method. That is, FIG. 3 shows an outline of an algorithm for estimating hidden state quantities and hidden variables. The calculation means 2 of the moving image processing apparatus 1 in FIG. 1 updates the time by “1” from time t = 1 (step S1) to time t = T (No in step S8) (step S9), Processes S2 to S7 are performed.

  At each time t, after the computing means 2 acquires (inputs) the observed value at that time (step S2), the hidden variable set estimation unit 21 estimates each hidden variable (set) (step S3: details are shown in FIG. 4). And later). At this time, the estimated distribution of hidden variables is expressed by holding a plurality of estimation results together with their weights. Next, the hidden state estimation unit 22 calculates the best state quantity assumed under the hidden variable (estimates the hidden state (quantity) at time t) (step S4: details will be described later in FIG. 11). These estimated state quantities depend on the result of the hidden variable. As described above, one estimation result of the hidden variable and the estimation result of the hidden state quantity accompanying the estimation result are called “particles”.

  Subsequently, the resampling unit 23 performs a process called resampling according to the weight of the particles (step S5: details will be described later in FIG. 12). Resampling is to estimate a portion with a high posterior distribution from the distribution of particles and their weights, and perform sampling many times from there. The hyper parameter update unit 24 receives the resampling result, and updates the hyper parameter, which is a dynamics parameter, based on the observation data at this time and the hidden state quantity estimated in step S4 (step S6: details). Is described later in FIG. 13). The computing means 2 outputs the hidden state quantity, the hidden variable, and the hyper parameter at time t (step S7), and displays the tracking result on the output means 5, for example.

FIG. 4 is a flowchart showing details of step S3 of FIG. 3, and is a flowchart showing a hidden variable (set) estimation method. Here, the predicted values and weights of s = 1,..., S hidden variables are calculated. First, various hidden variables at time t are collected and φ (t) = {{c _i (t)}, {j _m (t)}, {z _i (t)}, {ξ _k (t)}, {Ψ _k (t)}}.

Here, each hidden variable will be briefly described. c _i (t) is a variable indicating whether or not the i-th tracking target exists in the scene at time t. In the example of person tracking in a moving image, it is expressed whether or not a person exists in the moving image scene at time t. The current number of tracking targets can be determined by the values of c (t) = {c _i (t)}, i = 1,..., N _t .

j _m (t) represents the number of the tracking target that generated the m-th observation amount at time t. In the example of person tracking in a moving image, for example, the information indicates which person the mth pixel corresponds to. The value of j (t) = {j _m (t)}, m = 1,... M _t greatly affects the estimated value of the hidden state quantity of each tracking target.

z _i (t) is a dynamics pattern number that determines the state quantity change of the i-th tracking target at time t. In the example of person tracking in a moving image, the movement pattern of each person at time t, for example, information such as running or walking to the right. The values of z (t) = {z _i (t)}, i = 1,..., N _t also influence the estimated value of the hidden state quantity of each tracking target and the determination of j (t).

ξ _k (t) and ψ _k (t) represent the k th dynamics pattern at time t. More precisely, it is a parameter that characterizes each pattern. In the example of person tracking in a moving image, this corresponds to the amount of time change in the position vector of the person or the average or variance of observation noise.

  The hidden variable set estimation unit 21 inputs the observation amount y (t) at time t (step S31), s = 1 (step S32) to s = S (No in step S36), and sets the value of s to “1”. ”Are updated (step S37), and steps S33 to S35 are performed.

In step S33, the hidden variable set estimation unit 21 sets the hidden state quantity x ^(s) (t-1), the hidden variable φ ^(s) (t-1), and the weight w ^(s) (t-) at time t-1. Enter 1).

In step S34, the hidden variable determination unit 211 determines the s-th hidden variable φ ^(s) (t) at time t (details will be described later in FIG. 5). Specifically, φ ^(s) (t) is determined by sampling the s-th particle as shown in the following equation (2).

In step S35, the hidden variable weight calculation unit 212 calculates the weight w ^(s) (t) of the sth hidden variable (details will be described later in FIG. 10). Specifically, each particle is weighted by w ^(s) (t) as shown in equation (3).

As shown in the following expressions (4) to (8), a conditional distribution p (φ (t) | Φ (t−1)) can be defined.

  In the case of No in step S36, the hidden variable weight calculation unit 212 outputs S hidden variables φ (t) and weight w (t) at time t (step S38).

FIG. 5 is a flowchart showing details of step S34 in FIG. 4, and is a flowchart showing a calculation method of the predicted value of the hidden variable. FIG. 5 shows a sampling process of φ ^(s) (t) from the above-mentioned proposal distribution, which can be expressed as the following equations (9) to (13). That is, FIG. 5 represents sampling from q (•).

  The hidden variable determination unit 211 inputs the hidden state quantity x (t−1) and the hidden variable φ (t−1) at time t−1 (step S341), and inputs the observed quantity y (t) at time t. (Step S342).

  In step S343, the moving object number determination unit 2111 determines a hidden variable c (t) representing generation and disappearance of the tracking object (details will be described later in FIG. 6).

  In step S344, the motion pattern number determination unit 2112 determines a hidden variable z (t) that represents the number of the motion pattern to be tracked (details will be described later in FIG. 7).

  In step S345, the motion pattern parameter determination unit 2113 determines ξ (t) and ψ (t) representing the parameters of the motion pattern (details will be described later with reference to FIGS. 3, 4 and 5).

  In step S346, the movement target correspondence determining unit 2114 determines a hidden variable j (t) representing the correspondence between the observed value and the tracking target (details will be described later in FIG. 9).

  In step S347, the hidden variable determination unit 211 outputs the hidden variable φ (t) at time t.

  FIG. 6 is a flowchart showing details of step S343 in FIG. 5, which is sampling of the hidden variable c (t) that expresses the tracking target birth (addition) and death (deletion) using Expression (9).

First, an overview of the processing of FIG. 6 will be described. In Non-Patent Document 2, the time evolution of c (t) is modeled as a two-stage Bernoulli trial, and this model follows it. That is, i) The tracking target (c _i (t−1)) = 1) existing in the scene at the previous time disappears from the scene with the probability P _d (c _i (t) = 0). Otherwise, c _i (t) = 1. ii) the new target object is generated with a probability _{P b.} When the number of tracking targets newly generated at each time is limited to “1” (see Non-Patent Document 2), the probability of obtaining the value when c (t) is calculated for all tracking targets is expressed by the equation (14)

Here, n _s is the number of tracking targets currently existing in the scene, n _d is the number of tracking targets that disappeared from the scene at time t, and (n _b = {0, 1}) is the newly generated tracking target. Is a number.

Specifically, the movement target number determination unit 2111 first inputs c (t−1) (step S601), and from i = 1 (step S602) to i = N _t−1 (No in step S607). , I is updated by “1” (step S608), and the processing of steps S603 to S606 is performed.

The movement target number determination unit 2111 determines whether c _i (t−1) = 1 (step S603), and in the case of Yes, determines whether the tracking target i exists at the time t (step S604). If Yes in step S604, 1 is substituted for c _i (t) (step S606). In the case of No in either step S603 or step S604, 0 is substituted for c _i (t) (step S605).

In step S609, the mobile object number determination unit 2111 determines whether the new tracked occurs, in the case of Yes substituting _{N t-1} +1 to _{N t} (step _{S610), c} Nt _(t) 1 by substituting (step S611), If No substituting _{N t-1} to _{N t} (step S612). Finally, the movement target number determination unit 2111 outputs c (t) (step S613).

FIG. 7 is a flowchart showing details of step S344 in FIG. 5, and is sampling of z (t) using equation (11). First, an overview of the processing in FIG. 7 will be described. For the generation of z (t), Chinese Restaurant Process (CRP) (see Non-Patent Document 8) is used. CRP is an implementation example of Dirichlet Process Mixture (DPM), which is a kind of nonparametric Bayes model, and is a prior distribution of sample division (clustering). Now, there is a mixed distribution element (cluster) set indexed by k, and the cluster index of the i-th sample is written as z _i . When z ₁ to z _i−1 are given, the distribution of z _i is expressed by the following equation (15).

Here, m _k is the size of the k-th cluster. This sampling process is written as z _{i to} CRP (γ). As can be seen from Equation (15), a small number of large clusters are likely to be generated in clustering generated from CRP. This is an appropriate property as a prior distribution of clustering. A posterior distribution of z (t) can be derived by combining an appropriate likelihood function with clustering generated by CRP.

In this model, equation (15) is updated online to obtain the best clustering estimate at each time. Sampling of zi (t) at time t is performed as in the following equation (16) using Z (t-1) as prior information. That is, here, “online” means “incremental”, and the calculation of equation (15) is actually performed using the estimation result of Z up to time t−1 as in equation (16). It is to calculate.

m _k (t−1) is the size of the k th cluster obtained up to time t−1. | Z (t−1) | is the total number of cluster sizes at time t−1, that is, the sum of m _k (t−1).

In this model, the dynamics of each target can be changed by CRP, but such a change in dynamics may occur only at a certain frequency. In order to model the frequency of dynamics change, a new parameter π is introduced and the dynamics are changed with probability π. Finally, equation (11) is modeled as follows. First, as shown in equation (17), equation (11) is decomposed with object i.

Then, each z _i (t) is sampled as in the following equations (18) and (19).

Specifically, the motion pattern number determination unit 2112 first inputs z (t−1) and c (t) (step S701), and from i = 1 (step S702) to i = N _t (step S709). No), while updating the value of i by “1” (step S704), the processing of steps S703 to S708 is performed.

The motion pattern number determination unit 2112 determines whether c _i (t) = 1 (step S703). If No, the process proceeds to step S704. If Yes, whether the motion pattern of the tracking target i has changed. Judgment is made (step S705). If Yes in step S705, the index k of the operation pattern is determined according to CRP (step S706), and k is substituted for z _i (t) (step S707). If No at step _S705, the substituting _z i (t-1) to _z i (t) (step S 708). Finally, the operation pattern number determination unit 2112 outputs z (t) (step S710).

  FIG. 8 is a flowchart showing details of step S345 in FIG. 5, and is a process of sampling ξ (t) and ψ (t), which are normal distribution parameters of the Kalman filter, from Equation (12) and Equation (13), respectively. is there.

First, the outline of the processing of FIG. 8 will be described. As described above, similar modeling is possible if other models than the Kalman filter are characterized by a normal distribution. The number of clusters of particles s estimates and k _t at time t. When the hidden state quantity x _i (t) and the observation quantity y _m (t) to be tracked are generated from the k-th dynamics cluster (that is, when j _m (t) = i and z _i (t) = k) These generation processes are expressed by the following equations (20) and (21) using the Kalman filter.

Here, f and h are linear models having normal distribution noise. Let the system noise mean and covariance matrix be ξ _k (t) = {q, Q}, and the observation noise mean and covariance matrix be ψ _k (t) = {r, R}. In order to estimate these parameters, Normal Inverse Wishart distribution (NIW) (see Non-Patent Document 5) is introduced into the parameter prior distribution. NIW is set for the system noise and observation noise of each cluster, and the parameters are represented as θ _k = {θ ^ξ _k , θ ^ψ _k }.

From the posterior distribution based on the data up to time t−1, the estimated values ξk (t) and ψ _k (t) of the true parameters ξ _k and ψ _k at time t are sampled. For sampling at time t, a posteriori estimated value θ _k (t−1) = {θξ _k (t−1), θ ^ψ _k (t−1)} at time t−1 is used. Equations (12) and (13) are expressed as p (ξ (t) | θ ^ξ (t−1)) = Π _k (ξ _k (t) | θ ^ξ _k (t−1)), p (ψ ( t) | θ ^ψ (t−1)) = Π _k (ψ _k (t) | θ ^ψ _k (t−1)), and each element is expressed by the following equations (22) and (23). It expresses like this.

Specifically, the motion pattern parameter determination unit 2113 first inputs z (t), θ (t−1), and θ (0) (step S801), and k = 1 (step S802) to k = K. Until _t (No in step S808), while updating the value of k by “1” (step S809), the processing of steps S803 to S807 is performed.

The operation pattern parameter determination unit 2113 determines whether k is a new index generated at time t (step S803). If Yes, ξ _k (t) is sampled from NIW (θ ^ξ _k (0)). (Step S804), ψ _k (t) is sampled from NIW (θ ^ψ _k (0)) (Step S805). In the case of No in step S803, ξ _k (t) is sampled from NIW (θ ^ξ _k (t−1)) (step S806), and ψ _k (t) is sampled from NIW (θ ^ψ _k (t−1)). Sampling is performed (step S807). Finally, the motion pattern parameter determination unit 2113 outputs ξ (t) and ψ (t) (step S810).

  FIG. 9 is a flowchart showing details of step S346 in FIG. 5 and is the sampling of the data association variable J according to the equation (25). Here, the equation (24) will also be described at the same time.

First, an overview of the processing of FIG. 9 will be described. Assuming that each observation amount is independent, it is decomposed as the following equation (24).

In this model, it is assumed that p (j _m (t)) is distributed uniformly, that is, in general, there is no prior knowledge (information) regarding data association. However, the experiment described below shows an example in which prior knowledge is introduced.

If the search space of J is wide, outliers can be excluded by introducing pseudo likelihood (for example, “MK Pitt and N. Shephard,“ Filtering via simulation: Auxiliary particle filters ”, Journal of the American Statistical Association, Vol. 94, N0. 446, pp. 590-599, 1999 ”). In an experiment to be described later, a distribution represented by the following equation (25) was used as the proper distribution of j (see Non-Patent Document 2).

This equation attempts to calculate a distribution close to the posterior distribution using the likelihood in the “typical” state vector x ^ _i (t). In this model, the average value of x _i (t) prediction distribution (before filtering) was used as x ^ _i (t).

  Specifically, the movement target correspondence determining unit 2114 first inputs the hidden state quantity x (t−1) at time t−1 (step S901), and inputs the observation value y (t) at time t. (Step S902), J (t-1), c (t), z (t), ξ (t), and ψ (t) are input (Step S903).

Subsequently, the moving object correspondence relationship determining unit 2114, m = 1 to (step S904) until m = _{M t} (No in step S912), while updating the value of m by "1" (step S913), step S905 Processing of ~ S911 is performed.

The mobile object correspondence relationship determining unit 2114, i = 1 to (step S905) to i = _{N t} (No in step S910), while updating the value of i by "1" (step S907), step S906~ The process of S909 is performed.

The movement target correspondence determining unit 2114 determines whether or not c _i (t) = 1 (step S906). If No, the process proceeds to step S907. If Yes, z _i (t), ξ (t), x ^ _i is calculated from ψ (t) (step S908), and q (j _m (t) = i | J (t-1), y _m (t)) is calculated (step S909).

In step S911, the mobile object correspondence relationship determining unit _{2114, q (j m (t)} | J (t-1), y m (t)) to determine the _j m (t) in accordance with. Finally, the movement target correspondence determining unit 2114 outputs j (t) (step S914).

  Next, the process of step S35 in FIG. 4, that is, the process of calculating the weight of the hidden variable (hereinafter also simply referred to as “weight”) will be described. FIG. 10 is a flowchart showing details of step S35 in FIG. The formula for calculating the weight is as shown in the formula (3).

  Specifically, the hidden variable weight calculation unit 212 first inputs the hidden state quantity x (t−1) and the weight w (t−1) at time t−1 (step S1001), and the hidden variable at time t. φ (t) and observed value y (t) are input (step S1002).

  Subsequently, the hidden variable weight calculation unit 212 calculates q (φ (t) | Φ (t−1), Y (t)) (step S1003), and p (φ (t) | Φ (t−1). )) Is calculated (step S1004).

  Thereafter, the hidden variable weight calculation unit 212 calculates p (y (t) | φ (t)) (step S1005) and calculates w (t) (step S1006). Finally, the hidden variable weight calculation unit 212 outputs w (t) (step S1007).

  Next, the process of step S4 in FIG. 3, that is, the process of estimating the hidden state quantity based on the hidden variable obtained in step S3 will be described. FIG. 11 is a flowchart showing details of step S4 in FIG.

First, an overview of the processing of FIG. 11 will be described. By using the particle filter, the posterior distribution to be obtained is approximated as the following equation (26).

Now that the hidden variable Φ (t) ^(s) = {φ (t) ^(s) } is generated, this calculates p (x (t), Φ (t) ^(s) | Y (t)) It corresponds to doing. The calculation method itself is the same as the state estimation method using a general Kalman filter (Non-Patent Document 7).

  Specifically, the hidden state estimation unit 22 first inputs the hidden state quantity x (t−1) at time t−1 (step S1101), and the hidden variable φ (t) and observed quantity y (at time t). t) is input (step S1102).

  Subsequently, the hidden state estimation unit 22 updates the value of s by “1” from s = 1 (step S1103) to s = S (No in step S1115) (step S1116), and performs steps S1104 to S1114. Process.

Also, the hidden state estimation unit 22, i = 1 to (step S1104) to i = _{N t} (No in step S1114), while updating the value of i by "1" (step S1106), in step S1105~S1113 Process.

The hidden state estimation unit 22 determines whether c ^(s) _i (t) = 1 (step S1105). If No, the process proceeds to step S1106. If Yes, z ^(s) _i (t) = k. Ξ ^(s) _k (t) and ψ ^(s) _k (t) selected in step S1107 are set in the Kalman filter (step S1107).

Then, hidden state estimation unit 22, m = 1 to (step S1108) until m = _{M t} (No in step S1112), while updating the value of m by "1" (step S1110), step S1109, S1111 of Process.

The hidden state estimation unit 22 determines whether j ^(s) _m (t) = i (step S1109). If No, the process proceeds to step S1110. If Yes, y ^(s) _m (t) is used. Then, x ^(s) _i (t) is filtered (step S1111).

In the case of No in step S1112, the hidden state estimation unit 22 sets the average of the filtering results for each m as x ^(s) _i (t).

  In the case of No in step S1115, the hidden state estimation unit 22 outputs the hidden state quantity x (t) at time t (step S1117).

  Next, the process of step S5 in FIG. 3 will be described. This processing is performed in order to retain a more probable particle set by selectively copying particles (hidden variables and hidden state quantities) having a large weight, that is, a high posterior distribution, and replacing them with particles having a small weight. FIG. 12 is a flowchart showing details of step S5 in FIG. Note that the processing (calculation) in FIG. 12 is the same as the resampling in the conventional particle filter (see Non-Patent Document 7).

  The resampling unit 23 receives the weight w (t), the hidden variable Φ (t), and the hidden state quantity X (t) (step S1201), and normalizes the particle weight w (t) (step S1202).

  Thereafter, the resampling unit 23 updates the value of s by “1” from s = 1 (step S1203) to s = S (No in step S1208) (step S1209), and performs the processing of steps S1204 to S1207. Do.

The resampling unit 23 generates uε [0, 1] (step S1204), and Σ _{l = 1} ^{s ^ −1} w (t) ^(l) (from “l = 1” to “s ^ −1”. The sum of w (t) ^(l) of the same, and so on) <u ≦ Σ _{l = 1} ^{s ^} w (t) Find s ^ that satisfies ^(l) (step S1205).

Resampler 23, [Phi ^{^} a ^(s) (t) to ^{Φ (s ^) (t)} , X ^ a ^(s) (t) to ^{X (s ^) (t)} by substituting each (step S1206) , W ^ ^(s) 1 / s is substituted into (t) (step S1207).

    In the case of No in step S1208, the resampling unit 23 outputs the weight w ^ (t), the hidden variable Φ ^ (t), and the hidden state quantity X ^ (t) (step S1210).

  Next, details of the process in step S6 in FIG. 3 will be described. FIG. 13 is a flowchart showing details of step S6 in FIG.

First, the outline of the processing of FIG. 13 will be described. The processing of FIG. 13 performs tracking of an object using sampled ξ _k (t), ψ _k (t), and performs online estimation of the hyperparameter θ. The estimation of θ ^ξ _k (t) follows the recurrence formulas of the following formulas (27) to (29).

Here, θ ^ξ _k (0) is an initial value of θ ^ξ _k (t). From the conjugate property, θ ^ξ _k (t) can be _defined to be P (ξ _k | X (t), θ ^ξ _k (0)) = NIW (ξ _k ; θ ^ξ _k (t)), as shown in 30), estimated values of the hyper parameters ^θ ξ _{k (t)} has been shown to be computed using the likelihood of ^θ ξ _{k (t-1)} and the estimated hidden state quantity x (t) .

Similarly, θ ^ψ _k (t) can be estimated by the following equation (31).

These updated hyper parameters θ (t) are used in the equations (22) and (23) at the next time. Actually, since x (t) and y (t) are composed of a plurality of tracking objects i and observation data m, as shown in the following equations (32) and (33), data related to k Is used.

In the equation (32), “st” is an abbreviation for “such that”, which means that it applies only to a condition that meets the following conditions. That is, the expression (32) is calculated by replacing x (t) in the expression (30) with x [k] (t) (where x [k] (t) is the state of all the tracking targets i). It is a set of x _i (t) having i with z _i (t) = k in the quantity x _i (t).

  Specifically, the hyper parameter update unit 24 first inputs the hidden variable Φ (t), the hidden state quantity x (t), and the observed quantity y (t) at time t (step S1301).

  Subsequently, the hyperparameter update unit 24 updates the value of s by “1” from s = 1 (step S1302) to s = S (No in step S1320) (step S1321), and continues from steps S1303 to S1319. Process.

Furthermore, the hyper parameter updating unit 24, k = 1 to (step S1303) until k = _{K t} (No in step S1318), while updating the value of k by "1" (step S1319), in step S1304~S1317 Process.

Furthermore, the hyper parameter updating unit 24 was substituted for {null} in x [k] and y [k] (Step S1304) after, i = 1 to (step S1305) to i = _{N t} (No in step S1315) , I is updated by “1” (step S1307), and the processing of steps S1306 to S1314 is performed.

The hyperparameter update unit 24 determines whether c ^(s) _i (t) = 1 (step S1306). If No, the process proceeds to step S1307. If Yes, z ^(s) _i (t) = k. Whether or not (step S1308).

In step S1308, the hyper parameter update unit 24 proceeds to step S1307 if No, and substitutes {x [k], x ^(s) _i (t)} for x [k] if Yes (step S1309). ), m = 1 to (step S1310) until m = _{M t} (No in step S1314), while updating the value of m by "1" (step S1312), performs the processing of steps S1311, S1313.

The hyperparameter update unit 24 determines whether j ^(s) _m (t) = i (step S1311). If No, the process proceeds to step S1312, and if Yes, the y [k] is changed to {y [k]. , Y ^(s) _m (t)} is substituted (step S1313).

In the case of No in step S1315, the hyper parameter update unit 24 outputs θ ^ξ _k ^(s) (t−1) substituted for θ ^ξ _k ^(s) (t) using x [k] (step S1316). ^{), y [k] θ ψ} k (s) and outputs the (was substituted in _{^{t) θ ψ k (s)}} (t-1) with (step S1317).

  Thus, according to the moving image processing apparatus 1 of the present embodiment, the number of dynamics patterns and parameters can be estimated using a stochastic generation model, and at the same time, an unknown number of moving objects can be tracked.

  The greatest feature of the moving image processing apparatus 1 is that it can track a plurality of objects and simultaneously estimate the dynamics of each object. Specifically, as shown in FIG. 2B, FIG. 5, Equations (4) to (13), etc., by estimating five hidden variables c, z, ξ, ψ, j, Dynamics estimation can be calculated simultaneously.

  On the other hand, according to the present embodiment, only {c, j} (information that there are a plurality of tracking targets) or {z, x, y} (information that there are a plurality of dynamics) is estimated. Missing information due to variables that are not estimated when compared.

  Further, according to the moving image processing apparatus 1 of the present embodiment, even if the number of tracking objects changes with time, the change can be estimated and tracking can be performed correctly. This can be realized by a process of estimating c (t) representing generation and disappearance of an object at time t in equation (14).

  Furthermore, according to the moving image processing apparatus 1 of the present embodiment, the estimated number of dynamics can be automatically determined. This can be realized by increasing or decreasing the number of clusters in Expression (16) in which the calculation method of Expression (15) is changed for online calculation.

  In the conventional method that does not use particles (for example, only the Kalman filter is used), only one estimated value of a hidden variable that seems to be the best at each time is obtained. On the other hand, in this method, the Kalman filter and the particle filter are combined, and the hidden variable is estimated individually for each particle. Therefore, the estimation hypotheses differ by the number of particles.

  This difference appears as a result when, for example, the tracking target suddenly (instantaneously) changes the direction of movement. In the case of the conventional method that does not use particles, since the estimation is performed only in the direction of motion that seems to be the best based on the previous movement, it is difficult to cope with a sudden change, and the possibility of failure in tracking increases. On the other hand, in the case of this method using particles, a number of hypotheses are generated, so if any one of them predicts a sudden change in the direction of motion, there is a high possibility of being able to continue tracking. In this way, it is necessary to judge which one of the hypotheses should be trusted, so in this method, the reliability of each hypothesis is determined by the weight of the hidden variable (≈particle weight) w (t). Express. And if you put a small weight on estimates of events that are unlikely to happen, such as sudden turnarounds, tracking under normal (as expected) conditions is not important, but such hypotheses are actually You can leave room for what happens.

  Furthermore, the action and effect of each hidden variable will be described again. First, by estimating a hidden variable indicating the presence of a moving target (hidden variable indicating whether or not it exists in a moving image) (c (t)), the movement target is increased or decreased (disappearance from the screen and Appearance) can be expressed. If this variable does not exist, it is assumed that the number of tracking targets is constant throughout the entire data (time), so that the model cannot be applied to estimation of data other than that.

  In addition, by estimating the hidden variable (z (t)) representing the identifier of the motion pattern, it is possible to express the situation in which each tracking target acts while switching the motion pattern. If this variable does not exist, it is assumed that the tracking target always behaves in a certain motion pattern. Therefore, if the assumption is not correct, tracking itself becomes difficult. For example, the log likelihood shown in Table 3 is used. There is a high possibility that the tracking accuracy index such as will deteriorate.

  Further, by estimating the parameters (ξ, ψ) of the operation pattern, the tracking accuracy can be improved regardless of the initial setting (value) of this parameter. In other words, if this parameter is not estimated, even if a model that allows a plurality of motion patterns as described above is used, the accuracy of the model is completely determined by the initial parameters of the motion pattern. If the setting is not good, it is difficult to improve the tracking accuracy.

  In addition, it is possible to track a plurality of objects by estimating a hidden variable (j (t)) representing the correspondence between the observation data and the moving object. This estimation of hidden variables is indispensable when tracking multiple objects. If this hidden variable is not estimated, correct estimation cannot be performed when there are a plurality of tracking targets, that is, only a single tracking target can be tracked.

  In addition, this invention is not limited to the said embodiment, It can implement in the range which does not change the meaning. That is, specific configurations of hardware and software can be appropriately changed without departing from the gist of the present invention.

(Experimental example)
Next, an experiment for confirming the performance of this method using artificial data and actual moving image data will be described.
<Comparison method>
Two models with a reduced number of hidden variables are prepared for comparison with this method, and the effect of introducing hidden variables is evaluated. The characteristics of these three models are shown in Table 1.

  The first comparison method is a model similar to the model of Saerkkae et al. (See Non-Patent Document 2). This model does not update the dynamics hyperparameter θ (t) (Equations (30) and (31)) (no) and always uses the default initial value θ (0). Further, dynamics clustering using CRP (formula (16), formula (18), formula (19)) is not performed (no). As a result, ξ (t) and ψ (t) are always sampled from a single initial distribution for all tracked objects.

Second comparison technique, theta-line estimation (t) is performed (Equation (30), equation (31)) _(yes), clustering for _z i (t) (Equation (16), equation (18), Equation (19)) is not performed (no). Therefore, the estimated hyperparameter is learned independently for each tracking target. In this model, it is possible to express the fact that multiple tracking targets have different dynamics compared to the first model, so that it can cope with complicated cases (many moving targets), but parameter patterning (clustering) Is not done.

  Finally, in this method, as described above, dynamics clustering (formula (16), formula (18), formula (19)) is performed (yes), and hyperparameters are estimated online (formula (30), formula ( 31)) Yes (yes). The difference from the second comparison method is that a trajectory is modeled while temporally switching the motion pattern shared by each tracking target.

[Experiment using artificial data]
In this experiment, tracking and clustering of moving mass points in a virtual two-dimensional space of [0: 200] × [0: 200] are tasks. An example of the data is shown in FIG. FIG. 14 is a diagram corresponding to artificial data, where (a) is observation data (indicated by “◯”), and (b) is correct data. In FIG. 14B, “●” indicates a tracking target, and “◯” indicates noise. The hidden state quantity of the tracking target (mass point) is a two-dimensional real vector representing the position of each target. The observation amount is also a two-dimensional real vector. These represent information on the target position degraded by noise. These vectors also function as integer vectors. Each object follows a random walk model shown in the following equations (34) and (35).

Here, N (•) represents a normal distribution, and v (t) and w (t) are each sampled from the normal distribution. In this experiment, each object is assumed to generate a trajectory and output while randomly switching the four dynamics patterns shown in Table 2. In addition, {} ^T means a transposed matrix or vector, and diag {} means a diagonal matrix.

In all patterns, r is a zero vector. The initial value θ (0) of NIW is set so that the average value of ξk (t) and ψ _k (t) becomes the value of the following equation (36).

Other parameters necessary for estimating the track of the tracking target are listed. The time series data is 300 steps. The number of particles was S = 300, and the CRP concentration parameter was γ = 2. It is assumed that a new tracking target is generated with a probability of P _b = 0.1 at each time. The tracking target (c _i (t−1) = 1) existing in the scene is assumed to disappear from the scene with the probability (annihilation probability) P _d calculated by the equation (37). Note that t _n represents the last time when at least one observation amount y _m with j _m (·) = i exists. In addition, λ = 0.1.

Next, we will examine the experimental results. Table 3 shows the average value of the data log likelihood at each time.

  The value of the artificial data column (same as the actual moving image column value) indicates the possibility of generating observation data from the estimated hidden variables (that is, the model itself). Indicates a high degree. From Table 3, it can be confirmed that parameter estimation of Kalman filter and its clustering improve log likelihood. This indicates that the method has obtained better models and parameters through online tracking and clustering.

  FIG. 15 is a distribution diagram (average value distribution diagram of velocity noise in the artificial data experiment) at the final time of the parameter q of the normal distribution noise in the present method (π = 0.2). q represents the average velocity bias of the tracking target. The generated artificial data has four patterns of q as shown in Table 2. Each plot point in FIG. 15 represents an average value of q of one dynamics cluster. The size number represents the data size of each cluster (how many objects belonged to that cluster). Large four major clusters (size> 100) were acquired, and they obtained values close to the designed “ground truth”. In other words, this method succeeds in clustering dynamics patterns and estimating their parameters.

[Experiments with actual video data]
Next, an experiment using actual moving image data will be described. Tracking and clustering of feature points extracted from pedestrians in a scene are performed using a 320 × 240 pixel moving image captured by a digital camera.

  Target feature points were extracted as follows. First, background difference (comparing a background image obtained by photographing a specific object under a specific condition and an observation image separately photographed under the same condition) is performed, and foreground pixels are extracted by binarization. Subsequently, only black pixels are selected from the foreground portion and quantized to a small number of cluster centers by mean shift clustering. These cluster center coordinates are used as feature points. The position vector (two-dimensional vector) of these feature points is an observation amount. This position vector is generally a real vector, but also functions as an integer vector. 1-3 feature points were extracted from approximately one pedestrian.

In this experiment, a color model was set for the data association distribution (formula (24)) (see Non-Patent Document 1). An 8-bit RGB (Red Green Blue) histogram is used for the model. A Bhattacharya coefficient is calculated between the histogram in the surrounding pixels of each feature point and the histogram to be tracked obtained at the previous time. The histogram of the object i is the average of the histograms of observation points where j _m (t−1) = i. The probability that j _m (t) = i is calculated by the following equation (38).

  Here, σ is a constant parameter (σ = 0.15) set in advance, and d (m, i) is a Bhattacharya coefficient between the m-th observation data and the histogram of the i-th tracking target.

The same state space model as the previous artificial data experiment is used. The initial value θ0 of NIW was set so that the average value of ξ _k (t) and ψ _k (t) would be the value of the following equation (39).

List other parameter settings. The number of frames of the moving image is 200 frames. These 200 frames were extracted by thinning out from 1000 frames captured at 30 FPS (Frame Per Second). The total number of particles was S = 500, and the CRP concentration parameter was γ = 0.1. The generation probability of the new tracking target is P _b = 0.1, and the probability that the target in the scene disappears is the probability P _d calculated by Equation (37) (assuming λ = 0.1).

  Next, we will examine the experimental results. As described above, the actual moving image column in Table 3 indicates the average log likelihood. As in the case of the artificial data experiment, it can be confirmed that the log likelihood is improved by this method. Next, the distribution of q, which is the bias of the average speed, is shown in FIGS. Distributions in the 10th and 199th (final) frames when π = 0.1, respectively.

  Although it is real data, there is no “ground truth”, but when moving image data was observed, it was found that most pedestrians moved only in the vertical direction in the screen. Also, in order to enter the store occasionally, the direction is changed to the left and right. From the above, it is expected that there are basically two types of movement patterns in the vertical direction. In a state where the number of input frames is small (FIG. 16), since such a trend cannot be estimated yet, an individual dynamics pattern is set for each target to increase the likelihood. However, when information is accumulated (FIG. 17), only two types of dynamic clusters in the vertical direction finally exist.

  Finally, continuous snapshots (eight) of tracking results are shown in FIG. “# 0” or the like indicates a frame number along the time series. FIG. 18 is a photograph taken from above of the atrium portion of the shopping mall. The left and right ends of each screen are the second floor corridor portion of the shopping mall, and the center portion of the screen is the first floor passage. Each store on the first floor exists under the corridor on the second floor, and light leaks at their entrance. Actually, in the first floor passage, most people walking on the right side of the screen walk upward, and most people walking on the left side of the screen walk downward.

  The estimated position of the object is indicated by a rectangle. In addition, there is a part with a plurality of rectangular tracking marks for one person, but this is not tracking the person directly, but from the person and the obscured part to the black area (hair part and pants This is because the center point is extracted individually and the center point thereof is tracked, and the tracking result is correct.

  The number attached to the top of each rectangle corresponds to the index (z) of the dynamics cluster. The cluster that has acquired the bias to move upward corresponds to “1”, and the downward dynamics corresponds to “0”. The two people on the right side of the screen who walked up to the frame 100 are reversed to enter the store at the frame 125. In this method, it is detected and the dynamic pattern of the corresponding feature point is switched from “1” upward to “0” downward. That is, it can be confirmed that the tracking of a plurality of moving objects is performed with higher accuracy by this method.

It is a functional block diagram which shows typically the structure of the moving image processing apparatus which concerns on this embodiment. It is an image figure of multiple object tracking. It is a graphical model of the generation model used for the calculation of this method. It is a flowchart which shows the whole process of this method. It is a flowchart which shows the detail of step S3 of FIG. It is a flowchart which shows the detail of step S34 of FIG. It is a flowchart which shows the detail of step S343 of FIG. It is a flowchart which shows the detail of step S344 of FIG. It is a flowchart which shows the detail of step S345 of FIG. It is a flowchart which shows the detail of step S346 of FIG. FIG. 10 is a flowchart showing details of step S35 in FIG. It is a flowchart which shows the detail of step S4 of FIG. It is a flowchart which shows the detail of step S5 of FIG. It is a flowchart which shows the detail of step S6 of FIG. It is a figure corresponding to artificial data, (a) is observation data, (b) is correct data. It is a distribution map in the last time of parameter q of normal distribution noise in this technique (π = 0.2). It is a distribution map of q which is a bias of average speed in an experiment using an actual moving image (10th frame). It is a distribution map of q which is a bias of average speed in an experiment using an actual moving image (199th frame). It is a continuous snapshot (8 sheets) of the tracking result in the experiment using an actual moving image.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Moving image processing apparatus 2 Calculation means 3 Storage means 4 Input means 5 Output means 6 Bus line 21 Hidden variable set estimation part 22 Hidden state estimation part 23 Resampling part 24 Hyper parameter part 211 Hidden variable determination part 212 Hidden variable weight calculation part 2111 Number of movement target determination unit 2112 Operation pattern number determination unit 2113 Operation pattern parameter determination unit 2114 Movement target correspondence determination unit

Claims (7)

When observation data regarding a moving image in which a plurality of moving objects exist is given as an input, a time evolution function of a hidden state quantity indicating an estimated value of a true value regarding the plurality of moving objects using a probabilistic generation model; A moving image processing method by a moving image processing apparatus for estimating the number of hidden states of a plurality of moving objects by estimating the number of patterns and each feature of dynamics indicating an observation function for calculating observation data, ,
The moving image processing device includes the observation data, a hidden state quantity for each of the moving objects, a plurality of hidden variables indicating a relationship between the observation data and the hidden state quantities for each of the moving objects, and each of the hidden variables. Storage means for storing the weight and the hyper parameter of the dynamics, and an arithmetic means,
The computing means is
A hidden variable set estimation unit estimating the weights of the plurality of hidden variables and the hidden variables based on the observation data;
The hidden state estimation unit is configured to generate the hidden state based on the observation data, the hidden state amount at a time immediately before the currently calculated time, and the plurality of hidden variables estimated by the hidden variable set estimation unit. Estimating a state quantity;
Based on the plurality of hidden variables estimated by the hidden variable set estimation unit and the hidden state quantity estimated by the hidden state estimation unit, the resampling unit performs an estimation result of the hidden variable and the associated hidden Re-sampling by estimating the high posterior distribution from the particle weight distribution indicating the state quantity estimation result;
A hyperparameter update unit updating the hyperparameters of the dynamics based on a resampling result by the resampling unit;
The moving image processing method is characterized in that the hidden state amount of the plurality of moving objects is estimated by the hidden state estimation unit by repeatedly executing. In the step of estimating the weight of each of the plurality of hidden variables and the hidden variables based on the observation data, the hidden variable set estimation unit,
The hidden variable determination unit is based on the hidden state quantity at the previous time, the plurality of hidden variables at the previous time, and the weights of the hidden variables at the previous time, respectively. Estimating the plurality of hidden variables for each particle;
A hidden variable weight calculation unit, for each particle, based on the plurality of hidden variables estimated by the hidden variable determination unit, to estimate the weight of each hidden variable;
The moving image processing method according to claim 1, wherein: The hidden variable determination unit is configured based on the hidden state quantity at the previous time, the plurality of hidden variables at the previous time, and the weights of the hidden variables at the previous time. In the step of estimating the plurality of hidden variables for each particle,
Accepting the input of the hidden state quantity at the previous time, the plurality of hidden variables at the previous time, and the observation data at the currently calculated time,
A step of determining a hidden variable representing whether or not each of the moving objects exists in the moving image at a certain time among the plurality of hidden variables;
An operation pattern number determining unit determining the hidden variable representing an identifier of the movement target operation pattern among the plurality of hidden variables;
An operation pattern parameter determination unit determining a parameter of the operation pattern;
A movement target correspondence determining unit determining the hidden variable representing the correspondence between the observation data and the movement target among the plurality of hidden variables;
The moving image processing method according to claim 2, wherein: When observation data regarding a moving image in which a plurality of moving objects exist is given as an input, a time evolution function of a hidden state quantity indicating an estimated value of a true value regarding the plurality of moving objects using a probabilistic generation model; A dynamic image processing device that estimates the number of patterns and each feature of a dynamics indicating an observation function for calculating observation data, and estimates the amount of hidden states of the plurality of moving objects,
The observation data, hidden state quantities of the moving objects, a plurality of hidden variables indicating relationships between the observation data and the hidden state quantities relating to the moving objects, weights of the hidden variables, and the dynamics Storage means for storing the hyperparameters of
Based on the observation data, the plurality of hidden variables, and a hidden variable set estimation unit that estimates the weight of each hidden variable,
Hidden estimation of the hidden state quantity based on the observation data, the hidden state quantity at the time immediately before the currently calculated time, and the plurality of hidden variables estimated by the hidden variable set estimation unit State estimation part,
Based on the plurality of hidden variables estimated by the hidden variable set estimation unit and the hidden state amount estimated by the hidden state estimation unit, an estimation result of one hidden variable and an associated hidden state amount A resampling unit that performs resampling by estimating a high posterior distribution from the distribution of particle weights indicating
A hyperparameter update unit that updates a hyperparameter of the dynamics based on a resampling result by the resampling unit,
And the hidden state of the plurality of movement targets by the hidden state estimation unit by repeating the processing by the hidden variable set estimation unit, the hidden state estimation unit, the resampling unit, and the hyperparameter update unit. Computing means for estimating the quantity;
A moving image processing apparatus comprising: The hidden variable set estimation unit includes:
For each particle, based on the weight of the hidden state quantity at the previous time, the plurality of hidden variables at the previous time, and the weight of the hidden variable at the previous time, A hidden variable determination unit that estimates a plurality of hidden variables;
Based on the plurality of hidden variables estimated by the hidden variable determination unit, for each particle, a hidden variable weight calculation unit that estimates the weight of each hidden variable;
The moving image processing apparatus according to claim 4, further comprising: The hidden variable determination unit
Accepting the input of the hidden state quantity at the previous time, the plurality of hidden variables at the previous time, and the observation data at the currently calculated time,
Among the plurality of hidden variables, a moving object number determining unit that determines whether or not each of the moving objects exists in the moving image at a certain time,
Among the plurality of hidden variables, an action pattern number determining unit that determines the hidden variable representing an identifier of the movement target movement pattern;
An operation pattern parameter determination unit for determining a parameter of the operation pattern;
Among the plurality of hidden variables, a movement target correspondence determining unit that determines the hidden variable representing the correspondence between the observation data and the movement target;
The moving image processing apparatus according to claim 5, further comprising:   A moving image processing program for causing a computer to function as the moving image processing apparatus according to any one of claims 4 to 6.