JP4210292B2 - Image change prediction method and image change prediction apparatus - Google Patents

Description

  The present invention relates to an image change prediction method and an image change prediction apparatus for predicting time-series image changes.

  Unsteady behavior patterns represented by meteorological phenomena are handled in the field of weather prediction from weather information observed by images and sensors. This behavior pattern becomes a characteristic pattern in which various meteorological physical quantities such as airflow, cloud physics, heat radiation, humidity, temperature, atmospheric pressure, and latent heat are entangled. In the weather radar image, such characteristic patterns are observed every moment, and it is practiced to predict the near-future changes of the visual and intuitive patterns by experienced persons. On the weather radar, in the case of a two-dimensional image, the convection phenomenon accompanying the rise and fall of the air current is projected and displayed as a pattern.

  As a prediction method using a computer, a pattern matching method often used in the field of image processing is applied using two consecutive images.

  The signal processing method is widely applied as a maximum likelihood estimation method from a time series pattern (data) in which a Kalman filter is buried in noise. However, it is assumed that the probability density function of the signal state quantity is unimodal. There are also models such as an extended Kalman filter and a nonlinear Kalman filter, which improve the ability to follow a sudden change in data. Data prediction also uses various Kalman filters to achieve accurate predictions that suppress the effects of noise.

There are methods for predicting weather radar images based on the advection-diffusion equation (Patent Document 1 and Patent Document 2). In this method, a motion vector (velocity) is estimated from the two images using an optical flow method, and the latest one image pattern is predicted based on the motion vector.
Japanese Patent No. 3375038 Japanese Patent No. 3377075

  First problem) This is one of the factors that greatly reduce the accuracy when the computer predicts the radar pattern. Information with high spatial and temporal resolution determines accuracy, but in general, information exceeding meteorological radar information is hardly obtained.

  Second problem) In addition to strong nonstationarity, it is clearly difficult to predict deterministically that appearance / hiddenness is observed.

  Third problem) The strong non-stationarity suggests that there are almost no patterns similar to the past, and even if a similar pattern to the current pattern is searched from the past patterns, it is similar to the past. It is difficult to predict because there is nothing to do.

  Fourth problem) Since the Kalman filter assumes unimodality and Gaussianity for the probability density function of the signal state quantity, the discontinuity of data due to the visibility and hiding of weather radar patterns Because it has a non-Gaussian distribution, unimodality cannot be used.

  Fifth problem) As for weather phenomena in the old era, there are many things that remain only in the memory of experienced people, and how to reflect their passion on the learning and recognition (system) method is a challenge is there.

  In the present invention, 1) as a probabilistic prediction method, a weather pattern prediction method based on a method called CONDENSATION method, which is one of non-stationary prediction methods in computer vision, is applied. 2) The probability density function of the state quantity of data can be handled with a multimodal distribution. 3) Using a model in which the amount of variation is added to each of the advection diffusion equation and the velocity, the amount of variation is estimated by the CONDENSATION method. 4) The fluctuation amount is updated based on the change in the magnitude of the velocity and the magnitude of the image intensity from the past through the Lorentz distribution function which is one of the robust functions.

  According to the present invention, stable and accurate prediction can be realized even when there is a sharp change in the intensity or shape of an image pattern, which is seen when an image data pattern is observed.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a configuration diagram of an image change prediction apparatus according to the present embodiment.

  The apparatus includes an input unit 100, a storage unit 110, a speed estimation unit 120, a calculation unit 130, a change amount update unit 140, and an output unit 150.

  In this apparatus, the data input unit 100 inputs a plurality of time-series images, for example, weather radar images and general images, and the images are stored in the storage unit 110. The speed estimation unit 120 estimates a speed from an optical flow from two consecutive images, and the calculation unit 130 predicts a change in the image from the latest image and the speed. Note that the computing unit 130 performs image intensity prediction using an advection diffusion equation and velocity prediction using a Navier-Stokes equation and a continuous equation. The fluctuation amount update unit 140 updates the amount of change obtained by the prediction by the CONDENSATION method so that the prediction follows the actual change. Note that the fluctuation amount updating unit 140 updates the amount of change obtained by the prediction based on the speed change and the image intensity change from the past using the Lorentz distribution function which is a robust function. Then, the output unit 150 outputs (displays, prints, etc.) an image (predicted image) obtained via the change amount update unit 140.

  As a result, stable prediction accuracy is ensured regardless of discontinuous changes with respect to visibility due to steep development or decline. Further, the CONDENSATION method absorbs speed fluctuations and image intensity fluctuations due to discontinuities.

  The speed estimation unit 120, the calculation unit 130, the change amount update unit 140, and the output unit 150 may constitute the apparatus. That is, the input unit 100 and the storage unit 110 may be configured as separate devices.

  Subsequently, processing by each means will be described in detail.

  FIG. 2 is an explanatory diagram of an example of prediction of an image pattern by an advection diffusion equation and an optical flow.

  A weather radar pattern is observed in two consecutive images at time n-1 and time n, and the velocity is estimated in pixel units from the two images by the optical flow method (reference document 3). Next, the change of the image pattern after time n + 1 is obtained by calculating the time evolution according to the advection diffusion equation (Equation (1)).

The speed is described below as a known quantity. If the image intensity is I, and pixel (x, y), time or discretized time n is defined as I = I (x, y, n) (Reference Document 1). Here, Δt is 1.0. U is a two-dimensional velocity field vector, λ is a diffusion coefficient, and is 1.0 here. ∇ is a primary spatial differential operator, ∇ / ∇ is a secondary spatial differential operator, and [•] is a vector dot product.

  The calculation is performed by discretizing Equation (1) according to the difference method and repeating the integration along the two-dimensional image plane and the discrete time axis. However, the image intensity I of the initial image is the intensity of the input image (time n), U = (u, v) is a velocity vector, and (u, v) is a two-dimensional velocity component. Here, a predetermined number of time integrations is set, and the calculation is performed by that number. It is better to increase the number of integrations by taking a smaller time interval from the stability condition of the calculation. Even if the diffusion effect is not expected, by setting a small value as λ <0.0001, it is possible to suppress the vibration from the error and achieve the stabilization of the calculation.

  Detection of the observed weather radar time-series image movement (convection velocity component) is based on the optical flow method, but it is at the current time, so it is particularly necessary to predict the movement of the vortex and the future of the advection. Furthermore, since only radar image information is used, there is a problem that motion information is limited only to a region where an image pattern exists. This means that, for the purpose of predicting an image pattern, there is no speed for proceeding to the next region (pixel), and therefore large advection prediction cannot be performed. Therefore, a method for estimating some motion even in an area where no image pattern exists is indispensable. The remaining problem is how to change the detected advection speed, although the accompanying movement and deformation of the cell also changes the advection speed. Therefore, this apparatus performs a two-stage process.

  In the first stage, the advection velocity is interpolated in the entire image by repeatedly using the moving average filter (equation (2)) with the detected advection velocity as an initial value.

  In the second stage, the speed change is predicted by the equation (3), that is, the Navier-Stokes (NS) equation and the continuous equation (collectively referred to as NS equation). In Equation (3), ρ, p, and ν are density, pressure (atmospheric pressure), and viscosity coefficient, respectively. That is, non-linear and non-stationary velocity field prediction is performed by the equation (3). The continuity equation imposes an incompressible condition on the object by setting the velocity divergence to zero.

  The NS equation of equation (3) can be solved by the SOR method by the difference method according to the HSMAC method (reference document 4), using this as a nonlinear simultaneous equation.

The NS equation is widely applied from laminar flow to turbulent flow in the field of computational fluid dynamics, but is rarely used in image prediction.

  FIG. 3 is a diagram illustrating the CONDENSATION method for predicting time-series data. This method is also called a particle filter (PF). In PF, there is no assumption of unimodality or Gaussianity in the probability density function of the state. Therefore, an arbitrary probability density function can be handled in the PF. This is robust against discontinuities in spatio-temporal data such as weather radar patterns appearing suddenly developing, disappearing due to decline, or people and cars appearing and hiding between buildings and trees. There is a feature that there is. The outline of PF is described below.

The PF is one of means for sequentially estimating the state vector at each time. A state vector at time t is a vector x _t in the state space S, and an observation vector is y _t . Further, a history of observation vector y _t Yt = a {y1, y2, y3 ··· yt }. The general state estimation is formulated as a problem of estimating the probability density function p (x _t / Y _t ) of the state vector x _t .

For PF, as shown in FIG. 3, using N discrete hypothesis s _t and weights [pi _t corresponding thereto in the state space S, the probability density function p (x _t / Y _t) number of This is expressed as a hypothesis group {(s _t ; π _t )} (i = 1... N) formed from discrete hypotheses. Therefore, an arbitrary non-Gaussian probability density function can be approximated by p (x _t / Y _t ).

  The procedure for updating the PF is shown below. Here, processing is performed by the prediction unit and the observation unit.

In the prediction unit, (1-a) after selecting a hypothesis based on the probability distribution of the hypothesis group, (1-b) applying a predefined motion model to the selected hypothesis. Here, the variables of the motion model p (x _t / Y _t ) are the image intensity change by the advection diffusion equation (Equation (1)) and the velocity by the Navier-Stokes (NS) equation and the continuous equation (Equation (3)). Showing change.

  (1-c) By repeating this N times, N new hypotheses are generated. The observation unit (2-a) determines a weight corresponding to each hypothesis based on the observation model obtained from the observation amount at the current time, and (2-b) newly acquires a hypothesis group at the current time. To do. Note that the motion model used in (1-b) is generally composed of movement and diffusion. In movement, a hypothesis is moved by a predetermined method, and random noise is added to the hypothesis moved by diffusion.

  The problem with PF is the problem of reduced accuracy when the total number of hypotheses is limited. When the dimension of the state vector increases in a normal PF, the number of hypotheses required to maintain the approximate accuracy of the probability density function increases almost exponentially. For this reason, if the number of hypotheses is limited to such an extent that the real-time property can be maintained, a phenomenon that the density of hypotheses in the state space is insufficient depending on the dimension of the state vector may occur, and the accuracy may be greatly reduced. Therefore, in order to maintain an accuracy of a certain level or more using PF, a device for reducing the number of hypotheses is necessary.

  For this purpose, there are methods such as resampling, important sampling, and parted sampling.

  By these methods, it is possible to improve the estimation accuracy with almost the same calculation cost.

Generation of a new hypothesis group {(s _t ⁽ⁱ⁾ ; π _t ⁽ⁱ⁾ )} from the hypothesis group {(s _t-1 ⁽ⁱ⁾ ; π _t-1 ⁽ⁱ⁾ )} Prediction unit: new hypothesis { Generation of s _t ⁽ⁱ⁾ } (1-a) {(s _t-1 ⁽ⁱ⁾ ; π _t-1 ⁽ⁱ⁾ )} from hypothesis s _t-1 based on weight π _t-1 ⁽ⁱ⁾ (1-b) Generate a hypothesis s _t ⁽ⁱ⁾ from the motion model p (x _t | x _t-1 = s _t-1 ') (1-c) (1-a) and (1-b ) N times to obtain N hypotheses {s _t ⁽ⁱ⁾ } Observation unit: calculation of weight {π ⁽⁰⁾ } and determination of {s _t ⁽ⁱ⁾ ; π _t ⁽ⁱ⁾ } (2-a ) Calculate weight π _t ⁽ⁱ⁾ from observation model p (y _t | x _t = s _t ⁽ⁱ⁾ } Apply (2-b) (2-a) to all _st ⁽ⁱ⁾ , {s _t ⁽ⁱ⁾ ; π _t ⁽ⁱ⁾ } is acquired. In the apparatus, the above-described motion model corresponds to Formula (1) and Formula (3).

(Diffusion control method)
In Equation (4), w1 and w2 indicate fluctuation amounts related to image intensity and speed, respectively, and these are estimated and updated by the CONDENSATION method. The diffusion mentioned above refers to changes related to various disturbances, data discontinuities, and the like. In the present apparatus, a diffusion control method that varies the width of the diffusion according to the state from a certain level is applied to cope with a more unsteady change. Here, with respect to image intensity and speed, when the changes at time t and t−1 at each two-dimensional pixel are z1 and z2 (formula (5)), a nonlinear robust function as shown in formula (6) In other words, the diffusion levels w1 and w2 are changed by a size proportional thereto. σ is a variance, and is empirically set to 2.0.

  FIG. 4 is a comparative example of the present apparatus in which the change of the image pattern is predicted based on the conventional method using the advection diffusion equation and the NS equation and the CONDENSATION method.

  When the image pattern 200 at a certain time is predicted by the conventional method, an image pattern 210 having a shape different from that of the actual image pattern 200 is obtained due to the sudden appearance and hiding of the pattern. On the other hand, according to the present apparatus, an accurate image pattern 220 including vortex characteristics was obtained.

  FIG. 5 is a comparative example of a result of artificially generating and predicting an image pattern in which the image intensity changes sharply. According to this apparatus, the position error with the true position data is extremely small, and it follows a steep movement and a change in image intensity. On the other hand, according to the conventional method, when the image intensity is greatly changed, the prediction is largely off.

  As described above, according to the present apparatus, the visible / hidden problem on the radar image associated with the development and decline of the pattern that changes from moment to moment is alleviated, and a complex motion pattern can be predicted easily and accurately. It can be applied not only to weather patterns, but also to events that follow and predict the movement of traveling vehicles and pedestrians that are visible and hidden in buildings and trees, and is highly versatile. In other words, the present invention may be applied to a vehicle or a human motion monitoring camera or a security system that is often seen in a real environment change.

  A computer program that causes the apparatus to execute the image change prediction method described so far is stored in a computer-readable recording medium such as a semiconductor memory, a magnetic disk, an optical disk, a magneto-optical disk, or a magnetic tape, and is displayed. The computer program may be distributed or transmitted via a communication network such as the Internet.

(Reference 1) Nonlinear phenomena in fluids-Mathematical analysis and its applications-: Keisuke Ikeda: Asakura Shoten (Reference 2) M. Isard and A. Blake, “Icondensation: Unifying low-level and high-level tracking in a stochastic framework ”, Proc. (European Conference on Computer Vision) ECCV '98, pp.893-908, 1998.
(Reference 3) JLBarron, DJFleet, and SSBeauchemin, “Performance of optical flow techniques”, IJCV, vol.12, no.1, pp.43-77, 1994.
(Reference 4) Tadaichi Arakawa, “Computational Fluid Engineering”, University of Tokyo Press, 1995.

It is a block diagram of the image change prediction apparatus which concerns on this Embodiment. It is explanatory drawing about the example of the prediction of the image pattern by an advection diffusion equation and an optical flow. It is a figure shown about the CONDENSATION method which estimates time series data. It is the comparative example of this apparatus which predicted the change of the image pattern based on the conventional method and the CONDENSATION method which predicted using the advection diffusion equation and NS equation. This is a comparative example of the result of artificially generating and predicting an image pattern in which the image intensity changes sharply.

Explanation of symbols

100 Input means 110 Storage means 120 Speed estimation means 130 Calculation means 140 Change amount update means 150 Output means

Claims (5)

The input means configured in the image change prediction device inputs a plurality of time-series images,
The image is stored in storage means configured in the image change prediction device,
The speed estimation means configured in the image change prediction apparatus estimates the speed by optical flow from two consecutive images,
The calculation means configured in the image change prediction device predicts a change in image from the latest image and the speed,
A change amount updating means configured in the image change prediction device updates a change amount obtained by the prediction by a CONDENSATION method so that the prediction follows an actual change,
The output means configured in the image change prediction apparatus is an image change prediction method for outputting an image obtained via the change amount update means ,
The calculation means performs image intensity prediction using an advection diffusion equation, and speed prediction using a Navier-Stokes equation and a continuous equation,
In order for the change amount updating means to cope with non-stationary and non-linear changes, the change amount of the image intensity and the change amount of the speed obtained by the prediction are applied to the probability density function, and the change amount is determined by the CONDENSATION method. When updating

An image change prediction method characterized in that z1 and z2 in the above case are put into a robust function and the magnitudes of w1 and w2 are changed by a magnitude proportional thereto .   2. The change amount updating unit updates the change amount obtained by the prediction based on a speed change and an image intensity change from the past using a Lorentz distribution function which is a robust function. Image change prediction method. Input means for inputting a plurality of time-series images, storage means for storing the images, speed estimation means for estimating the speed by optical flow from two consecutive images, and the latest image and the image from the speed A calculation means for predicting a change in the output, a change amount update means for updating the change amount by a CONDENSATION method so that the prediction follows an actual change, and an output for outputting an image obtained via the change amount update means Means and
The calculation means performs image intensity prediction using an advection diffusion equation, and speed prediction using a Navier-Stokes equation and a continuous equation,
In order for the change amount updating means to cope with non-stationary and non-linear changes, the change amount of the image intensity and the change amount of the speed obtained by the prediction are applied to the probability density function, and the change amount is determined by the CONDENSATION method. When updating

And z1 and z2 in the case of a, placed in robust functions, in magnitude proportional to it, the image change projection apparatus according to claim Rukoto changing the size of the w1 and w2. The change amount update means, a change amount obtained by said prediction, using a Lorentz distribution function is a robust function, according to claim 3, wherein the update based on the speed change and image intensity variation from the past Image change prediction device. Computer program for executing the image change prediction method according to claim 1 or 2, wherein the image change predictor.

Images