JP2003288603A - Method, device, program, and recording medium for image scene prediction - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To generate a prediction pattern having high precision from image information including a pattern in which shapes and textures change intricately which cannot be approximated using a rigid body model. <P>SOLUTION: Using various sensors, measurement instruments, and the Web as information sources, time-series image data is input by an image input part 1 and accumulated in an image accumulation part 2. A parameter estimation part 3 estimates a speed component which is a parameter required in an advection term based on a so-called optical flow procedure only using the current and part consecutive image data. A prediction calculating part 4 predicts changes in the pattern of the image up to the necessary image future and the output part 5 outputs the predicted image. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

FIELD OF THE INVENTION The present invention relates to a method for predicting changes in imaged patterns obtained from WEB and various observation sources.

[0002]

2. Description of the Related Art Conventionally, most of the methods for predicting the change in gray value from a video pattern including a complicated shape or pattern assume a rigid body model, and the pattern shape at the start of prediction does not change during the prediction. Then, the result of the parallel movement of the initial pattern (the pattern at the current time immediately before the start of the prediction) was used as the prediction result. In the field of coding,
The mainstream method is to divide an image into a plurality of sub-blocks, such as MPEG4, a weather radar, a satellite image, and a lightning strike image, and to independently perform parallel translation of each divided block for prediction.

FIG. 14 shows a conventional weather radar image prediction method. Although four frames are shown, with respect to a swirling low-pressure pattern, in the conventional method, the similarity is calculated between successive time-series images (frames) by the cross-correlation coefficient, and Is used as the movement vector, and linear extrapolation is used as the pattern prediction. However, the basic parameter is the number of divided images. The number of divisions is 4 frames, and the number of sub-blocks obtained by dividing one initial image is 25, 10.
In both cases, there is a problem that blocks are unnaturally separated from each other.

[0004]

Since the conventional method described above assumes a rigid body model, it is possible to accurately predict from a video including a pattern in which the shape and texture change intricately, which cannot be easily approximated by the rigid body model. The pattern could not be generated.

An object of the present invention is to provide a video scene prediction method and apparatus for accurately generating a prediction pattern only from video information including a pattern whose shape and texture change intricately, which cannot be easily approximated by a rigid body model. Especially.

[0006]

In order to achieve the above object, a video scene prediction method of the present invention comprises a step of inputting time-series image information, a step of accumulating the input image information. The time-series image is approximated by the five patterns of development, decline, and stagnation obtained by classifying diffusion, advection, and difference images of time-series images of different time series, and using the parameters required for the advection term. For a certain velocity component, using only changes in the pattern of the current and past consecutive images, an estimation based on the optical flow, a step of repeating the prediction calculation up to the required prediction time, and a prediction result are displayed. Have steps.

By inputting a time-series image, integrating partial differential equations and difference functions relating to each element pattern such as advection with a prediction equation including time evolution, and repeating time products,
A (future) predicted image can be obtained.

In the meteorological field, the present invention leads to disaster prevention as meteorological prediction, and in the case of meteorology, it predicts the amount of precipitation by a precipitation radar image, and if it is a satellite image, it predicts clouds and clear from visible images and infrared images, and from lightning images If the forecast integrates precipitation and lightning strike information, concentrated heavy rain and thundercloud forecasts can be made, and if it is a general image other than weather, sensing of the wind direction and strength from the scene where the trees sway, wave forecasting to predict changes in ripples, It is not possible to clearly define the center of gravity such as prediction of vehicle patterns from traffic congestion and traffic flow images, prediction of human flow, change of blood flow patterns in medical images and prediction of cognitive processes, etc. It can be applied to various fields to create information with new value by predicting future video information in images and video scenes.

[0009]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Next, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram of a video scene prediction apparatus according to an embodiment of the present invention.

The video scene prediction apparatus of this embodiment comprises an image input section 1, an image storage section 2, a parameter estimation section 3, a prediction calculation section 4 and a display section 5.

Using various sensors, observation equipment, and WEB as information sources, time series images are input from the image input section 1 and stored in the image storage section 2. The parameter estimation unit 3 automatically estimates the parameters required by the mathematical equation every time the time-series image is updated, and the prediction calculation unit 4 predicts the change in the pattern in the image until the prediction time required. The image predicted by the display unit 5 is displayed.

FIG. 2 shows five basic spatiotemporal pattern models. Various input image patterns are approximated by five spatiotemporal patterns. 2 (1), (2),
(3), (4), and (5) are examples of changes in each pattern of anisotropic diffusion, advection with flow, decline / suction, springing / developing, and stagnant. Each pattern shows a characteristic change on a pixel-by-pixel basis over time. The given initial pattern shape is arbitrary.

FIG. 3 shows the instability of the prediction pattern due to the difference in the order of difference approximation. Input image is 100 × 1
A circle having a gray value of 120 was generated and arranged in the center of the number of 00 pixels. The radius of the circle is 20 pixels. Pixel value is 2 except circles
55. One pixel has 8-bit gradation. Using the advection equation (Equation 1), the difference in the change of the pattern when the order of the difference approximation is changed is shown.

[0015]

[Equation 1] The advection equation is one of the basic equations widely used in the field of fluid dynamics. In the formula, the grayscale value I is the position of one point in two dimensions.

[0016]

[Outer 1] , Time t, speed

[0017]

[Outside 2] Has as a parameter. The right side is the operator

[0018]

[Outside 3] , Which is the first-order spatial derivative of the gray value. By the difference method, the future gray value change can be easily obtained by discretizing the time and space and calculating the time evolution, as in Expression (2). The initial image gray value is here the gray value of the circle itself. The time term is the forward difference,
The spatial term is the case of the first order. In the case of the third order, the formula (3) is obtained. In discrete terms, Δt, Δ
x and Δy are discrete widths in the time, horizontal and vertical directions, respectively.

[0019]

[Equation 2] In this simulation, all were set to 1. Regarding the velocity field, a constant velocity is applied to the entire surface in the diagonally lower right direction. As a result of the numerical experiment, the difference in the prediction pattern due to the difference in the order of the difference approximation is the first approximation 1 and the third approximation 2. Even if the velocity field and time integration width are the same, in the case of the first-order approximation, the vibration solution has occurred midway,
Unstable forecast results are coming out. On the other hand, in the case of the third-order approximation, almost no vibration solution is seen until the end, and it can be said that it is stable. From this, it is appropriate for long-term pattern prediction to approximate the advection equation in a higher order such as a third order. Here, the basic advection equation is derived.

A variable

[0021]

[Outside 4] As a change of the time and the one-dimensional movement in the minute time of, the Taylor expansion is performed up to the second order as shown in Expression (4).

[0022]

[Equation 3] here,

[0023]

[Outside 5] If there is no change such as development or decline,

[0024]

[Outside 6] A one-dimensional advection basic equation for can be derived. In particular,
If the spatial differentiation is expanded, the two-dimensional advection equation (Equation (5))
To get

[0025]

[Equation 4] Next, when the equation (5) is discretized by the upwind difference method, the equation (6) is obtained. This method is characterized in that the difference error due to the influence of error propagation is controlled by adaptively selecting and calculating the forward difference and the backward difference according to the flow direction. Equation (6) is a first order.

[0026]

[Equation 5] The characteristic is analyzed by expanding the expression (6) by Taylor.

[0027]

[Equation 6] The second term on the right side of the equation (7) is a numerical diffusion term and has an effect of smoothing the numerical solution. The third term is the numerical variance term,
Vibrate the numerical solution. Next, a three-dimensional windward difference equation is derived from equation (6). When one point on the space is interpolated by a quadratic equation using three neighboring points, equation (8) is obtained.

[0028]

[Equation 7] When the equation (8) is combined into one using the absolute value of the velocity, the equation (9) is obtained.

[0029]

[Equation 8] To see the characteristics of equation (9), if you expand it by Taylor expansion,

[0030]

[Equation 9] Is included. For stabilization, the first term

[0031]

[Equation 10] To create an offsetting effect (Kawamura scheme). Furthermore, in order to mitigate the effect of the numerical diffusion term of the fourth derivative included in the second term of Expression (9),

[0032]

[Equation 11] Replace with. These are put together to obtain the third-order upwind difference equation (10) applied in this specification. Although it is one-dimensional here, it can be derived similarly in the case of two-dimensional.

[0033]

[Equation 12] The expression (10) is analyzed by Taylor expansion. It can be seen that the numerical diffusion effect due to the fourth derivative term is large depending on the magnitude of α.

[0034]

[Equation 13] FIG. 4 shows the difference between the isotropic diffusion method and the anisotropic diffusion method. Here, the input image (precipitation) pattern is an image relating to the rainfall intensity, which is obtained from the JMA radar image, and relates to the rainfall amount or the precipitation area of rain or snow. It has 8-bit gradation, and white is stronger and black is weaker in display. The precipitation pattern is 200
This is the so-called heavy rainfall pattern when the second highest rainfall ever recorded in Tokyo on July 4, 2000. The image size is 200 x 200 pixels, and one pixel is 2.5 km unit. Here, as a numerical experiment, the input initial image pattern (the weather radar image pattern I when a concentrated heavy rain pattern is shown) using the diffusion equation (Equation (11)) and the anisotropic diffusion equation (Equation (12)). Simulate how changes in.

[0035]

[Equation 14] The right side of the diffusion equation is the second derivative term for the gray value I. The diffusion coefficient λ. Controls the extent of spread per unit frame. The first term on the right side of the anisotropic diffusion equation describes the same isotropicity as the diffusion term, and the second term changes the diffusion coefficient according to the magnitude of the first-order gradient (edge) of the image gray value. . When the edge is large, the diffusion coefficient is small, and when the edge is small, the diffusion coefficient is large. This control mechanism has an advantage that the edge structure is easily maintained, although it has a diffusion effect. In addition, n in FIG.
It is the number of times in the middle of repeated calculation in 9).

When equations (11) and (12) are discretized by the difference method, equations (13) and (14) are obtained.
In the discrete term, Δt, Δx, and Δy are time,
It is a discrete width in the horizontal and vertical directions.

[0037]

[Equation 15] In this simulation, all were set to 1. Gray value I
The time development value of is the predicted gray value. In an actual precipitation pattern, the difference between isotropic diffusion (FIG. 4 (2)) and anisotropic diffusion (FIG. 4 (1)) changes remarkably as the number of iterations increases. The anisotropic diffusion method smooths the edges while preserving the edge structure, thus suppressing the apparent blurring of the pattern (reference: P. Perona and J. Malik, “Scal
e-space and edge detection using anisotropic diffu
sion ”, IEEE Trans. Pattern Analysis and Machine I
ntelligence, vol.12, no.7, pp.629-639, 1990.). The isotropic diffusion method performs uniform smoothing.

FIG. 5 shows the difference between the three states (development, decline, stagnation) of the difference image. The precipitation pattern is used here. A large typhoon pattern that landed in Kanto on July 8, 2000. The upper part of Fig. 5 (1) shows the actual precipitation pattern of continuous typhoons. There is a 10-minute observation time gap. In the lower part, the left side shows the result of the difference calculation as the development and decline regions, and the right side is the stagnation. Create a difference image from continuous time series images (Fig. 5 (1)),
There are three states (development, decline, stagnation) depending on the sign and the magnitude of it.
Can be created. The difference calculation is simply the amount of change in the gray value of the precipitation pattern I at each time per unit time Δt. I shown in equation (15)
(I, j, n) is the pixel value (i, j) at the discrete time n.

[0039]

[Equation 16] Histogram 2 of the difference value of the gray value when the difference is calculated
It is widely known that the Laplace distribution is approximately obtained by taking If it is roughly approximated to another level, it becomes easy to handle as a Gaussian distribution (equation (16)).

[0040]

[Equation 17] In order to classify the three states based on the difference value, three threshold values (equation (17)) regarding the density value are required.

[0041]

[Equation 18] At this time, if a Gaussian distribution is used as a model, it can be classified by the standard deviation on the histogram. That is,
When the standard deviation is 1, 60% of the total and when it is 2, 90
% Data is included. From this, assuming that development or decline is unlikely to occur in the whole change, the threshold corresponds to a data set whose standard deviation is more than 1 from the average. Since the stagnation is an intermediate change between development and decline, it is distributed in the vicinity of the average value smaller than the standard deviation 1.

FIG. 6 shows the difference in the estimated velocity fields when the regularization coefficient is constant. The pattern velocity was calculated by using two consecutive precipitation patterns of a typhoon as in FIG. 5 as an example.

Among various speed estimation methods, Horn is
& Schunkk, there is a pixel-based velocity field estimation method based on the standard regularization theory. In the present method, as the objective function, as shown in Expression (18), it is a functional with respect to the grayscale value I and the velocity field that is an unknown number.

[0044]

[Formula 19] The velocity should be calculated so as to minimize this function (reference: BKPHorn and BGSchunck, “Determining
Optical Flow ”, Artifical Intelligence, vol.17, n
o.1, pp.185-203, 1981.). To solve this equation,
Create Euler differentiation for velocity components, which are two unknowns. As a result, two linear equations can be created,
If the simultaneous linear equations are iteratively calculated by the Guass-Seidel method, the speed can be easily obtained. However, the weighting coefficient ω in the objective function is usually given a fixed value empirically. The time term is calculated using two image patterns.

The biggest drawback is how to estimate the optimal regularization factor. Here, six results are shown when the same regularization coefficient is applied to the entire image and the difference in velocity field is applied to a typhoon pattern having a vortex structure. The smaller the coefficient, the weaker the smoothing of the velocity field and the more noise. When the coefficient is large, the smoothing of the velocity field is strengthened, but the slow velocity component disappears or there is a drawback that it is underestimated.

FIG. 7 shows a sine wave pattern (FIG. 7 (1)) and an estimation example (FIG. 7 (2)) for estimating the optimum regularization coefficient.
Indicates. Here, the amplitude of the two-dimensional sine wave is shown by a gray value. However, eight different wavelength patterns are illustrated. f is frequency, amp is amplitude, and velocity is 1 pixel / frame. With the same frequency pattern, 1 pixel /
The patterns in which only the frame is advanced are shown side by side.
wavelength is the wavelength. In order to estimate the optimal regularization coefficient, for each image feature amount of the input pattern,
There is no choice but to search for all regularization coefficients within a certain range. H
Since the objective function in the Orn & Schunkk (H & S) method depends on the first derivative of time and space with respect to the gray value, the objective function can be seen only by the gray values of several adjacent points when viewed as a difference method. It turns out that it depends. That is, the input known pattern (FIG. 7 (1)) may be generated with various sine waves in the time and space directions. If the movement of the known pattern is also known, when the regularization coefficient is changed, the coefficient closest to the correct movement can be regarded as the optimum coefficient. To evaluate the similarity of velocity vectors, the inner product of the correct vector and the estimated vector may be taken. FIG. 7B shows the result of the inner product of the correct vector and the estimated vector when the pixel is shifted by 1 pixel per frame, for each of various regularization coefficients. The curves are for sinusoids with different image features. That is,
When a sine wave is handled, the basic image feature amount can be aggregated into three values: amplitude, wave number, and velocity. Amplitude is the variation in gray value,
It is sufficient to calculate the standard deviation of the gray value within a fixed image size, and the frequency component of the wave number can be easily obtained by Fourier transform. The speed is known.

FIG. 8 shows an example of the vortex velocity estimation result according to the present invention. The velocity can be calculated by the solution method described in FIG. Two frames of typhoon pattern were used. When the H & S method is applied using the optimal regularization coefficient, the vortex structure (Fig.
(1)) can be clearly detected. Here, the velocity vector could be estimated counterclockwise around the center of the vortex. Furthermore, for the velocity component, a moving average filter (equation (1
9)) is applied, the velocity field is generated and interpolated even in the area where the pattern does not exist (FIG. 8 (2)).

[0048]

[Equation 20] This is because when the pattern is predicted, the pattern does not advance to the region where the velocity is zero, and thus it is necessary to give a non-zero velocity.

FIG. 9 shows an example of fluid velocity field reconstruction and pressure estimation by the Navier-Stokes equation. There are various possibilities other than the moving average filter to form the velocity field required for pattern prediction. The moving average filter is good in that low-frequency components of the velocity field propagate, but it does not include fluid properties. Therefore, the Navier-Stokes equation (equation (20)) widely applied in the field of fluid dynamics is applied.

[0050]

[Equation 21] This equation has two main variables, velocity and pressure (pressure).
When the velocity and the atmospheric pressure are estimated with the velocity field being the initial value 1, the results shown in the figure are obtained (FIG. 9 (2)). In the case of a typhoon pattern, it can be estimated that a high atmospheric pressure (white) exists immediately after the movement and a low pressure area exists at the center of the vortex.

Here, a method of solving the Navier-Stokes (NS) equation will be shown.

In the text, the NS equation is described in HSM
It is solved by a method called AC method. In this way, N
NS is used to solve nonlinear simultaneous equations iteratively while adjusting the S and continuous equations with respect to pressure (or atmospheric pressure).
The equation contains two independent variables, velocity and pressure. Assuming that the target of the flow is an incompressible fluid, zero is a condition for the velocity field in the discretized continuous equation (equation (21)). On the calculation grid point, one pressure variable is arranged for each mesh, and for the velocity variable, its vertical and horizontal components are arranged on the grid point.

[0053]

[Equation 22] Further, it is assumed that the external force is not here to the precipitation pattern (equation (22)).

[0054]

[Equation 23] Next, in order to obtain the velocity field of the future, the forward difference is made with respect to time (Equation (23)).

[0055]

[Equation 24] In this specification, the velocity component estimated by the optical flow is given as an initial value when the NS equation is solved, and the pressure is estimated as zero in all regions. The boundary condition is
The continuous condition is imposed on the image contour. The viscosity coefficient and density were empirically determined.

When the pressure gradient is considered for each mesh, the direction and magnitude of the velocity are determined along the gradient. A solution is adopted in which the pressure is adjusted so that the equation (21) is satisfied for each mesh. Formulas (24) and (25)
Using, the velocity component and the pressure in small amounts for each mesh,
Repeat the calculation for all meshes. Conditional expression (21) continues to be less than a fixed minute value.

[0057]

[Equation 25] The converged result can predict the velocity component and the pressure from one discrete time to the next time step. The time integration may be repeated a predetermined number of times.

FIG. 10 shows an example of velocity field correction by singular value decomposition. It is an example of another vortex pattern (FIG. 10).
(1)). The estimated velocity field (FIG. 10 (2)) partially contains noise, and the velocity field is disturbed. Therefore, singular value decomposition is applied. Although this method is one of the information compression techniques, the two-dimensional velocity field is directly subjected to singular value decomposition to detect the principal component of velocity (FIG. 10 (3)). The velocity fields before (FIG. 10 (2)) and after (FIG. 10 (3)) are respectively enlarged and shown (FIGS. 10 (4) and (5)). Clearly, the noise component is found to be relaxed. Future velocity fields are highly correlated with noise-free principal components. For singular value decomposition, as shown in equation (26),
Original velocity field matrix it

[0059]

[Outside 7] If the cumulative value of singular values is 80% or more, the number of dimensions is approximated by KN. SVD (Singular Value Dec
In omposition: singular value decomposition), objective indicators can be set for noise and the amount of important information.

[0060]

[Outside 8] Are singular values, upper triangular and lower triangular matrix, respectively (reference:
Numerical Recipes in C: Technical Review Company).

[0061]

[Equation 26] The main equations required to predict the image pattern are listed below. Expression (27) is a unified set of five basic spatiotemporal patterns as a partial differential equation system. The image gray value I is a function of position and time. If Equation (27) is transformed into a form in which the change in the gray value of the next frame (time) can be calculated by the forward difference method for the time term, Equation (2)
8). The STE on the right side of Expression (28) is 3 here.
There is one, and is shown in formula (29). From the top, advection, anisotropic diffusion,
It is the difference (well, stagnation, suction). Each of the three is given by equations (30)-(33). For the velocity term in the advection term, the results of equations (37)-(41) are used. The respective mathematical expressions included in Expression (28) are Expression (29) to Expression (36). Expression (29) is an advection equation, an anisotropic diffusion equation, and a state equation. Equations (30) and (31) are anisotropic diffusion equations. The number (33) is the advection equation.
Here, the third-order approximation accuracy is shown. Expression (3
6) is a state equation based on difference images from consecutive frames. In order to identify the state, the difference value is statistically approximated by a Gaussian distribution (equation (34)). At that time, the threshold value is given by Expression (35).

[0062]

[Equation 27] Formula (37) is a formulation based on the standard regularization theory based on the H & S method, and the velocity can be estimated in pixel units. At this time, ω is a regularization coefficient, and an optimum value should be given for each pixel. The equation (38) is a moving average filter, and the equations (39), (40), and (41) are the Navier-Stokes equation and the continuous equation. It is applied to estimate fluid velocity and pressure fields.

[0063]

[Equation 28] The number (42) is an equation for calculating the optimum regularization coefficient, and the right side is the wave number k and the standard deviation SD of the gray value, and here,
This is a case where the image size is 32 × 32 pixels. The image is divided into sub-blocks, and each block is given an optimum regularization coefficient necessary for the calculation of the H & S method.

Equation (43) is a singular value decomposition. The matrix A on the left side is decomposed into a singular value W, an upper triangular matrix U and a lower triangular matrix V, and the matrix A can be approximated only by the number of terms KN. Here, the two-dimensional velocity field is approximated by the matrix A.

[0065]

[Equation 29] FIG. 11 shows the result of a statistical evaluation experiment of prediction accuracy. It is a result when the prediction accuracy is evaluated for the precipitation pattern by six methods. For the prediction pattern, equation (16) may be solved in a time evolution manner. The variable I includes the gray value of the image pattern in pixel units. At times n and n + 1, two continuous image patterns correspond to the initial input image.

Here, the method according to the present invention is called the DT method. For the prediction, the images of the present and one past are used. Horizontal axis 3
Up to the time ahead, every 10 minutes, the vertical axis is the rainfall error, which is the value per pixel. The worst is the conventional method, which is a combination of the cross correlation method (CC) and the linear extrapolation method. Next is the continuous forecast. For comparison, when the velocity field is subjected to another conventional method, that is, the NAGEL method and the ANANDAN method, in order to show the effect of the optimum regularization coefficient in H & S, there is not much difference from the conventional method. Regarding the method of predicting the velocity field, we will further compare the Navier-Stokes (NS) equation with and without singular value decomposition. The best accuracy is when all effects are included, followed by when the NS equation and singular value decomposition are not applied.
The error was improved by about 40% after 3 hours as compared with the conventional method.

FIG. 12 is an example of a predicted image in a general scene. It suffices to solve the equation (16) in a time evolution manner. The variable I includes the gray value of the image pattern in pixel units. At times n and n + 1, two continuous image patterns correspond to the initial input image.

A case where a general scene is predicted using the DT method is shown. The image size is 180 × 120 pixels (1, 2) and 300 × 300 pixels (3).

All actual images are home video images. The predicted image (future image) is generated by the present invention using only two continuous images input at the video rate. What is different from the weather pattern is the sampling rate and the image size. For each of the three types of scenes, from the left are the error of the gray value between the real image, the predicted image and the velocity field, and the real image and the predicted image. From top to bottom, smoke is scattered, trees are swayed by the wind, and zoomed in. The prediction result of 5 frames ahead is shown. There is almost no prediction error in any of the scenes. The effectiveness of local modeling such as precipitation patterns and general scenes was confirmed. Even with the conventional method, in a scene having such non-linear and non-stationary motion, a predicted image cannot be generated very well even with prediction one frame ahead.

FIG. 13 illustrates a method of predicting the respective changes using time-series imaged patterns observed from three radars of satellite, precipitation and lightning strike. In each case, the data has a size of 200 × 200 pixels and 10 minutes. 1 pixel 8-bit gradation. The satellite patterns are visible, infrared, water vapor, precipitation is observed with respect to precipitation amount and precipitation area, and lightning strikes are observed with respect to lightning strike position, number of lightning strikes, and lightning intensity, and all can be treated as a time-series two-dimensional pattern. The prediction method is based on the method described in FIG. 12 for each pattern. For the difference image information, two consecutive image patterns, one at the start of prediction and one before, are used.

The combination of each pattern and its effect are as follows. 1. Satellite radar visible and infrared, precipitation radar, lightning strike radar: clear, cloudy, and rain can be seen at one point in the image (Fig. 13
(1)). Satellite radar uses visible patterns during the day and infrared patterns during the night. It is used as information that can distinguish whether the visibility is cloudy or cloudy. On the other hand, since there is no sunlight at night, it becomes dark, so an infrared pattern is used instead. Visible and infrared may be relatively similar. In addition to whether there was a lightning strike at one location, how much rainfall occurred, whether the cloud was dense or whether it was thin, it is possible to predict the effective living information by integrating the information comprehensively (forecast ) Leads to. 2. Precipitation radar, lightning strike radar: Since lightning strike information is basically an aggregate of point sequences, when detecting the moving speed of lightning strike information from an image, it is difficult to find corresponding points between frames, so there is a high correlation. The moving speed of the precipitation pattern is substituted (FIG. 13 (2)). Moreover, the region where the intensity of the precipitation pattern is high (the amount of precipitation is large), the number of lightning strikes, and the lightning current are almost in the same region. It is suitable for so-called thundercloud prediction. 3. Water vapor / precipitation radar of satellite radar: Concentrated heavy rainfall can be understood from the precipitation pattern, but the sign can be predicted only after the rain becomes strong to some extent. Therefore, using a steam pattern, it is predicted that a heavy rain will occur in the same area within a few hours after a large amount of steam has passed through the area (FIG. 13 (3)). When predicting the precipitation pattern, it is easy to predict the water vapor pattern in the development term in the DT method, memorize the region and time where the amount was large, and control so that the precipitation amount increases in the development term. Can be realized.

In addition to the above-described video scene prediction method realized by dedicated hardware, a program for realizing the function is recorded in a computer-readable recording medium, and this recording medium is recorded in the recording medium. Load the recorded program into the computer system,
It may be executed. A computer-readable recording medium is a floppy (registered trademark) disk,
A recording medium such as a magneto-optical disk or a CD-ROM, or a storage device such as a hard disk device built in a computer system. Further, the computer-readable recording medium dynamically holds the program for a short time (transmission medium or transmission wave) such as when transmitting the program via the Internet, and serves as a server in that case. It also includes a volatile memory that holds a program for a certain period of time, such as a volatile memory inside a computer system.

Further, the present invention is not limited to the weather prediction, but can be applied to the prediction of a general image in which a moving object is present in the video.

[0074]

As described above, according to the present invention,
It has the following effects. Even if it is a complicated video scene, it is possible to generate a prediction scene without depending on the target, because it is locally approximated by five basic spatiotemporal patterns (diffusion, advection, difference image (decay, development, stagnation)). it can. Even if the change of the object in the scene is non-linear and non-stationary, the application target is selected, and each spatiotemporal pattern changes in the time and space directions,
Since it is integrated by one partial differential equation, it is possible to make a prediction considering various changes in the neighborhood, and as a result, the prediction error can be greatly reduced. Further, since the optimal regularization coefficient in the speed and the difference image are changed at every scene prediction, the parameter of the change of the future image can be reliable.

[Brief description of drawings]

FIG. 1 is a configuration diagram of a video scene prediction device according to an embodiment of the present invention.

FIG. 2 is a diagram showing five basic spatiotemporal pattern models.

FIG. 3 is a diagram showing instability of a prediction pattern due to a difference in order of difference approximation.

FIG. 4 is a diagram showing a difference between an isotropic diffusion method and an anisotropic diffusion method.

FIG. 5 is a diagram showing a difference between three states of a difference image.

FIG. 6 is a diagram showing a difference in estimated velocity fields when the regularization coefficient is constant.

FIG. 7 is a diagram showing a sine wave pattern for estimating an optimum regularization coefficient and an estimation example.

FIG. 8 is a diagram showing an example of an estimation result of a vortex velocity according to the present invention.

FIG. 9 is a diagram showing an example of fluid velocity field reconstruction and atmospheric pressure estimation by Navier-Stokes equations.

FIG. 10 is a diagram showing an example of velocity field correction by singular value decomposition.

FIG. 11 is a diagram showing a result of a statistical evaluation experiment of prediction accuracy.

FIG. 12 is a diagram showing an example of a predicted image in a general scene.

FIG. 13 is a diagram illustrating a method of predicting respective changes using time-series imaged patterns observed from three radars of a satellite, precipitation, and lightning strike.

FIG. 14 is a diagram showing a conventional prediction method for a weather radar image.

[Explanation of symbols]

1 Image input section 2 Image storage section 3 parameter estimation unit 4 Prediction calculator 5 Display

Claims (14)

[Claims] 1. A method of predicting changes in imaged patterns obtained from various observation sources, the method comprising inputting time-series image information, and storing the input image information. The input time-series image is approximated by five patterns of development, decline, and stagnation obtained by classifying diffusion, advection, and difference images of time-series images at different times, and the parameters required for the advection term For a velocity component that is, using only the change in the pattern of the current and past consecutive images, a parameter estimation step for estimating based on the optical flow, a prediction calculation step for repeating the prediction calculation until a required prediction time, and a prediction A video scene prediction method having a display step of displaying an image. 2. The pattern change using a partial differential equation in which the main variable including an advection term and a diffusion term is grayscale value information for each pixel of an image is predicted in the prediction calculation step. the method of. 3. A function having a differential term in the advection term is first-ordered or second-ordered at a discrete lattice point on a computer by a difference method.
The method according to claim 2, wherein the order is approximated to any one of the next order and the third order. 4. The method according to claim 2, wherein an isotropic diffusion method or an anisotropic diffusion method is applied to the diffusion term. 5. The method according to claim 2, wherein the partial differential equation is solved by discretization approximation by a difference method. 6. The method according to claim 1, wherein, in the prediction calculation step, prediction is performed while integrating highly correlated information among image information input from a plurality of different input sources. . 7. The method according to claim 1, wherein in the predicting calculation step, the velocity is predicted by using Navier-Stokes equation and / or moving average filter.
The method described in the section. 8. The method according to claim 1, wherein the image information is weather information. 9. An image pattern of a precipitation radar and a lightning strike radar using information of three radars of a satellite radar, a precipitation radar and a lightning strike radar. The satellite radar uses an image pattern visible in the daytime and an image pattern of infrared rays at the nighttime. 9. The method according to claim 8, wherein a predictive calculation is performed from the image pattern of 1., and these are integrated to predict fine, cloudy, rain, or lightning strike at one point in the image. 10. A precipitation pattern obtained from each radar using information from two radars, a precipitation radar and a lightning strike radar,
The method according to claim 8, wherein a predictive calculation is performed from the lightning strike pattern, and when detecting the moving speed of the lightning strike pattern, the moving speed of the precipitation pattern is substituted to predict the thundercloud at one point in the image. 11. A prediction calculation is performed from a water vapor pattern obtained from the satellite radar by using information of two radars, a satellite radar and a precipitation radar, and then a prediction calculation is performed from a precipitation pattern obtained from the precipitation radar. The method according to claim 8, wherein a heavy rainfall at one point in the image is predicted. 12. An apparatus for predicting changes in imaged patterns obtained from various observation sources, comprising image input means for inputting time-series image information and images for accumulating the input image information. Necessary for the advection term by approximating the accumulating means and the input time-series image with the diffusion, advection and five patterns of development, decline and stagnation obtained by classifying difference images of time-series images at different times. For the velocity component which is the parameter, the parameter estimation means for estimating based on the optical flow using only the current and past consecutive images, the prediction calculation means for repeating the prediction calculation until the required prediction time, and the predicted image A video scene prediction device having display means for displaying. 13. A video scene prediction program for causing a computer to execute the video scene prediction method according to any one of claims 1 to 11. 14. A recording medium on which the video scene prediction program according to claim 13 is recorded.