JP2003256844A - Pattern estimating method, pattern estimating device, program for the same method and recording medium with its program recorded - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To separate and extract the speed of a traveling vehicle from a bad weather scene such as a snowfall, rainfall, and fog being the factor of the generation of the fluctuation of illumination, luminance, and contrast in an image. <P>SOLUTION: An image inputting part 1 inputs the time-series image information of a pattern. An image storing part 2 stores the time-series image information. A parameter estimating part 3 estimates a model parameter for detecting the speed of the pattern from the stored time-series frame image. An output part 4 displays an image or information related with the speed of the pattern extracted based on the parameter. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a pattern estimation method and apparatus for detecting an imaged (patterned) moving object obtained from a WEB or various observation sources and estimating the moving speed of the object.

[0002]

2. Description of the Related Art If this kind of pattern itself and speed can be detected with high accuracy, the traffic live camera can detect a stable vehicle speed from a live camera scene that has a bad view due to traffic flow and monitoring of the vehicle and snow and rain. By detecting the traffic accident, it becomes possible to prevent traffic accidents in advance, and it becomes possible to predict vehicle patterns and traffic flow from video of traffic congestion and traffic flow.

[0003] In the meteorological field, the moving speed of the rainy area is known, which leads to disaster prevention as weather forecast. Forecast of precipitation amount by precipitation radar image, if satellite image, cloudy and clear weather forecast from visible image or infrared image, lightning strike If the lightning strike forecast from the image integrates the precipitation and lightning strike information, it is possible to forecast the heavy rain and thundercloud.
For general images other than weather, it is possible to sense the wind direction and strength from a scene in which trees fluctuate, and to predict waves that predict changes in waves.

Furthermore, the center of gravity cannot be clearly defined, such as changes in blood flow patterns in medical images and prediction of cognitive processes.
Further, the present invention can be used in various fields such as predicting future video information in an image or video scene that is not easily segmented.

[0005]

Various methods have been proposed for estimating a speed from a video pattern including a complicated shape or pattern as described above by image processing. In the field of image coding, the motion vector estimation method by block matching is the mainstream, and in the field of computer vision, the optical flow method is the mainstream.

In the latter, there are methods based on the gradient method and the block matching method. The gradient method is Horn & Sch
unck shows how to obtain a solution by imposing a smooth constraint of the first derivative with respect to velocity in the framework of standard regularization theory. Various proposals have been made for restraint conditions, such as the second derivative of velocity. No matter which constraint is applied, the biggest issue is how to set the optimal constraint.
Moreover, it is necessary to locally change the constraint condition according to the image feature amount.

In the gradient method and the block matching method, when deriving the basic equation, it is assumed that the gray value between consecutive image frames is constant. Therefore, if the local illumination, brightness, contrast, shadow, etc. fluctuate, the assumption is deviated.

[0008] Brightness variations in the image also occur when snow, rain, fog, etc. cover the entire scene. When a traveling vehicle is detected in bad weather, the speed from snowfall and the speed from the traveling vehicle are mixed in a complicated manner, and it is difficult to easily separate and estimate each speed.

The object of the present invention is to determine the speed from snowfall due to bad weather scenes such as snowfall, rainfall, fog, etc., which cause illumination, brightness, contrast fluctuations, etc. in the image, and the traveling vehicle etc. of interest. A pattern estimation method, a pattern estimation device, a pattern estimation program, and a recording medium in which the program is recorded, in which the speed is separated and extracted.

[0010]

The present invention for solving the above-mentioned problems is characterized by the following method, device, program and recording medium.

(Invention of Method) (1) A method for estimating an imaged pattern obtained from a WEB or various observation sources, which comprises an image input step of inputting time-series image information of the pattern, and the time An image accumulation process of accumulating sequence image information, a parameter estimation process of estimating a model parameter for detecting a pattern velocity from the accumulated time-series frame images, and an image relating to the velocity of the pattern extracted based on the parameter And an output process for displaying information.

(2) In the parameter estimation process of (1) above, an objective function is set and at least unknown values of speed, brightness and contrast are estimated.

(3) The objective function of the above (2) is characterized in that an unknown number is estimated by minimizing the non-linear function via a non-linear robust function.

(4) In the above (3), one of the steepest descent method, the neural network, and the Levenberg-Matsukart method is used for the minimization operation.

(5) In the above (3), the dispersion value included in the Lorentz function or the like is changed stepwise from a large value to a small value in the minimization process.

(6) In the parameter estimation process of the above (1), the image is smoothed in advance by the anisotropic diffusion method while leaving the edges of the image.

(7) Using the result of the output process of (1) above, k-means clustering, LBG algorithm, etc. are applied to points with speed to count the number of objects having speed. Characterize.

(8) The above (7) is characterized in that it includes a step of merging according to the distance for each cluster formed by converging the initial number of clustering.

(Invention of Apparatus) (9) An apparatus for estimating an imaged pattern obtained from a WEB or various observation sources, comprising image input means for inputting time-series image information of the pattern, and the time Image accumulation means for accumulating sequence image information, parameter estimation means for estimating a model parameter for detecting a pattern speed from the accumulated time-series frame images, and an image relating to the speed of the pattern extracted based on the parameter And an output unit for displaying information, the parameter estimating unit sets an objective function, and includes a unit for estimating at least velocity, brightness, and an unknown number of contrasts, the objective function being a nonlinear robust function. , A means for estimating an unknown value by minimizing a non-linear function, and the minimization operation includes a steepest descent method and a neural network. Or the Levenberg-Matsukart method, wherein the minimization means includes means for varying the dispersion value included in the Lorentz function or the like in a stepwise manner from a large value to a small value. The estimating means includes means for generating a smoothed image while leaving the edges of the image by the anisotropic diffusion method in advance, and using the result of the output means, k- mea
n clustering, LBG algorithm, etc. are included to count the number of individuals of an object having velocity, and each cluster formed by converging the initial number of clustering is merged according to distance. It is characterized by

(Invention of Program) (10) A program of a method for estimating an imaged pattern obtained from a WEB or various observation sources,
An image input procedure for inputting time-series image information of the pattern, an image storage procedure for storing the time-series image information, and a parameter for estimating a model parameter for detecting a pattern speed from the stored time-series frame image It has an estimation procedure and an output procedure for displaying an image and information relating to the speed of the pattern extracted based on the parameters, and the parameter estimation procedure sets an objective function and at least the unknowns of the speed, the brightness, and the contrast. The objective function includes a step of estimating an unknown number by minimizing a nonlinear function through a nonlinear robust function, and the minimizing operation includes a steepest descent method, a neural network, and a Levenberg -Including any one procedure of the Matsukart method, and included in the Lorentz function etc. in the minimization procedure The dispersion value includes a step of changing from a large value to a small value stepwise, and in the parameter estimation procedure, a procedure of generating an image smoothed while leaving edges of the image by the anisotropic diffusion method in advance. Using the result of the output procedure, the method includes performing a k-mean clustering, an LBG algorithm, or the like with respect to a point having a velocity to count the number of individuals of an object having a velocity, and the initial number of the clustering. It is characterized by including a procedure of merging according to the distance for each cluster formed by converging.

(Invention of Recording Medium) (11) A recording medium recording a program of a method for estimating an imaged pattern obtained from a WEB or various observation sources, in which time-series image information of the pattern is input. Image input procedure, an image accumulation procedure for accumulating the time-series image information, a parameter estimation procedure for estimating a model parameter for detecting a pattern speed from the accumulated time-series frame images, and based on the parameters. And an output procedure for displaying an image or information relating to the speed of the extracted pattern, wherein the parameter estimation procedure includes a procedure of setting an objective function and estimating at least velocity, brightness, and contrast unknowns, and the objective function Via the nonlinear robust function, by minimizing the nonlinear function,
The procedure includes a procedure for estimating an unknown number, and the minimization operation includes any one procedure of a steepest descent method, a neural network, and a Levenberg-Matsukart method, and in the minimization procedure, a variance value included in a Lorenz function or the like. Includes a step of varying from a large value to a small value in a stepwise manner, and in the parameter estimation procedure, a step of previously generating a smoothed image while leaving an edge of the image by an anisotropic diffusion method is included. , Using the result of the output procedure,
With respect to a certain point of velocity, a procedure of performing k-mean clustering, LBG algorithm, or the like to count the number of individuals of an object having velocity is included, and a distance is set for each cluster formed by converging the initial number of the clustering. It is characterized by including a procedure of merging accordingly.

[0022]

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, the present invention will be described in detail based on illustrated embodiments.

FIG. 1 is a system configuration diagram showing the entire processing of the present invention. The image input unit 1 includes various sensors,
An image including the speed estimation target is acquired using the observation device and the WEB as the information source. The image storage unit 2 stores time-series images of the images acquired by the image input unit 1. The parameter estimation unit 3 estimates model parameters from the time series images stored in the image storage unit 2. The output unit 4 displays an image and information regarding the speed of the pattern. It should be noted that each of these units also corresponds to the processing procedure diagram of the pattern detection method of the present invention.

FIG. 2 shows a traffic scene due to snowfall or rainfall. In the figure, A is an image of a traffic road, B is an image of a vehicle on the traffic road, and C is an image of snow or rain. Although it is clear that this image frame is a moving image, if there is snow or rain falling on the line of sight of the camera, there is a problem that it will be a great obstacle in detecting the speed of the vehicle.

That is, in the speed estimation method according to the conventional method, snowfall and the speed of the vehicle are mixed and detected in most cases. This is because the detection model does not include each characteristic. The snowfall pattern is highly non-stationary, and it is difficult to apply knowledge within a certain range regarding its shape, size, and speed. Further, it is difficult to distinguish the white color from the white color of the vehicle.

The following equations (S1) to (S12) are a group of equations applied in the present invention. Various models are conceivable for detecting the velocity between consecutive image frames. In many cases, assuming that the observation time is minute, a model is applied in which the gray value is preserved even if the object or pattern in the image moves. Hereinafter, this is referred to as an image storage type.

[0027]

[Equation 1]

[0028]

[Equation 2]

[0029]

[Equation 3]

In this case, the expression (S1) is given.
It is assumed that a certain two-dimensional position x on the image moves at a velocity u in the next frame. The moving velocity u of the object, which is an unknown number, can be solved as a least squares problem that minimizes the objective function in a certain region. That is, the objective function (S5, S6) is linear, and the unknown value of velocity 2
It has more components and has more gray values for the object to be fitted. From this, it can be easily calculated through the general pseudo inverse matrix.

Depending on the image, it is necessary to consider locally the influence of brightness variation, illumination variation, contrast variation, etc. even within a minute time. That is, the gray value is not stored between frames as in the formula (S1).

Therefore, the contrast variation is considered as the first extension of the image storage formula. Equation (S2) shows a contrast variation model.

As a second extension, a model including brightness variation and contrast variation is shown. This is as shown in equation (S3).

As a third extension, a plurality of factors such as shadows that impair the image storability are modeled. This is shown in formula (S4).

In any case, a linear objective function (S5,
Since S6 and S7), the unknown two velocity components, the luminance component, and the contrast component can be estimated by solving the inverse matrix. However, in the formula (S6), the time term (t +
1) is Taylor-expanded and approximated to express the time difference term as shown in Expression (S7).

In the least-squares method, it is generally known that outliers of data in the objective function reduce the estimation accuracy of unknowns. Outliers are values with large variances in the data to be fitted. Depending on the magnitude of the variance value, weighting may be performed according to the degree of deviation from the average value, and the overall variance may be reduced and applied to alleviate the effect.

In this case, there are various weighting functions, but a nonlinear robust function is applied. The series of robust functions called M-estimation include the Lorentz function shown in equation (S9) and the Biweight method. The value of the objective function (S8) changes depending on the deviation from the fitted data, but since the value of the objective function is a fitting error, incorporating this function in the framework of M-estimation mitigates the effect from outliers. Leads to accurate estimation of unknowns.

The unknown can be estimated by minimizing the nonlinear objective function. Steepest descent method, neural network, Levenberg-Marquard
t) Method etc. are used. The partial derivative of the nonlinear robust function and the partial derivative of the unknown are obtained in advance (S1
0, S11). The dispersion value of the Lorentz function is variably controlled from large to small in the iterative process so that it does not fall into a local minimum value stably. In the iterative calculation, the steepest descent method (S12) may be used. However, the differential value must be empirically multiplied by a very small parameter.

FIG. 3 shows the result of the conventional velocity detection method for the image of FIG. In an image in which brightness and contrast vary between image frames, there is a scene in which the entire screen becomes difficult to see due to bad weather such as snowfall or rain. When the number of vehicles or the number of pedestrians is counted in a traffic scene or a street scene, or when the situation is image-processed through a camera, bad weather is an obstacle. As a pre-processing, it is not possible to remove the effect of snowfall that crosses the scene just by applying a low-pass filter, and there is a problem that even the vehicles and people of interest are blurred. If an attempt is made to calculate by the gradient method of NAGEL et al., A plurality of motion vectors will be detected simultaneously from snowfall, a vehicle, and a person. Therefore, separation and extraction from scenes of vehicles and people cannot be performed well.

FIG. 4 shows a traffic scene, a brightness image and a contrast image to which the present invention is applied. In the present invention, the brightness
An experiment is conducted to detect a vehicle traveling from a snow scene using a contrast model. No pretreatment with a low-pass filter is applied. This is because there was a problem that the snow and the car were fused by the low pass filter.
The traffic scene P1 is covered with snow, and a plurality of vehicles are traveling in the left lane. From this scene, a contrast image P2 and a luminance image P3 are detected by a nonlinear robust function which is a luminance / contrast model. In both scenes, the snowfall pattern was detected rather than the vehicle.

FIG. 5 shows the vehicle range and velocity vector detected by the present invention. The vehicle area | region detected by the same model as FIG. 4 and the velocity vector V1 of each vehicle are shown. Further, the traveling vehicle range V2 and the velocity vector V3 are shown. Although snowfall also has a movement vector, the brightness image P3 and the contrast image P
As shown in 2, the snowfall motion component was absorbed by each image. Therefore, only the velocity vector of the vehicle could be detected separately. That is, the situation of snowfall or rainfall can be estimated from the brightness image and the contrast image by associating with the clear scene or by taking the difference from the static scene by the distribution of the frequency image components.

FIG. 6 shows a point sequence having a moving component detected according to the present invention and a point sequence integrated in the merging algorithm.
In order to automatically count the number of vehicles, a pixel having a motion vector is set as one point, and a point having a speed equal to or higher than a certain threshold in the scene is detected and clustering is performed. Although there are various clustering methods, the k-mean clustering or the LBG algorithm is applied. It is assumed that the number of clusters and the number of vehicles are almost the same. The initial number of centroids is several times the expected number of vehicles. for that reason,
Centroids should be further merged to reduce the centroid number after the LBG algorithm has converged.
Merge within a certain distance. Monitoring the change in the number of merged centroids reveals the change in the number of vehicles on the road.

FIG. 7 shows a time series change in the number of vehicles. It can be seen that the number of vehicles increases and then decreases. This is consistent with what the scene looks like visually. Here 300
A snowfall scene from a frame live camera was used.

In the parameter estimation, it is preferable to generate an image which is smoothed while leaving the edges of the image in advance by the anisotropic diffusion method.

Further, according to the present invention, some or all of the processing functions of the method or apparatus shown in FIG. 1 are configured as a program and realized by using a computer, or the processing procedure shown in FIG. 1 is used as a program. It can be configured and run on a computer. A computer-readable recording medium, such as an FD (floppy disk: registered trademark), stores a program for realizing the processing function of each part of the computer or a program for causing the computer to execute the processing procedure.
It can be recorded in MO, ROM, memory card, CD, DVD, removable disk, etc., and can be saved or provided, and can be distributed via a communication network such as the Internet. It is possible.

[0046]

As described above, according to the present invention, in an outdoor scene covered with snow or rain, it is possible to separately detect a traveling vehicle and snow or rain, and to remotely monitor the scene situation. In particular, when collecting information from live cameras of various sites on the WEB, it is possible to continuously and stably detect the movement of a predetermined object and lead to scene understanding.

[Brief description of drawings]

FIG. 1 is a system configuration diagram showing the overall processing of the present invention.

[Fig. 2] Traffic scene due to snowfall and rainfall.

FIG. 3 is a result of a conventional speed detection method.

FIG. 4 is a traffic scene, a brightness image, and a contrast image used in the present invention.

FIG. 5 is a diagram of a vehicle range and a velocity vector detected by the present invention.

FIG. 6 is a sequence of points having a moving component detected by the present invention and a sequence of points integrated into a merging algorithm.

FIG. 7 is a diagram showing a time series change in the number of vehicles.

[Explanation of symbols]

1 ... Image input section 2 ... Image storage unit 3 ... Parameter estimation unit 4 ... Output section

Claims (11)

[Claims] 1. A method for estimating an imaged pattern obtained from a WEB or various observation sources, comprising an image input process of inputting time-series image information of the pattern, and accumulating the time-series image information. An image accumulation process, a parameter estimation process for estimating a model parameter for detecting a pattern velocity from the accumulated time-series frame images, and an output for displaying an image and information relating to the pattern velocity extracted based on the parameter A pattern estimation method comprising: 2. The pattern estimating method according to claim 1, wherein in the parameter estimating process, an objective function is set and at least unknown values of velocity, brightness and contrast are estimated. 3. The pattern estimation method according to claim 2, wherein the objective function is estimated by minimizing the non-linear function via a non-linear robust function. 4. The steepest descent method, neural network, Levenberg
A pattern estimation method characterized by using one of the Matsukart methods. 5. The pattern estimation method according to claim 3, wherein in the minimization process, the variance value included in the Lorentz function or the like is changed stepwise from a large value to a small value. 6. The pattern estimating method according to claim 1, wherein an image which has been smoothed while leaving an edge of the image is generated by an anisotropic diffusion method in advance. 7. Using the result of the output process of claim 1,
A pattern estimation method characterized by performing k-mean clustering, an LBG algorithm, or the like on a point having a velocity to count the number of individuals of an object having a velocity. 8. The pattern estimation method according to claim 7, further comprising a step of merging clusters formed by converging an initial number of clustering according to a distance. 9. An apparatus for estimating an imaged pattern obtained from a WEB or various observation sources, comprising image input means for inputting time-series image information of the pattern, and accumulating the time-series image information. Image storage means, parameter estimation means for estimating a model parameter for detecting pattern speed from the stored time-series frame images, and output for displaying an image and information relating to the speed of the pattern extracted based on the parameters And a means for estimating an unknown number of at least velocity, brightness, and contrast, wherein the parameter estimating means sets an objective function, and the objective function minimizes the nonlinear function via a nonlinear robust function. By including a means for estimating an unknown number, the minimization operation includes a steepest descent method, a neural network,
Including any one means of the Levenberg-Matsukart method, wherein in the minimization means, the variance value included in the Lorentz function or the like includes means for varying from a large value to a small value in a stepwise manner. In advance, an anisotropic diffusion method is included, which includes means for generating a smoothed image while leaving an edge of the image, and using the result of the output means, k-means clustering or A means for counting the individual number of objects having a velocity by applying the LBG algorithm or the like, and means for merging according to a distance for each cluster formed by converging the initial number of the clustering. And a pattern estimation device. 10. A program of a method for estimating an imaged pattern obtained from a WEB or various observation sources, the image input procedure for inputting time-series image information of the pattern, and the time-series image information An image storage procedure for storing, a parameter estimation procedure for estimating a model parameter for detecting the pattern speed from the stored time-series frame images, and an image and information relating to the speed of the pattern extracted based on the parameter are displayed. And an output procedure to do, the parameter estimation procedure includes a procedure of setting an objective function and estimating at least velocity, brightness, and an unknown number of contrast, wherein the objective function is a nonlinear function through a nonlinear robust function. The procedure includes a procedure for estimating unknowns by minimizing Door,
Including any one procedure of the Levenberg-Matsukart method, and in the minimization procedure, the variance value included in the Lorenz function or the like includes a step of varying from a large value to a small value in a stepwise manner. In advance, an anisotropic diffusion method is used to include a procedure for generating a smoothed image while leaving an edge of the image, and using the result of the output procedure, k-means clustering or The method includes performing a LBG algorithm or the like to count the number of individuals of an object having a velocity, and including a procedure of merging according to a distance for each cluster formed by converging the initial number of the clustering. The program of the pattern estimation method. 11. A recording medium for recording a program of a method for estimating an imaged pattern obtained from a WEB or various observation sources, which comprises an image input procedure for inputting time-series image information of the pattern, An image accumulation procedure for accumulating time-series image information, a parameter estimation procedure for estimating a model parameter for detecting a pattern speed from the accumulated time-series frame images, and a speed of a pattern extracted based on the parameter And an output procedure for displaying an image or information, wherein the parameter estimation procedure includes a procedure of setting an objective function, estimating at least velocity, brightness, and contrast unknowns, and the objective function is a nonlinear robust function. Via a non-linear function minimization to estimate the unknowns, the minimization operation includes Law, neural net,
Including any one procedure of the Levenberg-Matsukart method, and in the minimization procedure, the variance value included in the Lorenz function or the like includes a step of varying from a large value to a small value in a stepwise manner. In advance, an anisotropic diffusion method is used to include a procedure for generating a smoothed image while leaving an edge of the image, and using the result of the output procedure, k-means clustering or The method includes performing a LBG algorithm or the like to count the number of individuals of an object having a velocity, and including a procedure of merging according to a distance for each cluster formed by converging the initial number of the clustering. Recording medium.