JP2007156954A - Reliability table creation method, optical flow estimation method, reliability table creation device, optical flow estimation device, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve reliability of calculation on optical flow. <P>SOLUTION: A video selection process 12 for inputting selection of two images among a video, a flow constraint reliability evaluation value calculation process 15 for evaluating an expected degree of reliability in calculation of the optical flow with consideration given to minute movement of respective points in the two image selected by the video selection process 12, a flow probability estimation process 16 for calculating a probability of transformation into a moving vector on each of vectors possibly becoming flow vectors in the respective points of the two images selected and inputted in the video selection process 12, and a reliability table generation process 17 for converting outputs from the flow constraint reliability evaluation value calculation process 15 and the flow probability estimation process 16 into a table for each of two sets of the respective points of the images, are carried out. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to optical flow estimation.

  For a long time, optical flow calculation methods have been known as means for detecting motion in a video, and efforts have been made to improve the accuracy by various devices.

  A typical example is a method of calculating the flow and using the surrounding information at a point where the reliability is considered to be low (see, for example, Patent Document 1).

  There is also a method of removing a flow considered to be unreliable as an outlier assuming continuity of movement (see, for example, Patent Document 2).

  Both methods are based on the premise that the flow can be calculated accurately. All are calculated after one velocity vector is assigned at each point, which depends largely on the initial state of the result. Moreover, it is necessary to make adjustments depending on the calculation method of each optical flow, and the system is complicated. In the optical flow constraint equation used, it is assumed that the brightness of the object is constant, and in the general environment, when illumination variation occurs, the accuracy decreases.

In addition, in the calculation of optical flow, noise is often generated and the result is not stable. In the first place, it was difficult to eliminate the influence of noise because processing was performed including the point that it was difficult to detect movement.
Japanese Patent No. 3435084 Japanese Patent No. 3408117

  In order to increase the reliability in the calculation of the optical flow, it was necessary to devise a method for increasing the accuracy for each method. In other words, it is necessary to repeat the process of giving reliability to the obtained flow and calculating the flow candidates again. As described above, only the method of improving accuracy is known only in a form depending on each method.

  In addition, since the “constant brightness” is assumed especially in the constraint formula of the optical flow, an error due to illumination fluctuation occurs in an actual problem, and various settings are required for each method.

  In addition, in general, in order to estimate the flow from the optical flow constraint formula, it is necessary to add other information (for example, continuity), so it is necessary to set parameters according to each situation. It was complicated.

  Accordingly, in order to solve the above-described problem, the invention according to claim 1 is a method of extracting information on motion by processing a video, wherein the video selection means inputs selection of two images in the video. When the image selection step and the flow constraint reliability evaluation value calculation means calculate the optical flow assuming a minute movement at each point of the two images selected in the image selection step, Each of the flow constraint type reliability evaluation value calculation step for evaluating whether or not the reliability is obtained, and the flow probability estimation means may be a flow vector at each point of the two input images selected in the video selection step. A flow probability estimating step for calculating a probability that the vector can be a movement vector for each vector, and a reliability table creating means for each two sets of each point of the image. A reliability table creation step of tabulating the output of the low-constraining reliability evaluation value calculation step and the flow probability estimating step, characterized by having a.

  Further, in the invention according to claim 2, in the flow constraint reliability evaluation value calculation step, a feature extraction process is performed on each of the two images selected in the video selection step, and similarities between these results are obtained. The degree is defined and used.

  The flow probability estimation step may include a local image feature amount calculation step in which a local image feature amount calculation unit performs a local image feature amount calculation based on a local calculation of an image. A change feature calculation step in which the change feature calculation means performs a change feature calculation based on a local temporal change of the image, and a probability estimation means uses a vector for each point from the local image feature amount calculation and the change feature result. And a probability estimating step for estimating the probability that the vector is a correct flow for each.

  The invention according to claim 4 is a method of extracting a motion for each point by processing an image, inputting the reliability table obtained by any one of claims 1 to 3, and integrating processing means Is the flow constraint reliability evaluation value calculation result stored in the reliability table in that the same movement can be assumed among the values of the flow probability estimation obtained as a local calculation result in the reliability table. Accordingly, an integration processing step of calculating a weighted product or an average to obtain a quasi-local flow probability estimation result, and a flow estimation processing means includes a flow estimation step of estimating an optical flow from the integration result. And

  According to a fifth aspect of the present invention, there is provided an apparatus for processing a video and extracting information relating to movement, wherein the video selection means for inputting selection of two images in the video, and the selection by the video selection means. A flow constraint type reliability evaluation value calculation means for evaluating how much reliability is obtained when calculating an optical flow assuming a minute movement at each point of the two images obtained, and the video selection A flow probability estimating means for calculating a probability that the vector can be a movement vector for each vector that can be a flow vector at each point of the two input images selected by the means, and two for each point of the image And a reliability table creating unit that tabulates the outputs of the flow constraint reliability evaluation value calculation unit and the flow probability estimation unit.

  Further, in the invention according to claim 6, the flow constraint reliability evaluation value calculation means performs a feature extraction process on each of the two images selected by the video selection means, and the similarities between these results. The degree is defined and used.

  According to a seventh aspect of the present invention, the flow probability estimation means includes a local image feature quantity calculation means for performing a local image feature quantity calculation based on a local calculation of an image, and a local temporal change of the image. Change feature calculation means for performing a change feature calculation based on, and probability estimation means for estimating the probability that the vector is a correct answer flow for each point and each vector from the local image feature amount calculation and the change feature result It is characterized by that.

  The invention according to claim 8 is an apparatus for processing an image and extracting a motion for each point, and inputs the reliability table obtained according to any of claims 5 to 7, and the reliability is obtained. The weighted product or the weighted product according to the flow constraint reliability evaluation value calculation result accumulated in the reliability table in that the same movement can be assumed among the values of the flow probability estimation values obtained as local calculation results in the table. It has an integration processing means for obtaining a quasi-local flow probability estimation result by calculating an average, and a flow estimation means for estimating an optical flow from the integration result.

  The invention according to claim 9 is characterized in that the method according to any one of claims 1 to 4 is written in a computer program so as to be executable.

  According to the first to ninth aspects of the present invention, whether the flow can be calculated independently of the optical flow calculation method, or whether the flow is satisfied for all velocity vector candidates, is a continuous numerical value, In addition, a scale of correlation coefficient and probability can be assigned, and as a result, basic information for realizing improvement in accuracy of flow calculation can be obtained.

  A method for obtaining a reliability table including an optical flow constraint reliability in an image and a probability distribution estimation method for an optical flow will be described with reference to FIG.

  In FIG. 1, 11 is an image sequence, 12 is a video selection process, 13 is an input image 1, 14 is an input image 2, 15 is a flow constraint reliability evaluation calculation process, 16 is a flow probability estimation process, and 17 is a reliability table. A creation process 18 is a reliability table.

  The image sequence 11 is an image sequence that is a source of processing. It may be an accumulated video, an already accumulated video, or a plurality of images that are independent one by one.

  The video selection process 12 selects two input images 13 and 14 to be processed from the image sequence 11. The video is I (x, y, t), and one image is expressed for each t. One point of the image has a pixel value (x, y). Therefore, it is considered as a two-dimensional function with (x, y) as a variable.

At this time, the two images to be selected are t = t ₀ and t = t ₀ +1. Basically, the two obtained images are images of the same number of pixels that are taken with the same image and with a short time difference. In the following processing, the image is a monochrome grayscale image, but if it is a color image, it may be converted to monochrome here.

  The flow constraint type reliability evaluation calculation process 15 is a flow constraint type reliability evaluation value calculation process for calculating how much reliability is obtained in the optical flow calculation process at each point of two images. Specifically, optical flow constraint formula for each point of two images

Is calculated as an evaluation value. here,

Is the optical flow (velocity vector),

Etc. are partial differentials with respect to I. Further, (x, y) or (x, y) is omitted unless necessary.

  Examples of reliability calculation formulas are collectively shown in the description of FIG. As shown in FIG. 2, the possibility of a flow vector at each point of the two input images is expressed by a vector field. This calculation formula is also summarized below as in the flow constraint type reliability evaluation calculation processing 15.

  Reference numeral 17 denotes a reliability table creation process, and reference numeral 18 denotes a result, which is a reliability table including the optical flow constraint reliability in the video and the probability distribution estimation result of the optical flow.

  FIG. 2 shows an embodiment of the flow probability estimation process 16 by a specific method. In FIG. 2, 21 is a local image feature amount calculation process, 22 is a change feature calculation process, and 23 is a probability estimation process.

  A specific method of the flow constraint reliability evaluation calculation process 15 and the flow probability estimation process 16 will be described with reference to FIG.

  First, the partial differentiation of the optical flow constraint equation 1 in the x and y directions is

Is obtained. here

And so on indicate the second derivative with respect to (x, y) among the variables of I.

Is the Hessian of function f. Time change is not included.
In this case, Equation 4 is

Can be rewritten. If the determinant of the Hessian of I is not zero,

Is obtained. (Document Uras (A computational approach to motion perception, S. Uras, F. Girosi, Averri, V. Torre, Biol. Cybern., Vol60, pp. 79-97, 1988)). This constraint equation is known to be resistant to illumination fluctuations (Brox (Brox, T., Bruhn, A., Papenberg, N., Weickert, j.
High Accuracy Optical Flow Estimate Based on a Theory of Warping,
ECCV 2004, 25-36 (Volume 3024 of Lecture Notes in Computer Science)).

  Based on the above preparation, in the flow constraint type reliability evaluation value calculation 13,

Is calculated as the flow constraint reliability evaluation value. Here, σ is a value of zero or more indicating the amount of noise assumed in advance.

  K is a parameter for calculating the reliability by weighting the amount of noise, and is a positive constant. For example, 1 is used. As a result,

And k is small or

Is large,

Is the correlation value after gradient calculation of two input images

It has a value close to. Therefore,

Can be regarded as an approximate value of a correlation coefficient in consideration of the presence of noise.

  On the other hand, k is large and

Is small,

Becomes smaller. This means that there are few image features in this respect and any result can be obtained due to the effects of noise, and the resulting flow is unreliable. Therefore, this value

Is larger (closer to 1), it is determined that the reliability of the flow constraint type is higher.

  Next, an example of a specific calculation method for the flow probability estimation 16 is shown.

  For example, in the simplest case, Hess (I) is changed to the local image feature amount 21 or

Is a change feature 22 over time. And in the probability estimation means 23,

Calculate Here, σ is the same value as in Equation 9.

  In this method, each point (x, y)

What is necessary is just to apply Formula 19 every time.

  In this step, each point (x, y), each vector

Each can be calculated independently and there is no repetitive operation.

  Fix (x, y)

Given this field, a probability density distribution is obtained. The degree of spread of this distribution is determined by the characteristic of Hess (I). For example, the distribution is concentrated in a narrow range at characteristic points, extended in a long range at edges, and spread in a two-dimensional shape at flat points without features.

  In fact, if this method is implemented programmatically, vector candidates

Since is a continuously periodic value, it is impossible to calculate this for all.

  Accordingly, when it is desired to obtain the distribution, the vector candidates are sampled with fineness as necessary, and the equation 19 is calculated. In some cases, Equation 19 may be calculated only for necessary vector candidates. For example, when the flow is estimated by another optical flow calculation method, the probability of the estimated vector may be confirmed. That is, it is a natural method to calculate probabilities only for necessary vectors.

  FIG. 3 shows the above process in a specific image example. In FIG. 3, 31 is the first image of the input image, 32 is a part of the first image of the input image, 33 is a part of the second image of the input image and is cut out from the same position as 32, and 34 is a flow An example of a constraint-type reliability evaluation process result, and 35 is an example of a flow probability estimation process result.

  The first image 31 of the input image is a grayscale image that was taken while moving.

  The first part 32 of the input image and the second part 33 of the input image are obtained by cutting a part of the image from the same position. Originally 20 × 20 pixels, this is reduced by ¼ and shown as 5 × 5 pixels. It can be seen that the same pattern has moved to the left.

  The flow constraint type reliability evaluation processing result example 34 is a result of calculating the reliability using Equation 9 for 5 × 5 points.

  The white color indicates higher reliability and the black color indicates lower reliability. In this example, the lower left has low reliability. In fact, it can be seen that even if the original two images are compared, there is a significant change.

  The flow probability estimation processing result example 35 is a result of performing the flow probability estimation using Equation 19 at the same point (25 points in total) as the flow constraint reliability evaluation processing result example 34.

  Each square represents a movement of ± 4 dots by a size vector candidate in units of 0.25 dots. The center point means no movement. In these 25 examples, the distribution varies greatly according to the characteristics of each point. However, in any point, the characteristics of the relatively high reliability are common among the leftward flow candidates, which is consistent with the fact that the pattern moves to the left.

  In the reliability table, the flow constraint type reliability evaluation process result and the flow probability estimation process result are accumulated together.

  Next, an optical flow estimation method in a video will be described with reference to FIG. In FIG. 4, reference numeral 41 denotes a reliability table, which is an output of the reliability table 18. 42 is an integration process, and 43 is a flow estimation process.

  In the integration process 42, the integration result is obtained for each point and each vector by using the data of the reliability table 41 regarding the points where the same movement can be assumed. In concrete terms, for example,

Etc. are considered. Where w _i is

Assume that a coordinate shift (x _i , y _i ) is defined in conjunction with the coefficient i. The range of (x _i , y _i ) limits the range in which the same motion can be assumed, and w _i is a weight. There is a method of setting this weight in proportion to the flow constraint reliability evaluation value. In addition, a method is also conceivable in which threshold processing is performed on the flow constraint reliability evaluation value and only data having a high evaluation value is used.

More specific examples of (x _i , y _i ) and w _i will be shown. Most simply, (x _i , y _i ) is set to y _i = −1.0,1, x _i = −1.0,1, and w _i = 1/9. This is a case where the same motion is estimated in the eight neighboring regions and the weights are the same at all points. For the same neighborhood, the weight may be proportional to Equation 19.

  As another example of the integration process 42, for example,

It is also possible to think about.

  The flow estimation process 43 selects an optical flow that is considered to have the highest probability from the result of the integration process 42 described above. The selected (u, v) is output as an optical flow at each point. This can be expressed in terms of equations 24 and 25 respectively.

Can be expressed as Equation 29 is

Is equivalent to In this case, there is no influence of the value of σ.

  The above definition has been described using only one point (x, y). Actually, the integral (or sum) is calculated in the vicinity of Hess (I (t)), ▽ I,

The same argument can be made even if the integration value of the region is used in the calculation of. This method is expected to improve the stability of the results.

  In practice, some pre-processing may be performed on the image in practice. For example, smoothing or logarithmic transformation can be considered. The smoothing process can be expected to remove the influence of high frequency components. After removing the high frequency components, it is possible to simultaneously reduce the image. Further, by adding logarithmic conversion, the influence of illumination variation can be further reduced.

(About device configuration)
The configuration of the reliability calculation device and the flow estimation device that perform each of the above processes will be described.

  The reliability calculation device includes an input unit that inputs an input image, a flow constraint reliability evaluation calculation process, a flow probability estimation process, a recording unit that records mathematical formulas necessary for a reliability table creation process, and an output unit that outputs a reliability table And a control unit that performs a flow constraint reliability evaluation calculation process, a flow probability estimation process, and a reliability table creation process.

  For example, when a control unit is realized using a CPU, a memory, a display as an output unit, a keyboard as an input unit and an image input device, and a hard disk as a storage unit are connected to the CPU. The hard disk stores a program for executing flow constraint reliability evaluation calculation processing, flow probability estimation processing, and reliability table creation processing, a mathematical expression, and an operating system. The program can be installed from a CD-ROM via a CD-ROM drive.

  The flow estimation apparatus includes an input unit that inputs a reliability table, an integration process, a recording unit that records mathematical expressions necessary for the flow estimation process, an output unit that outputs a flow, and a control unit that performs the integration process and the flow estimation process. Can be configured.

  For example, when the control unit is realized using a CPU, a memory, a display as an output unit, a keyboard as an input unit, a reliability input unit, and a hard disk as a storage unit are connected to the CPU. The hard disk stores programs, mathematical formulas, and an operating system for executing integration processing and flow estimation processing. The program can be installed from a CD-ROM via a CD-ROM drive.

(Effect of the embodiment)
According to the above embodiment, a result of determining whether the flow constraint equation can be established for each point independently of the optical flow calculation algorithm is obtained, and further, independently of this, for all the vectors required at each point. By indicating the possibility by a probabilistic numerical value, it is possible to provide information that can improve the accuracy without depending on the initial state of the flow calculation result.

  This is because the various parameters necessary for optical flow calculation and the above results are completely independent, and parameter setting does not depend on the flow calculation method and is simple.

  Furthermore, even if it is not a feature point, such as an edge, for example, the speed cannot be determined uniquely, the range of the movement can be specified, and it has been considered that the number of features has been discarded or the reliability is low. Can also provide a method for extracting information.

  Further, since the basic equation includes a filtering process for reducing the influence of illumination fluctuation, a method can be provided in which the result is not affected even if the illumination fluctuation changes.

  Furthermore, this result is presumed to be the same motion, and the optical flow itself can be estimated by integrating the mutual information of the points when the constraint equation is sufficiently reliable. This method has features including the above-described illumination fluctuation stability.

  Since there is no process of optimizing parameters by iterative calculation and calculation is possible directly from local information, it can also be applied to parallelization.

Explanatory drawing of reliability table acquisition. Explanatory drawing of a flow probability estimation process. Explanatory drawing of the Example of reliability table acquisition. Explanatory drawing of the optical flow estimation method.

Explanation of symbols

11 Image sequence 12 Video selection process 13 Input image 1
14 Input image 2
15 Flow constraint type reliability evaluation calculation processing 16 Flow probability estimation processing 17 Reliability table creation processing 18 Reliability table 21 Local image feature calculation processing 22 Change feature calculation processing 23 Probability estimation processing 31 First image of input image 32 Input image Part of the first image 33 Part of the second input image 34 Example of result of flow constraint reliability evaluation process 35 Example of result of flow probability estimation process 41 Reliability table 42 Integration process 43 Flow estimation process

Claims (9)

A method of processing video and extracting information about movement,
A video selection step in which the video selection means inputs the selection of two images in the video;
How much reliability can be obtained when the flow constraint reliability evaluation value calculation means calculates an optical flow assuming a minute motion at each point of the two images selected in the video selection step A flow constraint reliability evaluation value calculation step for evaluating
A flow probability estimating step for calculating a probability that the vector can be a movement vector for each vector that can be a flow vector at each point of the two input images selected in the video selection step;
A reliability table creating means for creating a table of outputs of the flow constraint reliability evaluation value calculation step and the flow probability estimation step for each set of two points of each point of the image; and A method of creating a reliability table, comprising: The flow constraint type reliability evaluation value calculation step includes:
The reliability table creation method according to claim 1, wherein a feature extraction process is performed on each of the two images selected in the video selection step, and the similarity between these results is defined and used. The flow probability estimation step includes:
A local image feature amount calculating means for performing a local image feature amount calculation based on the local calculation of the image;
A change feature calculation step in which the change feature calculation means performs a change feature calculation based on a local temporal change of the image;
The probability estimating means includes a probability estimating step for estimating a probability that the vector is a correct flow for each point and for each vector from the local image feature amount calculation result and the change feature result. Item 3. The reliability table creation method according to Item 1 or 2. A method of processing video and extracting motion for each point,
Input the reliability table obtained by any one of claims 1 to 3;
The flow processing reliability evaluation value accumulated in the reliability table in that the integrated processing means can assume the same movement among the flow probability estimation values obtained as local calculation results in the reliability table. An integrated processing step for obtaining a quasi-local flow probability estimation result by calculating a weighted product or an average according to the calculation result;
A flow estimation processing unit comprising: a flow estimation step of estimating an optical flow from the integration result. A device that processes video and extracts information about movement,
Video selection means for inputting selection of two images in the video;
Flow constraint reliability evaluation value calculation for evaluating how much reliability can be obtained when calculating an optical flow assuming a minute movement at each point of two images selected by the video selection means Means,
Flow probability estimation means for calculating a probability that a vector can be a movement vector for each vector that can be a flow vector at each point of the two input images selected by the video selection means;
A reliability table creating unit that tabulates the output of the flow constraint reliability evaluation value calculation unit and the flow probability estimation unit for each two sets of each point of the image. Degree table creation device. The flow constraint type reliability evaluation value calculating means is:
6. The reliability table creation apparatus according to claim 5, wherein a feature extraction process is performed on each of the two images selected by the video selection means, and the similarity between these results is defined and used. The flow probability estimation means includes:
A local image feature amount calculation means for calculating a local image feature amount based on a local calculation of an image;
Change feature calculation means for performing change feature calculation based on local temporal changes of the image;
The probability estimation means for estimating the probability that the vector is a correct flow for each point and each vector from the local image feature value calculation result and the change feature result, respectively, according to claim 5 or 6 Described reliability table creation device. A device that processes video and extracts motion for each point,
Input the reliability table obtained according to any one of claims 5 to 7,
According to the flow constraint reliability evaluation value calculation result accumulated in the reliability table in that the same movement can be assumed among the values of the flow probability estimation obtained as the local calculation result in the reliability table. An integrated processing means for obtaining a quasi-local flow probability estimation result by calculating a weighted product or an average;
An optical flow estimation apparatus comprising: flow estimation means for estimating an optical flow from the integration result.   A program characterized in that the method according to any one of claims 1 to 4 is written in a computer program and can be executed.