CN104157006A - Multi-scale differential optical flow computation method based on matching and oriented smoothness constraint optimization - Google Patents

Abstract

The invention discloses a high-precision optical flow field computation method for calculating the moving speed of picture frames. Based on the multi-scale Laplacian image pyramid, the flow speed is roughly estimated from large-scale gray scale information in an SSD (Sum-of-Squared Difference) matching method and corrected in small-scale gray scale information. Smoothness constraint optimization is used to spread and optimize the estimated flow speed, the flow speed in the direction vertical to the gradient direction fluctuates as slightly as possible, and excessive smoothness in positions where shielding and movement are not continuous is avoided.

Description

The multi-scale differential optical flow computation method based on coupling and oriented smoothness constraint optimization

Technical field

The present invention is relevant with computer vision and image understanding, along with the widespread use of video flowing, needs the accurately movement velocity of computed image interframe, the present invention relates to a kind of accurate optical flow field method of computed image interframe movement speed.

Background technology

Optical flow field is the projection on image plane of the 3D velocity field of visible point in scene, it has not only comprised the movable information of observed object, and carrying the abundant information of relevant scenery three-dimensional structure, therefore, at aspects such as the extraction of moving object detection, robot navigation, autokinesis information and three-dimensional structure recoveries, there is important application, but, from image sequence, calculate accurate optical flow field is a very difficult problem always, the differential method is typical the Computation of Optical Flow, but, in practical application, obtain accurate diff normally very difficult.

Formerly method [1] is (referring to P. Anandan, A computation framework and an algorithm for the measurement of visual motion Int J. Comp. Vision, 1988,2:283-310), Anandan has proposed the Computational frame that matching algorithm and even smoothness constraint are combined, and from the bent principal curvature of a surface of SSD, introduced the concrete letter of putting and measured, there is certain novelty;

Formerly method [2] is (referring to A. Singh. An Estimation-Theoretic Framework for Image Flow Computation. Pro. 3rd Intern. conf. Comp. Vision, Osaka, 1990,168-177) calculating of light stream is divided into based on the conservative initial estimation of information of image and the refinement process of neighborhood information, and the eigenwert of the covariance variance matrix of the average velocity that SSD curved surface is calculated is measured as the letter of putting of light stream, also has certain representativeness.

The present invention shows by research, the method that letter measurement is put in formerly method [2] calculating more can reflect the degree of reliability of optical flow computation than the method for Anandan, but, because iteration restrains, the refinement that utilizes neighborhood information is unnecessary, and formerly method [1] utilizes even smoothness constraint to propagate initial optical flow field and the thought optimized is outstanding, still, evenly smoothly can cause the excess smoothness to border and the discontinuous place of motion.

Thus, the present invention sets up the multiple dimensioned optical flow algorithm based on SSD coupling and oriented smoothness constraint optimization.This method is positioned at the thought of edge and angle point, model Laplce multi-Scale Pyramid from the important information of image; In the image layer of large scale, adopt SSD matching algorithm to obtain the initial estimation of optical flow field, and initial estimation is optimized and is propagated by oriented smoothness constraint at this layer, and be mapped to the image layer of small scale, it is revised.

Summary of the invention

The present invention sets up the optical flow approach of the large movement velocity of a kind of computed image interframe, model multiresolution analysis framework, and this framework hypograph is decomposed into the image pyramid of different resolution; Secondly, under multiresolution analysis frame system, set up differential algorithm; The 3rd, anisotropy diffusion is applied to light stream in the relaxative iteration of different resolution calculates, alleviate the error propagation effects while processing larger motion in even multi-grid algorithm.

Ultimate principle of the present invention is as follows:

One, the matching algorithm based on SSD, supposes that two adjacent two field pictures are respectively with , and the time interval of establishing between adjacent image be the unit interval, the displacement of interframe is speed, sets up the pixel on the first two field picture around size is neighborhood, be associated window, on the second two field picture around pixel set up region of search, search for window, utilize squared difference and criterion, can obtain search window scope about speed error distribution function, by formula (1), described

(1)

Formula (1) is minimized, can obtain the first two field picture pixel best pixel coupling on the second frame ; whole pixel speed during for optimum matching; Foundation is around optimal match point symmetrical SSD error distribution function

(2)

This error distribution function can be regarded as around the discrete Conicoid fitting at whole pixel optimum matching place, therefore can be by the method for its matching being tried to achieve to the minimum value of curved surface, thus near the velocity shifts of the sub-pixel precision whole pixel optimum matching obtained , utilize like this matching algorithm to obtain the optimum matching of sub-pixel precision, its side-play amount is

, (3)

The error function of formula (2) can be converted into all possible speeds response distribute

(4)

, be normalized parameter, and now, be in a way around the weighted mean of asking at SSD curved surface, thereby, within the scope of the SSD surface coordinates of formula (2), covariance matrix be

(5)

For the light stream that will calculate , the present invention makes with error minimize

(6)

Two, oriented smoothness constraint is propagated and is optimized, and equation (6) has two known variables, thereby needs additional constraint condition.On the other hand, the optical flow field being calculated by matching algorithm is rough and the non-overall situation, need to be propagated into neighborhood and optimization, and oriented smoothness constraint requires the square upward velocity altering a great deal in vertical ash value variation should be as much as possible little, can effectively suppress the excess smoothness to the discontinuous place that moves, minimize oriented smoothness constraint and be expressed as by formula (7)

(7)

gradient operator, represent that velocity is with respect to a partial derivative matrix of image coordinate; it is the weight matrix of oriented smoothness constraint, weight matrix can make smoothness constraint very little or do not have in vicissitudinous direction and propagate along variation of image grayscale, and suppress along the propagation of image ash value great changes have taken place direction, conventionally, the border that the region correspondence that alters a great deal of ash value imaging surface, can obtain the solving equation of optical flow field like this

(8a)

(8b)

(8c)

for controlling the weights of directional smoothing degree, it is former frame image , be unit matrix, power.

Technique effect of the present invention:

With technology [1 formerly, 2] difference is, the present invention is positioned at the thought of edge and angle point from the important information of image, model Laplce multi-Scale Pyramid, in the image layer of large scale, adopt SSD matching algorithm to obtain the initial estimation of optical flow field, and initial estimation is optimized and is propagated by oriented smoothness constraint at this layer, and be mapped to the image layer of small scale, it is revised.Oriented smoothness constraint can effectively suppress the excess smoothness at edge and the discontinuous place of motion.

Accompanying drawing explanation:

Fig. 1 is the pyramid structure of the multiple dimensioned decomposition that provides of the present invention

Fig. 2 is that the present invention provides Yosemite sequence image, and left figure and right figure are respectively the 8th frame and the 9th frame

Fig. 3 is three layers of laplacian pyramid setting up corresponding to reference frame image that the present invention provides

Fig. 4 is standard light flow field between the Yosemite sequence consecutive frame that provides of the present invention

Fig. 5 is the optical flow field between the 8th and the 9th frame that calculates of the present invention

Fig. 6 is the Taxi sequence image that the present invention provides: left figure and right figure are respectively the 15th, 16 frames

Fig. 7 is the optical flow field between Taxi sequence the 15th frame and the 16th that calculates of the present invention.

Embodiment:

Present embodiment is specifically introduced in conjunction with Fig. 1-7 couple the present invention:

(1) take multiresolution analysis as basis, set up the multiple dimensioned framework of image, in the multiple dimensioned framework of image, high resolving power correspondence little yardstick, and low resolution correspondence large yardstick, and total presents pyramid, shown in Figure 1, wherein, original image has highest resolution and minimum yardstick, in pyramid structure in the 0th layer.Obviously, increase along with yardstick, the movement velocity of image will reduce pro rata, under little movement velocity, the priori conditions of differential optical flow algorithm can be better met, and therefore, first by thick resolution, motion is estimated, then improve gradually resolution, further motion calculated.

(2) a kind of matching algorithm based on SSD. suppose that two adjacent two field pictures are respectively with , and the time interval of establishing between adjacent image be the unit interval, the displacement of interframe is speed.Set up the pixel on the first two field picture around size is neighborhood, be associated window.On the second two field picture around pixel set up region of search, search for window.Utilize squared difference and criterion, can obtain search window scope about speed error distribution function, by formula (1), described

(1)

Formula (1) is minimized, can obtain the first two field picture pixel best pixel coupling on the second frame ; whole pixel speed during for optimum matching; Foundation is around optimal match point symmetrical SSD error distribution function

(2)

This error distribution function can be regarded as around the discrete Conicoid fitting at whole pixel optimum matching place, therefore can be by the method for its matching being tried to achieve to the minimum value of curved surface, thus near the velocity shifts of the sub-pixel precision whole pixel optimum matching obtained , utilize like this matching algorithm to obtain the optimum matching of sub-pixel precision, its side-play amount is

, (3)

The error function of formula (2) can be converted into all possible speeds response distribute

(4)

, be normalized parameter.And now, be in a way around the weighted mean of asking at SSD curved surface, thereby, within the scope of the SSD surface coordinates of formula (2), covariance matrix be

(5)

For the light stream that will calculate , the present invention makes with error minimize

(6)

(3) oriented smoothness constraint is propagated and is optimized, and equation (6) has two known variables, thereby needs additional constraint condition.On the other hand, the optical flow field being calculated by matching algorithm is rough and the non-overall situation, need to be propagated into neighborhood and optimization.Oriented smoothness constraint requires the square upward velocity altering a great deal in vertical ash value variation should be as much as possible little, can effectively suppress the excess smoothness to the discontinuous place that moves, minimize oriented smoothness constraint and be expressed as by formula (7)

(7)

gradient operator, represent that velocity is with respect to a partial derivative matrix of image coordinate, it is the weight matrix of oriented smoothness constraint, weight matrix can make smoothness constraint very little or do not have in vicissitudinous direction and propagate along variation of image grayscale, and suppress along the propagation of image ash value great changes have taken place direction, conventionally, the border that the region correspondence that alters a great deal of ash value imaging surface, can obtain the solving equation of optical flow field like this

(8a)

(8b)

(8c)

for controlling the weights of directional smoothing degree, it is former frame image , be unit matrix, power.

(4) optical flow equation solves. the equation to (8) formula, must adopt the variational method, and just can solve two components of velocity.At weight matrix in, ignore the secondary partial derivative of image function, above-mentioned equation can be written as

(9)

， (10)

New weight matrix can be written as

(11)

In formula, .By being integrated second of item in (11) formula substitution equation (9) formula, obtain

(12)

If the contrary of covariance matrix is , and , hereinafter, we use represent, now, according to (9) formula and (12) formula, by the variational method, obtain two Eulerian equations

(13a)

(13b)

Finally obtain the solution of relaxative iteration form

(14a)

(14b)

Subscript represent iterations, the contrary determinant of covariance matrix, wherein

(15a)

(15b)

, to be respectively speed , average.And variable with be respectively

(16a)

(16b)

(16c)

(16d)

(5) the present invention adopts the synthetic image sequence Yosemite sequence of lineup's work, and one group of real image sequence Taxi sequence is tested, to artificial synthetic image sequence, the standard light flow field between consecutive frame is known, thereby can obtain quantitative assessment, to real image sequence, the light stream figure that can calculate with other optical flow algorithm carries out contrast and analysis, obtaining assessment qualitatively. the left figure in Fig. 2 and right figure are respectively the 8th and the 9th frames of artificial composograph Yosemite sequence, image size is 256 * 256, the cloud of top with the speed of 2 pixel/frame to right-hand translation, the below motion left of the mountain valley of below, some is dispersed at the volley, velocity magnitude is about 4-5 pixel/frame, this image sequence is due to the mutual coverage that has comprised different velocity ranges and mountain valley and sky, the algorithm of light stream is had to certain challenge, Fig. 3 is three layers of laplacian pyramid setting up corresponding to reference frame image, the optical flow field of the standard that Fig. 4 is known, and Fig. 5 is the optical flow field that the present invention calculates, from the present invention and method [1] formerly and formerly the error ratio of method [2] can find out, flow velocity average angle error of the present invention and standard angle deviation have certain reduction, wherein the improvement of standard angle deviation is obvious, owing to having comprised different motions in image, so the accurate Calculation of the flow velocity at moving boundaries place is very difficult.Optical flow field with standard is compared, the optical flow field that the present invention calculates calculate better the background sky of translation motion and the mountain of dispersing between flow relocity calculation, to the flow relocity calculation of background sky medium cloud, be also very accurately simultaneously, embody diffusion and the effect of optimization of oriented smoothness constraint; In Fig. 6, left figure and right figure are respectively the 15th and the 16th frames of familiar true picture Taxi sequence, have 3 obviously targets of motion in street scene, mark in the drawings, and target 1 street corner of just turning right wherein, its movement velocity is about 1.0 pixel/frame; Target 2 is moved from right to left, and speed is about 3.0 pixel/frame; Target 3 is moved from left to right, and speed is about 3.0 pixel/frame.Because the optical flow field of standard is unavailable, thereby can only assess intuitively, Fig. 7 is the optical flow field that the present invention calculates when iterations is 30 times.

Claims (6)

1. the matching algorithm based on SSD, supposes that two adjacent two field pictures are respectively with , and the time interval of establishing between adjacent image be the unit interval, the displacement of interframe is speed, sets up the pixel on the first two field picture around size is neighborhood, be associated window, on the second two field picture around pixel set up region of search, search for window, utilize squared difference and criterion, can obtain search window scope about speed error distribution function, by formula (1), described

(1)

Formula (1) is minimized, can obtain the first two field picture pixel best pixel coupling on the second frame ; whole pixel speed during for optimum matching.

2. set up around optimal match point symmetrical SSD error distribution function

(2)

This error distribution function can be regarded as around the discrete Conicoid fitting at whole pixel optimum matching place, therefore can be by the method for its matching being tried to achieve to the minimum value of curved surface, thus near the velocity shifts of the sub-pixel precision whole pixel optimum matching obtained .

3. utilize like this matching algorithm to obtain the optimum matching of sub-pixel precision, its side-play amount is

, (3)

The error function of formula (2) can be converted into all possible speeds response distribute

(4)

, be normalized parameter, and now, be in a way around the weighted mean of asking at SSD curved surface, thereby, within the scope of the SSD surface coordinates of formula (2), covariance matrix be

(5)

For the light stream that will calculate , the present invention makes with error minimize

(6)

Can find out, this method has distinct physical significance.

4. oriented smoothness constraint is propagated and is optimized, and equation (6) has two known variables, thereby needs additional constraint condition.

5. on the other hand, the optical flow field being calculated by matching algorithm is rough and the non-overall situation, need to be propagated into neighborhood and optimization, and oriented smoothness constraint requires the square upward velocity altering a great deal in vertical ash value variation should be as much as possible little, can effectively suppress the excess smoothness to the discontinuous place that moves, minimize oriented smoothness constraint and be expressed as by formula (7)

(7)

gradient operator, represent that velocity is with respect to a partial derivative matrix of image coordinate; be the weight matrix of oriented smoothness constraint, weight matrix can make smoothness constraint very little or do not have in vicissitudinous direction and propagate along variation of image grayscale, and suppresses along the propagation of image ash value great changes have taken place direction.

6. the border common, the region correspondence that alters a great deal of ash value imaging surface, can obtain the solving equation of optical flow field like this

(8a)

(8b)

(8c)

for controlling the weights of directional smoothing degree, it is former frame image , be unit matrix, power.