JP2861278B2 - Moving vector detecting method and apparatus - Google Patents

Description

Description: BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and an apparatus for obtaining a motion vector of an object being photographed from a temporally continuous image (for example, a television image). It is.

[Conventional technology]

Utilizing a relationship that is established between the spatial and temporal differential values in the moving image and the moving vector on the image of the object captured in the moving image, called a gradient method for detecting the moving vector of the object. (Takahiko Fuukiki, "Measurement of moving amount and speed of moving object by image signal", IEICE Technical Report, IE78-67, 1978-10, and Masahiro Wada, Hirohisa Yamaguchi, "Iterative gradient method" Detection of Motion Amount of Moving Image Signals by IEICE ", IEICE Transactions D, 1985-4, pp.663-670).
Next, the gradient method will be described.

When the pixels of the moving image are expressed as a function of the coordinates x, y of the screen and the time t, the expression showing the relationship between the differential values of the image on which the gradient method is based is as follows.

E _x u + E _y v + E _t y0 (1) where E _x , E _y , and E _t are partial derivatives of the image brightness in the x, y, and t directions, respectively, and u, v are the motion vectors at that point.
x and y components. Although the relationship represented by the equation (1) is given independently at each point on the image, u and v cannot be determined only by the relationship at one point. Therefore, as shown in FIG. S is set, and the motion vector u, v in S is solved assuming that it is constant. This concept is one model for detecting a movement vector, and can be arranged as a movement vector detection model as follows.

Model for Detecting Motion Vector The relationship between the partial differential coefficient represented by the equation (1) and the motion vector is established everywhere in the region S.

-The movement vector is constant in the small area S on the image.

This model is interpreted mathematically as follows. E _x , E _y , E _{t in} equation (1) can be seen as a function defined on the region S, and the assertion of this model is that the partial derivative E _t calculated from the image is it is to be represented by a linear sum of E _x and E _y are calculated from the same images on S.

E _t (x, y) = − uE _x (x, y) −vE _y (x, y) (2) However, each partial derivative actually calculated from the image is completely satisfied by the conditions described above. does not hold, that is, by this model does not apply to the full, can not be decomposed as shown in equation (2), there is residual E _r is a portion that can not be represented as a linear combination of E _x and E _y.

E _t (x, y) = − uE _x (x, y) −vE _y (x, y) + E _r (x, y)
(3) Here, these relationships will be described using an analogy of the function space and the vector space, as follows. FIG. 3 is a diagram in which the functions _Ex , _Ey , _Et , and _Er on the small area S are each represented as a vector in a three-dimensional space. What is meant by Equation (2) is that the vector E _t rides on a plane spanned vector E _x, the E _y, equation (3) is completely the plane by noise or model is not complete Means that it is necessary to add the vector _Er without riding on.

Now, the fitting model shown in the result detected from the actual moving image, the size of the E _r of formula (3) | E _r |
U and v should be selected so as to minimize. This idea of minimizing | E _r | is the basis for applying the model.

Model fitting criterion • Minimize the magnitude | E _r | of the residual _Er , u, v determined in this way, ie, the movement vector to be detected by u, v that gives the minimum value of | E _r | (U ₀ , v ₀ )
give.

In order to determine the movement vector using the fitting criterion with the model described above, the function _Er should be selected so as to be orthogonal to the functions E _x and E _y , as is clear from the analogy with the vector.

(E _r · E _x ) = 0, (E _r · E _y ) = 0 (4) where (f ₁ · f ₂ ) indicates the inner product of the functions f ₁ and f ₂ .

(F ₁ · f ₂ ) = ∫∫ _s f ₁ (x, y) · f ₂ (x, y) dxdy (5) Assuming that the movement vector to be obtained is (u ₀ , v ₀ ), equation (3)
And make the inner product of E _x and E _y, the following equation is obtained when applying the condition of Equation (4).

Where a = ∫∫ _s E _x (x, y) ² dxdy b = y _s E _y (x, y) ² dxdy f = ∫∫ _s E _y (x, y) E _t (x, y) dxdy _{g = ∫∫ s E t (x} , y) E x (x, y) dxdy h = ∫∫ s E x (x, y) E y (x, y) dxdy Therefore, the movement vector to be obtained is as follows become.

[Problem to be Solved by the Invention] Although the movement vector is calculated by the method described above,
In an actually obtained image, the reliability of a movement vector calculated based on noise included in the image, a pattern of a captured object, and the like is affected. However, the conventional method has no method for estimating the reliability.

It is an object of the present invention to provide an indicator of the reliability of a motion vector, to use only reliable detection results, or even to propagate a highly reliable part to a lower part, thereby obtaining a motion vector of the entire image. It is an object of the present invention to provide a moving vector detection method and apparatus which can accurately obtain the motion vector.

[Means for solving the problem]

The first invention, showing the relationship between the temporally continuous image E (x, y, t) spatial derivative E _x a, E _y and motion vector (u, v) on the time derivative E _t and the image The equation E _x .u + E _y .v + E _t = 0 and the condition that the movement vector (u, v) is constant within the region S set on the image are defined by each differential value calculated from the actual image. A method of using a movement vector (u ₀ , v ₀ ) that minimizes the sum | E _r | of the amount of the right side of the equation in the region S calculated using the movement vector of the region S on the image, the value a = ∫∫ _s E _x calculated from the spatial differential values ^{(x, y) 2 dxdy b} = ∫∫ s E y (x, y) 2 dxdy h = ∫∫ s E x (x, y) E y Quadratic form with (x, y) dxdy coefficients Or a value derived from this value is used as a value indicating the reliability of the detected motion vector in the angle θ direction.

According to the second invention, for each of a plurality of regions S _ij which are set on a temporally continuous image E (x, y, t) and extend in the space x, y directions, a first The motion vector (u _0ij , v _0ij ) is calculated by applying the motion vector detection method of the present invention,
An energy functional that represents a constraint that the distribution of the motion vector on the image is smooth and a constraint that the correction amount from the motion vector is small is set, and the motion vector is set so as to minimize the value of this functional. in motion vector detecting method for detecting the entire image of the movement vector field of the optical flow by modifying, that reflect the index R ² representing the reliability constraints that correction amount from the movement vector of the functional is small Features.

A third invention is a converter for converting a temporally continuous image sequence obtained by an image pickup device into a discrete representation, and a frame memory for holding temporally adjacent frames of the converted discrete representation image. If, spatial difference E _x of the image held in the frame memory, an image difference calculation device for calculating the E _y and frame difference E _t, wherein for each of a plurality of areas S _ij set on the image Sum of squares and products of each difference value a _ij = ΣE _x (x, y) ² b _ij = ΣE _y (x, y) ² h _ij = ΣE _x (x, y) E _y (x, y) f _ij = ΣE _y (x, y) E _t (x, y) g _ij = ΣE _t (x, y) E _x (x, y) and a motion vector using the sum And a motion vector calculation device for calculating a motion vector for each of the regions. Eigenvalues λ _1ij , λ _2ij and eigenvectors e _1ij , e _2ij corresponding to these eigenvalues are calculated, and the values of these eigenvalues and eigenvectors, or the values derived from these eigenvalues and eigenvectors, are calculated as the reliability of the detected moving vector. And a reliability index calculating device that outputs the index as an index representing

A fourth invention is a converter for converting a temporally continuous image sequence obtained by an image pickup device into a discrete representation, and a frame memory for holding temporally adjacent frames of the converted discrete representation image. If, spatial difference E _x of the image held in the frame memory, an image difference calculation device for calculating the E _y and frame difference E _t, wherein for each of a plurality of areas S _ij set on the image Sum of squares and products of each difference value a _ij = ΣE _x (x, y) ² b _ij = ΣE _y (x, y) ² h _ij = ΣE _x (x, y) E _y (x, y) f _ij = ΣE _y (x, y) E _t (x, y) g _ij = ΣE _t (x, y) E _x (x, y), and an initial motion vector using the sum And a motion vector calculation device for calculating the respective regions, wherein a simultaneous linear equation relating to u _ij and v _ij generated from the motion vector and the sum total is 0 = (a _ij + 4α) u _ij + hv _{ij -α (u i + 1 j} + u i-1 j + u i j + 1 + u i j-1) -a ij u 0ij -h ij v 0ij 0 = (b ij + 4α) v ij + hu ij -α (v _{i + 1 j + v i-} 1 j + v i j + 1 + v i j-1) -b ij v to calculate the solution of _0ij -h _ij u _0ij, comprising a moving vector correction unit for outputting a final motion vector It is characterized by the following.

[Action]

According to the above-described model for detecting a motion vector, the movement of an image pattern is observed by its time differential function _Et (x, y) and detected as its magnitude | _Et |. Accordingly, as the magnitude of | E _t | generated by the movement to a certain size is larger, the influence of noise or the like is not so large, and the movement of the image pattern is easily detected. That is, if the ratio of the magnitude of the movement vector to the magnitude of | E _t |
The reliability of the detected motion vector can be estimated based on the value of R.

Now, assuming that the pattern of the image has moved in the θ direction by δr, the movement vector (δu, δv) can be expressed as follows.

δu = δrcosθ δv = δrsinθ the magnitude of E _t which is observed by the mobile δ | E _t | that becomes as follows.

δ | E _t | = δ r | E _x cos θ + E _y sin θ | (9) R is defined as the following equation, and R ² is calculated as follows.

From equation (11), it can be seen that R ² is a quadratic form of cos θ and sin θ. This quadratic coefficient matrix depends only on the pattern of the image, and the change in R according to the moving direction θ of the pattern is expressed as a function of θ.

To obtain the maximum and minimum of R, the quadratic form of the equation (11) is converted into a principal axis as follows.

Here, λ ₁ and λ ₂ are eigenvalues of the coefficient matrix, and e ₁ and e ₂ are eigenvectors of magnitude 1 corresponding to λ ₁ and λ ₂ , respectively. Accordingly, if the chosen so that λ ₁ ≧ λ ^_2, R ₂ is θ takes a maximum value lambda ₁ when the direction of e _1, it is found to take lambda ₂ when e _2. Therefore, R is the same, and its value is the square root of each eigenvalue. To illustrate these relationships,
If the ease of detection of the movement is represented by a vector having a direction θ and a size R, the locus is elliptical as shown in FIG. 4, and the directions of the major axis and the minor axis are the above-described eigenvectors e ₁ and e ₁ , respectively.
In the direction of e ₂ , the length is the square root of the eigenvalues λ ₁ , λ ₂ . Therefore, the vector r ₁ as the index indicating the most reliable directional component and the degree of reliability of the motion vector calculated by the equation (8), and the vector r ₂ as the index indicating the least reliable directional component and the degree thereof. Is defined as follows using the eigenvalues and eigenvectors appearing in equation (12).

Furthermore, these eigenvalues and eigenvectors are given by equation (7)
Is calculated as follows using the value of

Here, depending on the application, a value representing this vector using polar coordinates can be used, and the eigenvalue and the eigenvector can be used as they are as an index indicating reliability.

Next, the motion vector calculated by the above-described method for each area S _ij on the image shown in FIG. 2 is corrected while taking into account the index representing the reliability of the motion vector,
A method for obtaining a movement vector of the entire image will be described.
The movement vector of the entire image is generally called an optical flow, and there is an example in which the property of the optical flow uses an assumption that the distribution of the movement vector is smooth (BERTHOLD KPHORN, BRIAN G. SCHUNCK: “Deter
mining Optical Flow ", Artificial Intelligence, volum
e 17, pages 185-203, 1981). Here, this assumption is adopted, and the individual motion vectors are determined by the deterministic relaxation method (Katsuhiko Sakagami, Naokazu Yokoya: “Relaxation method and regularization”, information processing, volume 30, No. 7, September 1989). The optical flow is corrected while considering the reliability.

In order to solve a problem by the relaxation method, an energy functional to be minimized must first be set. The energy functional Φ is represented by the sum of a functional Ψ representing a constraint on smoothness and Ω representing a constraint on a detected movement vector.

Φ (u (x, y), v (x, y)) = αΨ (u (x, y), v (x,
y)) + Ω (u (x, y), v (x, y)) (15) where α is a constant that balances the two constraints.

The functional Ψ representing the constraint of smoothness expresses the deviation from the nearby movement vector by the square of the gradient of each component u, v,
Assume that the integration is performed over the entire vector field.

Functional Ω representing constraint from detected motion vector
In consideration of the reliability of each motion vector, it is determined that correction in the direction of high reliability requires a large amount of energy, and conversely, energy in the direction of low reliability is low in energy.

Ω = ∫∫ (r ₁ Δv) ² + (r ₂ Δv) ² dxdy (17) where Δv = (u−u ₀ , v−v ₀ ) (18) where r ₁ and r ₂ is the reliability vector shown in equation (13), and u ₀ and v ₀ are the detected motion vectors shown in equation (8). Meaning of the energy of the formula (17), as shown in FIG. 5, is the r ₁ direction size weights of the components r ₁ of correction vectors Delta] v, similarly as r ₂ direction component r It is the square of the distance measured by multiplying the weight of ₂ .
By the way, the energy functional Ω is expressed by the relation of the equation (13) and the equation (1
Using the relationship used in the transformation from 1) to equation (12), the following transformation is possible.

The functional that represents the constraint from the movement vector has been set as described above. Here, the power | E _r | ² of the residual _Er of the time differential function in each region is calculated from Equation (3). , As follows:

| E _r | ² = a (u−u ₀ ) ² + 2h (u−u ₀ ) (v−v ₀ ) + b
(V−v ₀ ) ² + d (20) where d = au ₀ ² + 2hu ₀ v ₀ + bv ₀ ² + 2gu ₀ + 2fv ₀ + c (21) The term relating to u and v in equation (20) is expressed by equation (20). Equivalent to the inside of the integral of the last equation in 19). Therefore, the energy functional Ω represents the sum of the residual power of the time differential function, which is increased by correcting the moving vector, in all vector fields.

Now that the energy functional has been set, a moving vector field that minimizes it is determined. Therefore, when the Euler equation of the energy functional Φ represented by the equation (15) is obtained, the following is obtained.

a (u−u ₀ ) + h (v−v ₀ ) −α (u _xx + u _yy ) = 0 h (u−u ₀ ) + b (v−v ₀ ) −α (v _xx + v _yy ) = 0.・ (22) If this equation is discretized and the derivative is replaced with the difference, the following equation is obtained.

0 = (a _ij + 4α) u _ij + hv _ij (23) −α (u _{i + 1 j} + u _{i−1 j} + u _{i j + 1} + ui _j−1 ) −a _ij u _0ij −h _ij v _0ij 0 = _{(b ij + 4α) v ij} + hu ij -α (v i + 1 j + v i-1 j + v i j + 1 + v i j-1) -b ij v 0ij -h ij u 0ij here, i, j is This is a subscript of x, y walking representing a grid point of the movement vector field, and corresponds to a value calculated in an area _Sij on the image. The differential values of u and v are replaced with differences as follows.

_{u xx = u i + 1 j} -2u ij + u i-1 i (24) u yy = u i j + 1 -2u ij + u i j-1 v xx = u i + 1 j -2v ij + v i-1 _{i v yy = u i j +} 1 -2v ij + v i j-1 Now, moving vector field that gives the minimum value of the energy functional Φ solves a system of linear equations u, relates v represented by the formula (23) Given by. Various methods for solving this simultaneous linear equation by numerical calculation (Hayato Togawa:
"Numerical calculation of a matrix", Ohmsha, 1971). Here, assuming that the SOR method is used, a scheme based on the SOR method of Expression (23) is obtained as follows.

Here, the subscript k on the shoulder of u, v is the k of the repetition operation.
Indicates that the value is for the first step. Ξ is a relaxation acceleration coefficient which is a value of 0 or more and 2 or less, but generally a value of about 1.8 to 1.9 is used. The value of α is 2 when the input image is 8 bits and the size of the area S _ij is 16 × 16 pixels.
About 000 is appropriate. When calculating the movement vector field using equation (25), there is a portion where no pixel exists around the vector field, but the vector there is calculated as being the same as the outermost value of the vector field at that stage. . This is a condition at the open end according to the expression used in splines and the like.

〔Example〕

 FIG. 1 shows a first embodiment of the present invention.

The analog video signal obtained by the imaging device 1 is A /
The image is discretized by the D converter 2 and becomes a digital image. This digital signal is held in the frame memory 3. The frame memory 4 is a memory for holding a digital image of one frame before. When an image of a new frame is sent from the A / D converter 2 to the frame memory 3, the image stored in the frame memory 3 until then is stored. To frame memory 4
Holds.

The image difference computing unit 5, to calculate the spatial and temporal difference E _x, E _y, E _t with images of adjacent frames is holding two frame memories are. That is, assuming that the image held in the frame memory 3 is E (x, y, t ₀ ) and the image held in the frame memory 4 is E (x, y, t ₀ -1), the respective difference values are as follows. Calculated by formula.

E _x = E (x, y, t ₀ ) −E (x−1, y, t ₀ ) E _y = E (x, y, t ₀ ) −E (x, y−1, t ₀ ) E _t = E (x, y, t 0) -E (x, y, t 0 -1) ··· (26) where, E _x, for E _y by averaging the difference between the contents of the two frame memories Is also good.

The sum calculating device 6 calculates the sum of the square and the product of the difference values shown in Expression (7) using the respective difference values calculated by the image difference calculating device 5. Here, the range for calculating the sum is the area _Sij shown in FIG. 2, and the sum is calculated by the number of areas. As the size of the region _{Sij, when} the size of the entire image is about 640 × 400 pixels, about 4 × 4 pixels to about 16 × 16 pixels are often used.

The motion vector calculation device 7 uses this sum to obtain the equation (8).
Is performed, and a movement vector (u ₀ , v ₀ ) is output.

The reliability index calculating device 8 similarly calculates the expression (1
Calculate and output the index indicating the reliability shown in 3).

FIG. 6 shows a second embodiment of the present invention. Imaging device 1,
A / D converter 2, frame memory 3, frame memory 4,
The processing and signal flow of the image difference calculation device 5, the sum total calculation device 6, and the movement vector calculation device 7 are the same as those in the first embodiment described above. The output of the reliability index calculation device 8 is input to the movement vector correction device 9, which performs an iterative operation according to the schema shown in equation (25) to correct the movement vector, and converts the result into a final movement vector field (optical). Flow).

In each of the above embodiments, the image capturing apparatus 1 has been described as being included in the movement vector detection position. However, without including this, it is also possible to process the output of a video tape recorder or the like as an input.

〔The invention's effect〕

According to the present invention, it is possible to obtain an index indicating the reliability of a movement vector detected from a moving image. For example, in an application for detecting a moving object, a highly accurate movement vector can be obtained by using only a highly reliable movement vector. Object detection becomes possible.

Further, the present invention makes it possible to accurately detect a movement vector field (optical flow) of an image. For example, in a use of detecting a three-dimensional movement of a camera from this movement vector, it is possible to accurately detect the movement. Will be possible.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a diagram showing a first embodiment of the present invention, FIG. 2 is a diagram showing a small area set on a moving image, and FIG. FIG. 4 is a diagram showing the relationship between the derivative of the motion vector according to the direction of the reliability, FIG. 5 is a diagram showing the relationship between the vector representing the reliability and the correction vector, and FIG. It is a figure showing a 2nd example. DESCRIPTION OF SYMBOLS 1 ... Imaging device 2 ... A / D converter 3 ... Frame memory 4 ... Frame memory 5 ... Image difference calculation device 6 ... Summation calculation device 7 ... Moving vector calculation device 8 ... Reliability index calculation device 9 ... Moving vector correction device

Claims (4)

(57) [Claims] Spatial differentiation E _x of 1. A time-sequential image E (x, y, t) , equation representing the relationship between the movement vectors on E _y and the time derivative E _t and the image (u, v) and _{E x · u + E y ·} v + E t = 0, the said movement vector by setting the area S on the image (u, v) a condition that is constant, with the differential values calculated from the actual image A motion vector (u ₀ , v ₀ ) that minimizes the sum | E _r | of the quantity of the right side of the equation calculated in the area S in the area S as the motion vector of the area S on the image; differential value value is calculated from _{a = ∫∫ s E x (x} , y) 2 dxdy b = ∫∫ s E y (x, y) 2 dxdy h = ∫∫ s E x (x, y) E y ( x, y) quadratic form with dxdy coefficients Or a value derived from this value is used as a value indicating the reliability of the detected movement vector in the angle θ direction. 2. A plurality of regions S _ij extending on a space x, y direction set on a temporally continuous image E (x, y, t).
The motion vector (u _0ij , v _0ij ) is calculated by applying the motion vector detection method according to claim 1 to each of the following, and the constraint that the distribution of the motion vector on the image is smooth and the motion vector , An energy functional that represents a constraint that the correction amount is small is set, and the motion vector is corrected so as to minimize the value of this functional, thereby detecting a motion vector field (optical flow) of the entire image. in vector detecting method, motion vector detecting method characterized by reflecting the index R ² representing the reliability constraints that correction amount from the movement vector of the functional is small. 3. A converter for converting a temporally continuous image sequence obtained by an image pickup device into a discrete representation, and a frame memory for holding temporally adjacent frames of the converted discrete representation image. an image difference calculation device for calculating a spatial difference E _x, E _y and frame difference E _t of the image held in the frame memory, each for each of the plurality of areas S _ij set on the image Sum of squares and products of difference values a _ij = ΣE _x (x, y) ² b _ij = ΣE _y (x, y) ² h _ij = ΣE _x (x, y) E _y (x, y) f _ij = ΣE _y (x, y) E _t (x, y) g _ij = ΣE _t (x, y) E _x (x, y), and a motion vector using the sum And a motion vector calculation device for calculating a motion vector for each of the regions. Eigenvalues λ _1ij , λ _2ij and eigenvectors e _1ij , e _2ij corresponding to these eigenvalues are calculated, and the values of these eigenvalues and eigenvectors, or the values derived from these eigenvalues and eigenvectors, are calculated as the reliability of the detected moving vector. And a reliability index calculating device that outputs the index as an index representing the motion vector. 4. A converter for converting a temporally continuous image sequence obtained by an image pickup device into a discrete representation, and a frame memory for holding temporally adjacent frames of the converted discrete representation image. an image difference calculation device for calculating a spatial difference E _x, E _y and frame difference E _t of the image held in the frame memory, each for each of the plurality of areas S _ij set on the image Sum of squares and products of difference values a _ij = ΣE _x (x, y) ² b _ij = ΣE _y (x, y) ² h _ij = ΣE _x (x, y) E _y (x, y) f _ij = ΣE _y (x, y) E _t (x, y) g _ij = ΣE _t (x, y) E _x (x, y), and an initial movement vector using the sum And a motion vector calculation device for calculating the motion vector for each of the regions, wherein a simultaneous linear equation relating to u _ij and v _ij generated from the motion vector and the sum total is 0 = (a _ij + 4α) u _ij + hv _{ij -α (u i + 1 j} + u i-1 j + u i j + 1 + u i j-1) -a ij u 0ij -h ij v 0ij 0 = (b ij + 4α) v ij + hu ij -α (v _{i + 1 j + v i-} 1 j + v i j + 1 + v i j-1) -b ij v to calculate the solution of _0ij -h _ij u _0ij, comprising a moving vector correction unit for outputting a final motion vector A movement vector detection device, characterized in that: