JPH0440576A - Method and device for detecting moving vector - Google Patents

Abstract

PURPOSE:To find the moving vector of the whole image with high accuracy by using the value of a quadratic form having the value calculated from the spatially differentiated value as its coefficients as the value representing reliabil ity in the direction of angle for detected moving vector. CONSTITUTION:The moving vector (u,v) which minimizes the sum ¦Er¦ or resid ual of equation calculated when equation Ex.v+Ey.v+Et=0 representing the relation of the spatial derivatives Ex, Ey and the time derivative Et of an image E (x,y.t) continued in point of time and the moving vector (u,v) on the image, and a condition that the moving vector (u,v) is kept constant in an area S set on the image are applied to each differentiated value calculated from an actual image is set as the moving vector of the area S on the image. Then, the value of quadratic form represented in equation II in which a value represented in equation I calculated from the spatial differentiated value is set as the coefficient is used as the value representing the reliability in the direction of angle theta of the detected moving vector. In such a way, it is possible to detect a mobile body with high accuracy.

Description

DETAILED DESCRIPTION OF THE INVENTION [Field of Industrial Application] The present invention relates to a method for determining the movement distance of an object being photographed from temporally continuous images (for example, television images), and It is related to the device.

[Conventional technology]

This method is called the gradient method, which detects the movement vector of an object by using the relationship between the spatial and temporal differential values in the moving image and the movement vector of the object on the image. (Takahiko Fukinuki, "Measurement of moving amount and speed of moving objects using image signals", Institute of Electronics and Communication Engineers Technical Report, IE78-67, 1978-10,
and Masahiro Wada and Hirohisa Yamaro, “Detection of the amount of motion in moving image signals using the iterative gradient method,” Transactions of the Institute of Electronics and Communication Engineers, 1985.
-4. pp, 663-670).

Next, the gradient method will be explained.

When a pixel of a moving image is expressed as a function of screen coordinates x and y and time t, the equation representing the relationship between the differential values of the image, which is the basis of the gradient method, is as follows.

Etu 1 Eyv 1 Et −0 (1) Here, Ex, By, and Et are the brightness of the image x
, y and the partial differential coefficients in the direction, u and v are the x and y components of the movement vector at that point. The relationship shown in equation (1) is given independently at each point on the image, but -
Since it is not possible to determine u and v only by the relationship between points, we set a small head area S on the image as shown in Figure 2, and
It is solved assuming that the moving hectares u and v within are constant. This concept is a model for detecting movement vectors, and the movement vector detection model can be summarized as follows.

(1) Model for detecting moving hectares The relationship between the partial differential coefficient and the moving vector expressed by equation (1) holds true everywhere within the region S.

-The movement vector is constant within the small area S on the image.

This model can be interpreted mathematically as follows. formula(
Ex, Ey, and Et in 1) can be seen as functions defined on the domain S, and what this model claims is that the partial derivative Et calculated from the image can be similarly defined on the domain S. This can be expressed by the linear sum of E8 and Ey calculated by E8 and Ey.

Et(x,y) 1-u EX(x,y)-v Ey(x,y)(2
) However, each partial derivative actually calculated from the image cannot be decomposed as shown in equation (2) because the above conditions do not completely hold, that is, this model does not completely apply. There is a residual Et that cannot be expressed as a linear combination of EX and By.

Et (x, y) u EX (X, y) - v By (x, y) + Er
(x, y) (3) Here, these relationships will be explained using an analogy between a function space and a vector space as follows.

FIG. 3 is a diagram in which the functions Ex, Ey, Et, and Et on the small region S are expressed as vectors in a three-dimensional space, respectively. Equation (2) means that vector Et is on the plane spanned by vectors Et and Ey, and equation (3) means that vector Et is not completely on this plane due to noise or incomplete model. This means that it is necessary to add the vector Et instead.

Now, in order to apply the model indicated by ■ to the results detected from the actual moving image, the magnitude of Et in equation (3) IEt
It is sufficient to choose u and v so as to minimize l. This I
The idea of minimizing [Er] is the standard when applying a model.

■Model fitting criteria - Minimize the magnitude lEr1 of the residual Et.

u, v determined in this way, that is, Etl
u and v give the minimum value of the motion vectors (uo, vo) that should be detected.

Moving Haku 1 using the model of ■ and the fitting criteria of ■
- To determine the function Er, the function Er should be chosen to be orthogonal to the functions Ex and Ey, as is clear from the analogy with vectors.

(Er-Et) −〇, (Et-Ey)=O(4)
However, (r, -r2) does not indicate the inner product of the functions f1 and f2.

(r+・rz)=l Ssr+ (x, y)・fz(x
,y)dxdy (5) Assuming that the movement vector to be determined is (uo, vo), the formula (
By creating an inner product of 3), EX, and Ey, and applying the condition of equation (4), the following equation is obtained.

However, a=S b=s f=S g=S h=S Therefore, 55EX(x, y)”dxdy (7)
X sEy (x, y) 2d x d yrsE
y(x, y)Et(x, y)dxdySsEt
(x, y)EX(x, y)dxdySsEx(
x, y) Ey (x, y) dxdy The movement vector to be calculated is as follows [Problem to be solved by the invention] The movement vector is calculated by the method described above, but in the image actually obtained, the image The reliability of the calculated moving objects depends on the noise contained in the image and the pattern of the object being photographed. However, conventional methods do not have a way to estimate this reliability.

The purpose of the present invention is to provide an index indicating the reliability of a movement vector, and to use only highly reliable detection results, or even propagate a highly reliable part to a low reliable part.
It is an object of the present invention to provide a moving hectare detection method and apparatus that enable accurate determination of the moving hectare of the entire image.

[Means to solve the problem]

The first invention is based on a temporally continuous image E(x).

y, t) and the temporal differential Et and the movement vector (u, v) of the image, and the movement within the area S set on the image. The moving hector (
In the movement vector detection method in which uo, vo) is the movement vector of the area S on the image, the value calculated from the spatial differential value a=s5sExCx,y)2dxdyb=SSsEy
(x, y) 2dxdy h = S S 5EX (x, y) Ey (x
, y) dxdy as coefficients, or a value derived from this value as a value indicating the reliability of the detected movement vector in the angle θ direction.

A second invention is a method according to the first invention for each of a plurality of regions S ij set on a temporally continuous image E(xy, t) and extending in the spatial x and y directions. By applying the movement vector detection method, the movement vector (u O4j + V
0ij), and set an energy functional that expresses the constraint that the distribution of moving edges on the image is smooth and that the amount of correction from the moving vector is small, and the value of this functional is In a movement vector detection method that detects the movement hector field (optical flow) of the entire image by correcting the movement hector so as to minimize it, reliability is imposed on the constraint that the amount of correction from the movement vector of the functional is small. It is characterized by reflecting the index R2.

A third invention provides a converter that converts a temporally continuous image sequence obtained by an image capturing device into a discrete representation, and a frame memory that holds temporally adjacent frames of the converted discrete representation images. and an image difference calculation device that calculates the spatial differences Ex, Ey and frame difference EL of the image held in the frame memory, and each of the difference values for each of the plurality of regions Sij set on the image. Sum of squares and products a, J = ΣEx (x, y) 2 b ij = ΣEy (x-y) 2 h ij - ΣEX (X, y) Ey (x, y) f, j = Σ
Ey (x, y) Et (X, y)g ij
= ΣEt (x, y) A movement vector detection device comprising a sum calculation device that calculates Ex (x, y) and a movement vector calculation device that calculates a movement hector for each region using the summation, Reliability of calculating the eigenvector e + ij + e 2ij, which is a matrix created from the summation, and outputting the numerical values of these eigenvalues and eigenvectors, or the numerical values derived from these eigenvalues and eigenvectors, as an index representing the reliability of the detected moving hectare. It is characterized by comprising a gender index calculation device.

A fourth invention provides a converter that converts a temporally continuous image sequence obtained by an image capturing device into a discrete representation, and a frame memory that holds temporally adjacent frames of the converted discrete representation images. and an image difference calculation device that calculates the spatial differences Ex, Ey and frame difference Et of the images held in the frame memory, and each of the difference values for each of the plurality of regions Sij set on the image. Sum of squares and products aiJ = ΣEx (x, y) 2 b, = ΣEy (x, y)" h, J - ΣEX (x, y) Ey (x, y) f, J = ΣE
y(x,y) Et(x,y)g2, -IEt(x,y
) Ex(x, y); and a movement hector calculation device that calculates an initial movement vector for each area using the summation. U8, generated from the vector and summation.

The present invention is characterized in that it includes a movement vector correction device that calculates a solution to the simultaneous-order equations regarding Vij and outputs it as a final movement hectare.

[Effect]

According to the model for detecting the movement vector described above, the movement of the image pattern is expressed by its time differential function E.

(x, y) and detected as its magnitude IEt. Therefore, I caused by a constant amount of movement
The larger the size of Etl, the easier it is to detect movement of the image pattern because it is less affected by noise, and conversely, the smaller the Etl is, the more difficult it is to detect. That is, if R is the ratio between the magnitude of the movement vector and the magnitude of IEt1 caused by it, the reliability of the detected movement vector can be estimated by the value of R.

Now, if the image pattern is moved by δr in the θ direction, the movement vector (δU, δV) can be expressed as follows.

δU-δr cos θ δV-δr sin θ Letting the magnitude of Et observed by this movement be δIEt, the following is obtained.

δ Et-δr EX cos θ + Eysin θ (
9) Define R as shown below and calculate R2 as shown below.

R-δ1Etl/δR(10) R” = l Et cosθ+Eysinθ Formula (11
), it can be seen that R2 has a quadratic form of cos θ and sin θ. This quadratic coefficient matrix depends only on the pattern of the image, and the change in R due to the moving direction θ of the pattern is expressed as a function of θ.

In order to find the maximum and minimum of R, the quadratic form of 2 (11) is converted to the principal axis as follows.

R2= ・ ・ ・ (12) Here, λ1. λ2 is the eigenvalue of the coefficient matrix, eI+82
are respectively λ1. It is an eigenvector of magnitude 1 corresponding to λ2. Therefore, if λ is selected so that ≧λ2, R2 takes the maximum value λ when θ is in the direction of e, and e
2, it can be seen that λ2 is taken. Therefore, R is the same, and its value is the square root of each eigenvalue. To illustrate these relationships, the ease of detecting movement is expressed as direction θ and magnitude R.
When expressed as a vector, its locus becomes an ellipse as shown in FIG. 4, its long axis and short axis are in the direction of the aforementioned eigenvector el+eZ, and its length is equal to the eigenvalue λ1. λ
It is the square root of 2. Therefore, the most reliable directional component and the degree of reliability of the movement vector calculated by equation (8) are used as an index of 1 to 1, and conversely, the least reliable directional component and its degree are shown as an index of r2 is defined as follows using the eigenvalues and eigenvectors appearing in equation (12).

r, -ff, e,
(13) r2 e2 Further, these eigenvalues and eigenvectors are calculated as follows using the values of equation (7).

e, −(h, λ+a)/! 5 ez - (h, λza)/r = 7 (14) Here, depending on the purpose, it is possible to use a value expressed by using polar coordinates for this vector, or furthermore, the above eigenvalues and eigenvectors can be expressed as It can also be used as is as an index representing reliability.

Next, the movement vector calculated in the above-described manner for each region S ij on the image shown in FIG. We will explain how to obtain the overall movement vector. The movement vector of the entire image is generally called an optical flow, and there is an example using the assumption that the distribution of movement hectares is smooth as a property of this optical flow (Bel?THOLD K, P, ll0R
N, BRIAN G, 5CtlUNCK: “Dete
rmining 0ptical Flow,
Herti Oral Intelligence
, volume 17+ pages 185-20
3.1981).

Here, we adopt this assumption and use the deterministic relaxation method (Katsuhiko Itagami, Naokazu Yokoya, "Relaxation method and regularization", Information Processing, vol.
ume 30. No. 7. September 1989), each movement vector is corrected while considering its reliability, and the optical flow is determined.

To solve a problem using the relaxation method, we must first set the energy functional to be minimized.

The energy functional Φ is the functional ψ expressing the smoothness constraint
and Ω representing the constraint of the detected movement vector.

Φ(u (x, y), v (x, y)) − αψ(u (
x, y), v (x, y)) + Ω(u (x, y)
, v (x, y) ) (15) Here,
α is a constant that balances the two constraints.

The functional ψ expressing the smoothness constraint expresses the deviation from the neighboring movement vector as the square of the gradient of each component u and v,
Assume that it is integrated over the entire vector field.

・ ・ ・(16) The functional Ω representing the constraint from the detected movement vector is
Considering the reliability of each movement vector, it is determined that a correction in a direction with high reliability requires a large amount of energy, while a correction in a direction with low reliability requires less energy.

Ω-5s(r,・Δv)2+(r2・Δv)2dxdy
・ ・ ・(17) However, Δv−(u−u,,v−vo) (1
8) Here, r and r2 are the reliability vectors shown in equation (13), and uo and vo are the detected movement vectors shown in (8). The meaning of the energy shown in Equation (17) is as shown in Figure 5, the component of the correction vector ΔV in the r1 direction is weighted with the magnitude of r, and the component in the r2 direction is similarly weighted with the magnitude of r2. It is the square of the distance measured by multiplying the weight by the distance.

Now, the energy functional Ω is determined by the relationship of equation (13) and the equation (
Using the relationship used in transforming equation (11) into equation (12), the equation can be transformed as follows.

Ω-5SλI(el・Δv)2 10λ2(e2・Δv)2dxdy・Δv dxdy = S S a (u-u,) 2+2h (u-u
o) (v+ (v-vo)2 dxdy vo) ・ ・ ・ (20) However, d = a uo”+2 h uovo ten b vo2+2
g uo+2 f v, +c (2
1) The terms related to u and v in equation (20) are equal to the inside of the integral of the last equation of equation (19). Therefore, the energy functional Ω represents the sum of the residual power of the time differential function, which is increased by modifying the shift tit hector, over the entire hector field.

Now that we have set the energy functional, we will next find the moving vector field that minimizes it. For this purpose, the Euler equation of the energy functional Φ shown in equation (15) is obtained as follows.

a (u-uo) tenh (v-vo)h (u u
o) +b (V VO)・・・・(19)
become.

Above, the functional expressing the constraint from the movement vector is
a (u-uo) 10h (v-vo)-α(uxx+
uyy) −〇 was set, but now from equation (3), the time h (u uo) +b (v
vo) a (Vxx+vyy) =.

Calculate the power IEt+2 of the residual Et of the differential function
°゛°(22)
If you try it, you will get something like this:
If we discretize this equation and replace the differential with the difference, we get the following E
r12=a (u u6) 2
The formula is obtained.

+2h (u-uo) (v-vo) 10b (v-Vo
) 2+d 0-(atJ+4α)utJ+hv8,
(23) α (ui++ , + ui−1
j+ ui j+1+ 11 i j-1) a i
j u Oij h ij V 0ijO= (b
tJ+ 4 α) Vzzj 1h ufjOr: (v
=++ j+ V4-11+ V= j+I +Vt
J-+) b ij V Oij h ij u O
ij where i, j represent the lattice points of the moving vector field x
, is a subscript in the y direction, and corresponds to the value calculated in the area S ij on the image. Also, the differential values of u and v are replaced with the difference as follows.

uxx=ui++ j 2 uij+ui−+ j
(24) uVV-ur j+I2 u, J-1
-ui j-1VXx-V i+1 j 2 V i
j+ Vi−1jvyy=Vi j++ 2 Vi
J+Vi j-+Now, the moving vector field that gives the minimum value of the energy functional Φ is U, V shown in equation (23)
It is given by solving simultaneous linear equations for .

There are various methods to solve this simultaneous linear equation numerically (Hayato Toyo: “Numerical calculation of matrices”, Ohmsha,
1971), but assuming that the SOR method is used here, the scheme of equation (23) using the SOR method is determined as follows:+a+Juo=i+1J(Vat; V)l/
(aB+4α) (ξ−1)u ■ : (ξ−1)V Here, the subscripts on the shoulders of u and v indicate the value of the second step of the iterative operation.

In addition, ξ is the acceleration coefficient of relaxation and has a value of 0 to 2,
Generally, a value of about 1.8 to 1.9 is used. Further, when the input image is 8 bits and the size of the area S ij is 16×16 pixels, the appropriate value of α is about 2000. When calculating a moving hector field using equation (25), there are parts around the hector field where there are no pixels, but the vectors there are calculated as being the same as the outermost value of the vector field at that stage. . According to the expression used in splines, this is an open end condition.

〔Example〕

A first embodiment of the invention is shown in FIG.

The analog video signal obtained by the imaging device 1 is A/
It is discretized by the D converter 2 and becomes a digital image. This digital signal is held in the frame memory 3.

The frame memory 4 is a memory that holds the digital image of the previous frame, and when the image of a new frame is sent from the A/D converter 2 to the frame memory 3, the image that was held in the frame memory 3 until then is is held in the frame memory 4.

The image difference calculation device 5 calculates spatial and temporal differences Ex, Ey, and Et using images of adjacent frames held in two frame memories. That is, the image held in the frame memory 3 is E (x, y, to)
, the image held in the frame memory 4 is E (x,
y, to-1), each difference value is calculated by the following equation.

Et −E (x, y, to) E (x
i y, to)Ey=E(x, y, t
, o) -E (x, y-1, to) El =E
(x, y, to) E (x, y,
to 1) (26) Here, for Ex and Ey, the difference between the contents of the two frame memories may be averaged.

The sum calculation device 6 uses each difference value calculated by the image difference a1 calculation device 5 to calculate the sum of the squares and products of the difference values shown in equation (7). Here, the range for calculating the sum is the area Sij shown in FIG.
The total sum is calculated for the number of regions. When the size of the entire image is about 640 x 400 pixels, the sizes of the regions S and J are often about 4 x 4 pixels to about 16 x 16 pixels.

The movement vector calculation device 7 uses this summation to perform the calculation shown by equation (8), and calculates the movement vector (uo, Vo).
Output.

Similarly, the reliability index calculation device 8 calculates the formula (13
) is calculated and output.

A second embodiment of the invention is shown in FIG. Imaging device 1, A
/D converter 2, frame memory 3, frame memory 4,
The processing and signal flow of the image difference calculation device 5, summation calculation device 6, and movement vector calculation device 7 are the same as in the first embodiment described above, but in this embodiment, the output of the movement vector calculation device 7 and The output of the reliability index calculation device 8 is input to the movement vector correction device 9, which performs an iterative calculation according to the schema shown in equation (25) to correct the movement vector, and uses the result as a final movement vector field (optical output as a flow).

In each of the above embodiments, the image capturing device 1 is included in the movement vector detecting device, but it is also possible to process the output of a video tape recorder or the like as input without including this device.

〔Effect of the invention〕

According to the present invention, it is possible to obtain an index indicating the reliability of movement vectors detected from moving images. For example, in applications for detecting moving objects, by using only highly reliable movement vectors, highly accurate movement can be achieved. It becomes possible to detect objects.

Furthermore, the present invention makes it possible to accurately detect the movement vector field (optical flow) of an image. For example, in applications where three-dimensional movement of a camera is detected from this movement vector, this movement can be detected with high accuracy. It becomes possible.

[Brief explanation of the drawing]

Figure 1 is a diagram showing the first embodiment of the present invention, Figure 2 is a diagram showing a small area set on a moving image, and Figure 3 is a diagram showing the relationship between each derivative of the detected image. , FIG. 4 is a diagram showing changes in the reliability of movement vectors depending on the direction, FIG. 5 is a diagram showing the relationship between vectors representing reliability and correction vectors, and FIG. 6 is a diagram showing a second embodiment of the present invention. It is a diagram. 1... Imaging device 2... A/D converter 3... Frame memory 4... Frame memory 5... Image difference calculation device 6...・Total calculation device 7...Movement hectare calculation device 8...Reliability index old calculation device 9...Movement hectare correction device

Claims (4)

[Claims] (1) Equation E_x・u showing the relationship between the spatial differentials E_x, E_y and the temporal differential E_t of a temporally continuous image E(x, y, t) and the movement vector (u, v) on the image
+E_y・v+E_t=0, which is calculated when applying the condition that the movement vector (u, v) is constant within the area S set on the image to each differential value calculated from the actual image. In a movement vector detection method in which a movement vector (u_0, v_0) that minimizes the sum of residuals |E_r| of the equation is set as a movement vector of a region S on the image, =∫∫_sE_x(x,y)^2dxdyb=∫∫_
sE_y(x,y)^2dxdyh=∫∫_sE_x(
x, y) E_y (x, y) dxdy is the quadratic form ▲ There are mathematical formulas, chemical formulas, tables, etc. A movement vector detection method characterized in that it is used as a value indicating reliability. (2) Multiple areas S extending in the spatial x and y directions, set on both temporally continuous images E (x, y, t)
The movement vector detection method according to claim 1 is applied to each of the movement vectors (u_0_i_j, v
_0_i_j), set an energy functional representing a constraint that the distribution of the movement vector on the image is smooth and a constraint that the amount of correction from the movement vector is small, and minimize the value of this functional. In a movement vector detection method that detects a movement vector field (optical flow) of the entire image by correcting the movement vector as shown in FIG. A movement vector detection method characterized by reflecting ^2. (3) a converter that converts a temporally continuous image sequence obtained by the image capturing device into a discrete representation; a frame memory that holds temporally adjacent frames of the converted discrete representation image; An image difference calculation device that calculates spatial differences E_x, E_y and frame difference E_t of images held in a frame memory, and a square and product of each difference value for each of a plurality of regions S_i_j set on the image. Total sum a_i_j=ΣE_x(x,y)2 b_i_j=ΣE_y(x,y)^2 h_i_j=ΣE_x(x,y)E_y(x,y)f_
i_j=ΣE_y(x,y)E_t(x,y)g_i_
j=ΣE_t(x,y)E_x(x,y), and a movement vector that uses the summation to calculate the movement vector ▲There are mathematical formulas, chemical formulas, tables, etc.▼ for each area. In a movement vector detection device consisting of a calculation device, the eigenvalues λ_1_i_j, λ_2_i_j of the matrix ▲ where there are mathematical formulas, chemical formulas, tables, etc.
i_j and outputs numerical values of these eigenvalues and eigenvectors, or numerical values derived from these eigenvalues and eigenvectors, as an index representing the reliability of the detected movement vector. Movement vector detection device. (4) a converter that converts a temporally continuous image sequence obtained by the image capturing device into a discrete representation; a frame memory that holds temporally adjacent frames of the converted discrete representation images; An image difference calculation device that calculates spatial differences E_x, E_y and frame difference E_t of images held in a frame memory, and a square and product of each difference value for each of a plurality of regions S_i_j set on the image. Total sum a_i_j=ΣE_x(x,y)^2 b_i_j=ΣE_y(x,y)^2 h_i_j=ΣE_x(x,y)E_y(x,y)f_
i_j=ΣE_y(x,y)E_t(x,y)g_i_
A sum calculation device that calculates j=ΣE_t(x,y)E_x(x,y), and a movement that uses the sum to calculate the initial movement vector ▲There are mathematical formulas, chemical formulas, tables, etc.▼ for each of the above regions. In a movement vector calculation device consisting of a vector calculation device, simultaneous linear equations regarding u_i_j and v_i_j generated from the movement vector and the sum 0=(a_i_j+4α)u_i_j+hv_i_j−
α(u_i_+_1_j+u_i_-_1_j+u_i
＿j＿＋＿１＋u＿i＿j＿−＿１−a＿i＿ju＿０
_i_j−h_i_jv_0_i_j0=(b_i_j
+4α)v_i_j+hu_i_j−α(v_i_+_
1_j+v_i_-_1_j+v_i_j_+_1+v
_i_j_-_1)-b_i_jv_0_i_j-h_
A movement vector detection device comprising a movement vector correction device that calculates a solution to i_ju_0_i_j and outputs it as a final movement vector.