JP2882136B2 - Video analysis method - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention detects information (motion parameters) relating to the motion of a camera using a moving image obtained from a camera installed on an object moving in a stationary environment such as a car. The present invention also relates to a technology for detecting an object that moves in a stationary environment other than itself, such as a preceding vehicle, using this information.

[0002]

2. Description of the Related Art A process for obtaining motion information of a camera on which a moving image is captured using moving information of a moving image generally has a configuration shown in FIG. First, movement vector detection processing 2
In step 1, a motion vector (optical flow) on the image is detected from the moving image, and then, in a motion parameter calculation process 22, a motion parameter representing the self-motion is calculated by analyzing the motion vector. Regarding the movement vector detection processing, for example,
"Measurement of moving amount and speed of moving object by image signal", IEICE Technical Report, IE-67, 1978-10,
Or Masahiro Wada, Hirohisa Yamaguchi, "Motion Detection of Moving Image Signal by Iterative Gradient Method", IEICE Transactions D, 19
85-4, pp. 663-670. The motion parameter calculation process is described in, for example, Anna R. Bruss, Bert
hold K. P. Horn, "Passive Na
navigation ", COMPUTER VISIO
N, GRAPHICS, ANDIMAGE PROCE
SSING, 21, 3-20, 1983.

[0003]

Although the motion parameters can be detected from a moving image by the method described above, there are the following problems. The detection accuracy of the movement vector detected by the movement vector detection processing differs in each part of the image. Therefore, the motion vector includes a large error, and the above-described motion parameter calculation process calculates the motion parameter also using such an erroneous motion vector. I can't calculate. Therefore, in order to calculate an accurate motion parameter, it is necessary to consider the accuracy of the detected motion vector. For example, Naoya Ota, “Detection of Moving Vector with Reliability Index”, Computer Vision '90, Symposium Papers, 19
90-8, p. 21-30, can be known by introducing the reliability index reported in 21-30. However, conventionally, there has been no method of calculating a motion parameter by effectively using such information on the accuracy of a movement vector. It is an object of the present invention to provide a technique which makes this possible,
To be able to detect motion parameters with high accuracy, and to use motion parameters and motion vectors and information on their accuracy to detect other moving objects even if they are moving in the environment. Is to make it possible.

[0004]

According to a first aspect of the present invention, there is provided a moving image analyzing method using a moving vector (optical flow) at each point on a moving image captured by a camera which moves relative to an environment. , in the moving image analysis method for detecting information (motion parameters) relating to exercise of the camera, the motion vector v _di detected from the moving image, the moving vector calculated from a certain motion parameters theoretically v _i
The difference epsilon _i ² with using the vector r _{1 i,} r _{2 i} representing the reliability of the detected moving vector from the moving ^{_{image, ε i 2 = ((v}} di -v i) · r 1 i ) ² + ((v _di −v _i ) · r _{2 i} ) ^2, and the sum of the differences for each motion vector

[0005]

(Equation 2)

[0006] It is characterized in that a motion parameter that minimizes the above is detected as a motion parameter of an actual camera.

[0007] moving image analysis method of the second invention, in the moving picture analysis method of the first aspect of the invention, movement vector _{v i = (u i, v} i) which is calculated from the motion parameters theoretically and
Parameter indicative of the rotational movement _{(Ω 1, Ω 2, ω} 3) and parameters representing the translational movement (x _e, y _e) from the formula _{u i = (x i -x e} ) t i + Ω 1 x i y i - ^{_{Ω 2 (x i 2 + f}} 2) + ω 3 x i v i = (y i -y e) t i + Ω 1 (y i 2 + f 2) -Ω be calculated by _{2 x i y i -ω 3 x} i It is characterized by.

[0008] moving image analysis method of the third invention, in the moving picture analysis method of the first aspect of the invention, movement vector _{v i = (u i, v} i) which is calculated from the motion parameters theoretically and
Parameter indicative of the rotational movement (u _r, v _r) and parameters representing the translational movement (x _e, y _e) from the formula _{u i = (x i -x e} ) t i + u r v i = (y i -y and calculating by _e) t _i + v _r.

A moving image analysis method according to a fourth aspect of the present invention is the moving image analysis method according to the second aspect, wherein the entire difference between the movement vector detected from the moving image and the movement vector theoretically calculated from the movement parameters is obtained. The motion parameters that minimize P are the parameters representing the rotational motion (Ω ₁ , Ω ₂ ,
ω ₃ ), t _i and parameters (x _e ,
y _e ) is obtained by sequentially and repeatedly performing optimization for each of y _e ).

According to a fifth aspect of the present invention, in the moving image analysis method according to the third aspect of the present invention, an entire difference between a movement vector detected from the moving image and a movement vector theoretically calculated from the movement parameters is obtained. the motion parameters that minimize the P, the parameter representative of the rotational movement _{(u r, v r),} t i
And optimization for each of the parameters (x _e , y _e ) representing the translational motion is sequentially and repeatedly performed.

According to a sixth aspect of the present invention, there is provided a moving image analysis method, comprising: a movement vector (optical flow) at each point on a moving image captured by a camera moving relative to an environment; In the moving image analysis method for detecting an object moving with respect to the environment using the movement parameter), the movement vector v _di detected from the moving image and the movement vector v _di calculated theoretically from the movement parameter the difference epsilon _i ² with the movement vector v _i, by using the vector r _{1 i,} r _{2 i} representing the reliability of the detected moving vector from the moving image, wherein ^{_{ε i 2 = ((v di}} -v i ^{_{) · r 1 i) 2 +}} ( calculated by _{(v di -v i) · r} 2 i) 2, and detects the object that the mobile using the value of the difference.

A moving image analysis method according to a seventh invention is characterized in that, in the moving image analysis method according to the sixth invention, the motion parameters of the camera are obtained by the moving image analysis method according to the fourth invention.

An eighth aspect of the present invention provides the moving image analysis method according to the sixth aspect, wherein the motion parameters of the camera are obtained by the moving image analysis method according to the fifth aspect.

[0014]

First, a motion vector field (referred to as an optical flow) on an image generated by a motion of a camera in a stationary environment will be described. The coordinate system (X, Y, Z) of the camera is set as shown in FIG. That is, the lens center is set to the origin 0, and the Z axis is selected so as to coincide with the optical axis of the camera.
The plane represented by Z = f is defined as an imaging plane, and the coordinate system (x, y) of the imaging plane is set so that the x and y axes are oriented in the same direction as the X and Y axes of the camera coordinate system. Here, f is the focal length. Assuming that p = (x, y) is a point on the imaging surface where the point P = (X, Y, Z) on the object is projected, there is a relationship expressed by Equation 1 between them.

[0015]

(Equation 3)

Any instantaneous motion (velocity) of the camera can be represented by a combination of translational and rotational components. Now, it is assumed that the coordinate system of the camera has a translation speed T with respect to the stationary environment and a rotation speed ω about the origin 0 of the camera coordinate system. These two vectors T and ω are motion parameters. At this time, the point P on the stationary environment is equivalent to the movement of the translation speed -T and the rotation speed -ω with respect to the origin 0 with respect to the camera coordinate system. Therefore, if each component of T and ω is represented by the coordinate system of the camera, the speed V of the point P with respect to the coordinate system of the camera is as follows, where r is the position vector of the point P.

V = −ω × r−T (2) where V = (X ′, Y ′, Z,) ^T (3) r = (X, Y, Z) ^T T = a _{(τ 1, τ 2, τ} 3) T ω = (ω 1, ω 2, ω 3) T, obtaining a formula 4 if Kakikudase each component of V.

[0018] _{X '= - τ 1 -ω 2} Z + ω 3 Y (4) Y' = - τ 2 -ω 3 X + ω 1 Z Z '= - τ 3 -ω 1 Y + ω 2 X image caused by movement of the point P Speed v of point p above =
(U, v) is obtained by time-differentiating the coordinates (x, y) of the point p expressed by the equation (1).

[0019]

(Equation 4)

By substituting equation (4) into the above equation and rearranging using the relationship of equation (1), the following equation is obtained.

[0021] _{u = u t + u r (} 6) v = v t + v r u t = (τ 3 x-T 1) / Z (7) v t = (τ 3 y-T 2) / Z u r = Ω ₁ xy−Ω ₂ (x ² + f ² ) + ω ₃ y (8) v _r = Ω ₁ (y ² + f ² ) Ω ₂ xy−ω ₃ x where T ₁ = fτ ₁ , T ₂ = fτ ₂ (9 Ω ₁ = ω ₁ / f, Ω ₂ = ω ₂ / f Each point in the stationary environment is a position (x,
By y), the speed (u, v) represented by Expression 6 is obtained on the imaging surface. Therefore, this (u, v) is an optical flow generated by the movement of the camera. Equation 6
Shows that the optical flow (u, v) is represented by the sum of the translation component (u _t, v _t) and the rotation component caused by the rotational movement (u _r, v _r) caused by translational movement of the camera . Rotation component (u _r, v _r) is located on the rotational speed ω and the imaging surface of the camera (x, y) by but is uniquely determined, the translation component (u _t, v _t) camera translation speed T and imaging It can be seen that it depends not only on the position (x, y) on the surface but also on the value of the Z coordinate of the point P in the three-dimensional space. Here, it rewrites the translation component (u _t, v _t) and as follows.

U _t = (x−x _e ) t (10) v _t = (y−y _e ) t where x _e = f (τ ₁ / τ ₃ ), y _e = f (τ ₂ / τ ₃ ) since _{t = τ 3 / Z (11} ) t is dependent on _{Z, (u t, v t} ) of magnitude is dependent on Z, but that direction the point on the imaging surface does not depend on the Z (x _e , Y _e ). This point is FOE
(Focus of Expansion).

Detecting the movement parameter of the camera from the optical flow is to determine T and ω when the distribution of the movement vector v on the imaging surface is given. It cannot be determined only from optical flow. This depends on the following circumstances.
Each value shown in Expression 11 does not change even if each component of T and Z are multiplied by a constant. This indicates that the same optical flow occurs if the ratio of each component of T to Z is the same. Therefore, all that is possible is to determine the ratio of each component of T, in other words, the direction of T. Since this arbitrariness is included only in the relationship of Equation 11, (x _e ,
y _e ), t, ω can be determined from the optical flow. Hereinafter, (x _e , y _e ) and ω are referred to as motion parameters. t is an amount reflecting the z-coordinate value Z of the point P, and exists independently for each position on the imaging surface.

From the above discussion, the problem to be solved is that the motion parameters (x
_e , y _e ) and ω. In principle, this problem can be solved if a motion vector is given at eight points on the image (HC Longust-Higgi).
ns, “A computer algorithmmf
or restructuring a scene
from twoviews ", Nature, 29
3-10, 1981 pp. 133-135). However, since it is impossible to detect an accurate movement vector such that this method has realism, a movement vector that generates the (u, v) that best fits them using movement vectors measured at many points is used. Find the parameters and use them as solutions. Now, it is assumed that there are N points at which the movement vector is measured, and the i-th measured movement vector is represented by v _di =
Let (u _di , v _di ) and the motion vector at that point calculated from the motion parameters be v _i = (u _i , v _i ).
When the quantity ε _i ² representing the difference between these vectors is defined in consideration of the reliability of the movement vector, the following equation is obtained.

Ε_i ^Two= (Δv_i・ R_{1 i})^Two+ (Δ_vi・ R_{2 i})^Two (12) δv_i= V_di-V_i Here, (ab) represents the inner product of the vectors a and b. This
Where the vector r_{1 i}, R_{2 i}Is the reliability of the movement vector
, For example, the reliability R in the θ direction_θ Is
(Naoya Ota, "Moving vector with reliability index
Detection, "Computer Vision '90, Symposium
Proceedings, 1990-8, pp. 139-143. 21-30).

[0026]

(Equation 5)

Assuming that the direction of δv _i is θ, δv _i = | δv _i | e, so ε _i ^{2 in} equation 12 is expressed by the following equation.

The ^{_{ε i 2 = (R θ |}} δv i |) 2 From this, the difference in vector, weights the difference .delta.v _i of the actual vector in that direction of the reliability, to be measured in those squares Become. Now, the difference of the entire motion vector field is measured by a value P obtained by adding the differences of the individual vectors.

[0029]

(Equation 6)

This value P is called energy. Problem to be solved is when given a _{"v di, r 1 i, r} 2 i,
(X _e , y _e ), t _i , and ω to minimize the energy P ”. Here, the suffix i added to each variable indicates that it relates to the i-th point, and there are N in total. Since this problem is an unconstrained nonlinear optimization problem, this problem can be solved using the steepest descent method, the conjugate gradient method, etc. (“Optimization”, Iwanami Kogaku Informatics-19,
Iwanami Shoten, 1982). According to the method described above, it is possible to detect a motion parameter in consideration of the reliability of a motion vector detected from a moving image.

Next, a moving object detection process using the detected motion parameters will be described. The basic idea is as follows. Once the motion parameters are determined, the optical flows that can be caused by the motion are known. Since this optical flow is a result calculated under the assumption that the camera is moving in a stationary environment, if there is a moving part that can not be represented by the optical flow, that part is moved to a stationary environment. On the other hand, it is considered that the object is moving. Ε _i ² shown in Equation 12 is
Calculate the difference between the theoretically calculated motion vector v _i = (u _i , v _i ) and the actually detected motion vector v _di = (u _di , v _di ) in consideration of the reliability of the motion vector. As a result, a large value is taken when the movement vector does not match the theoretically calculated movement vector even though the movement vector has been detected with high reliability. Therefore, the region of the moving object described above can be obtained by detecting a portion where ε _i ² shows a large value.

[0032]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS As described above, a problem to be solved for obtaining a motion parameter in consideration of the reliability of a motion vector is an unconstrained nonlinear optimization problem, which can be solved by using various nonlinear optimization techniques. . However, in this problem, the variables are (Ω ₁ , Ω ₂ , ω ₃ ), t _i and (x _e , y _e )
In each of the three groups, since each group has a feature that a value that minimizes P can be analytically obtained, an efficient algorithm utilizing this feature can be configured. That is, first (Ω
_1, Omega _2, by fixing the omega ₃₎ other than the variable, to minimize P (Ω _1, Ω _2, obtaining the omega _3). Then (x _e , y _e )
And the same for t _i . With these processes as one cycle, this process is repeated until the solution converges. First, we describe calculation method of (x _e, y _e) for the value that minimizes the _{P (x e 0, y e} 0). The value to be obtained can be obtained by calculating partial derivatives of P with respect to x _e and y _e and setting each of them to 0.

[0033]

(Equation 7)

Here, r _{1 xi} and r _{1 yi} and r _{2 xi} and r _{2 yi} are x and y components of the vectors r _{1 i} and r _{2 i} representing reliability, respectively. Also,
u _ri and v _ri are calculated by Expression 8. Next, similarly, t _i that minimizes P is obtained by partially differentiating P with t _i and setting it to 0. The value to be found is t
_{Notated as i 0} .

[0035]

(Equation 8)

Here, a _i , b _i , and h _i are given by Equation 20, u
_ri and _vri are calculated by Expression 8. Similarly, Omega ₁ to minimize P, Ω _2, ω ₃ of the Ω _{1 0,} Ω _{2 0,} if omega _{3 0,} these values are given by the following equation.

[0037]

(Equation 9)

[0038]

(Equation 10)

[0039]

[Equation 11]

In the above equation, a _i , b _i , and h _i are given by Equation 2.
0, u _ti and v _ti are calculated using Equation 10. From these results, the algorithm for determining the motion parameters is as shown in FIG. First, in step 1, an initial value is given to each parameter. Since it is desirable that the initial value be closer to the actual value, for example, for an image obtained from a camera facing the traveling direction, Ω ₁ = Ω ₂ = ω ₃ = 0, t
Let _i = 0, and let x _e and y _{e be} the center of the image. Then calculate the current energy P in step 2, and sets the variable P _b, which sets a variable k to hold the number of repetitions. In step 3, optimization regarding (Ω ₁ , Ω ₂ , ω ₃ ) is performed. That is, using Equation 22, (Ω ₁₀ ,
Ω ₂₀ , ω ₃₀ ) and calculate this as a new (Ω ₁ ,
Ω ₂ , ω ₃ ). In step 4, t _{i 0} is calculated by using equation 21 to obtain a new t _i, and in step 5, (x _{e 0} , y _{e 0} ) is calculated by using equation 17, and this is newly calculated. (X _e , y _e ). Next, in step 6, convergence is determined. That is, the energy P at this stage was calculated and compared with the previous cycle energy P _b. Reduction in energy finishes repeated if less than or equal to P _e. Further, repeated rotation k is also terminated the iteration is equal to k _e. Here, _Pe is a convergence determination constant, and _ke is the maximum number of iterations. If the end condition of the repetition is not satisfied, the process proceeds to step 7, where the energy P calculated in step 6 is substituted for P _b ,
And returns to step 3. Thus, steps 3, 4, 5, 6, and 7 constitute one cycle of the iterative operation. Here, the convergence is determined based on the decrease amount of the energy P, but can be simply determined only by the number of repetitions k. The motion parameters (x _e , y
_e ), ω and t _i are obtained.

Although the above discussion has dealt with complete rotational movement, objects moving on the ground, such as mobile robots and automobiles, are generally bound to the ground plane, so that the optical axis of the camera should be installed parallel to the ground plane. For example, it can be assumed that the rotation about the Z axis is small. If a camera having a small angle of view (close to a telephoto lens) is used, it can be assumed that the position (x, y) of the object on the imaging surface is sufficiently smaller than the focal length f. Making these assumptions can simplify the calculation of the motion parameters to some extent. These assumptions can be expressed as follows.

[0042]

(Equation 12)

When Expression 8 is formed using these conditions, the following expression is obtained.

[0044] _{u r ≒ Ω 1 xy-Ω} 2 f 2 (24) v r ≒ Ω 1 f 2 -Ω 2 xy In addition, the formula Ω ₁ xy of u _r can be ignored with respect to formula Ω ₁ f ² of v _r since v formula Omega ₂ xy of _r it can be ignored with respect to formula Omega ₂ f ² of u _r, equation 24 can be further modified as follows.

[0045] ^{_{u r ≒ -Ω 2 f 2 (}} 25) v r ≒ Ω 1 f 2 In other words, a constant value regardless of the position on the (u _r, v _r) is the image. To calculate the motion parameters, the following changes may be made to the algorithm shown in FIG. Equations used in steps 4 and 5, u included in equations 21 and 17
replace _ri, the v _ri constant u _r, to v _r. Step _{3 (Ω 1, Ω 2,} ω 3) is optimized regarding (u _r, v
_r ), which is calculated by the following equation. In the following equation _{(u r 0, v r 0} ) is to minimize the P (u _r,
v _r) and is, in step 3, this value is set as a new (u _r, v _r).

[0046]

(Equation 13)

From this algorithm, x that minimizes P
_e, y _e, t _i and u _r, but v _r is computed, if necessary, Omega ₁ by Equation 25, it is converted into Omega _2.

Next, a process of detecting a moving object using the motion parameters detected by the above method will be described. The motion parameters (Ω ₁ , Ω ₂ , ω ₃ ) and (x _e ,
y _e ) and t _i are given, equations (6), (8), (10),
Alternatively, a point (x
_i, with y _i), the movement vector v _i = (u _i constituting the optical flow, v _i) is calculated. Next, the actually detected movement vector v _di = (u _di , v _di ) and its reliability vector r _{1 i} = (r _{1 i} , r _1i ) and r _{2 i} = (r _{2 i} , r _{2) Using i} ), ε _i ² is calculated by Expression 12. A portion where ε _i ² is larger than a certain threshold is detected as a moving object region. In addition, if it can be expected that the area where the moving object appears is somewhat large, it is effective to perform smoothing processing on the image before performing threshold processing. In the above description, the method described in the present patent is used for the motion parameter detection processing, but it is also possible to use the motion parameter detected by another method. In this case, since t _i is not generally given, ２１ _i ² is calculated after obtaining t _i using Equation 21.

[0049]

According to the present invention, the motion parameters of a camera can be accurately calculated by effectively using the reliability of a motion vector detected from a moving image, and an object moving in an environment can be accurately detected. It becomes possible to do.

[Brief description of the drawings]

FIG. 1 is a diagram showing a method for calculating a motion parameter.

FIG. 2 is a diagram showing a configuration of a general motion parameter detection process;

FIG. 3 is a diagram illustrating a relationship between an imaging system of a camera and an object to be imaged.

[Explanation of symbols]

 21 Motion vector detection processing 22 Motion parameter calculation processing

Claims (8)

(57) [Claims] In a moving image analysis method for detecting a movement parameter of a camera using a movement vector at each point on a moving image captured by a camera moving relative to an environment, the moving image Movement vector v detected from image
_di and is the difference epsilon _i ² from motion parameters theoretically calculated moving vector v _i, the vector r ₁ which represents the reliability of the detected motion vector from the moving image _i, r
The ^{_{2 i ε i 2 = ((}} v di -v i) · r 1 i) 2 + ((v di -v i) · r 2 i) 2 and then, a total of [number 1] for each movement vector of the difference A moving image analysis method for detecting a motion parameter that minimizes a motion as a motion parameter of an actual camera. 2. A movement vector v _i = (u _i , v _i ) theoretically calculated from a movement parameter is converted into a parameter (Ω ₁ , Ω ₂ , ω ₃ ) representing a rotational motion and a parameter (Ω ₁ , ω ₂ , ω ₃ ) representing a translational motion x _e, from y _e), the formula _{u i = (x i -x e} ) t i + Ω 1 x i y i -Ω 2 (x i 2 + f 2) + ω 3 y i v i = (y i -y e _{) t i + Ω 1 (y} i 2 -f 2) -Ω 2 x i y i -ω 3 moving image analyzing method according to claim 1, wherein the calculating the x _i. Movement vector v 3. is calculated from the motion parameters theoretically _{i = (u i, v i} ) a parameter indicative of the rotational movement (u _r, v _r) and represents the translational motion parameters (x _e, from y _e), wherein _{u i = (x i -x e} ) t i + u r v i = (y i -y e) t i + v moving picture according to claim 1, wherein the calculating the _r analysis Method. 4. A motion parameter for minimizing an overall difference P between a motion vector detected from a moving image and a motion vector theoretically calculated from a motion parameter is defined as a parameter (Ω ₁ , Ω ₂₎ representing a rotational motion. , Ω ₃ ), t _i, and parameters (x _e , y _e ) representing translational motion are obtained by sequentially and repeatedly performing optimization. 5. The motion parameters for the entire difference P to minimize the movement vector from the movement vector and the motion parameter detected from the moving image is theoretically calculated, representing the rotational motion parameters (u _r, v _r ), t _i and translation represent the parameters (x _e, y _e) optimization for each successive moving picture analysis method according to claim 3, wherein the determination by performing repeated. 6. An object that is moving with respect to the environment is detected using a motion vector at each point on a moving image captured by a camera that moves relative to the environment and a motion parameter of the camera. In the moving image analysis method, the difference ε _i ² between the moving vector v _di detected from the moving image and the moving vector vi theoretically calculated from the motion parameters is detected from the moving image. vector r _{1 i} representing the reliability of the movement vector, with r _{2 i,} wherein ^{_{ε i 2 = ((v di}} -v i) · r 1 i) 2 + ((v di -v i) · r 2 _i ) A moving image analysis method, wherein the moving object is detected by calculating according to ² and using the value of the difference. 7. The moving image analysis method according to claim 6, wherein the motion parameters of the camera are obtained by the moving image analysis method according to claim 4. 8. The moving image analysis method according to claim 6, wherein the motion parameters of the camera are obtained by the moving image analysis method according to claim 5.