JPH07244735A - Method and device for three-dimensional shape restoration - Google Patents

Abstract

PURPOSE:To stably obtain more accurate information by merging distance information, obtained from respective frames of a motion picture, with the motion of a camera with time. CONSTITUTION:The three-dimensional shape restoration device, composed of an optical flow calculation means 10, a distance information calculation means 20, and a distance information merging means 30, is equipped with a distance information merging means 30 using Kalman filters 40 and 50 which are independent of an estimated value of the distance to a body calculated by the distance information calculation means 20 and an estimated value of the motion of the camera.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a three-dimensional shape restoration method and apparatus for analyzing an image to obtain distance information when information on the distance to an object in the environment is required.

[0002]

2. Description of the Related Art In a mobile robot that autonomously moves in an environment or an autonomous vehicle that travels by understanding the road environment, an image obtained by a TV camera is analyzed by a computer to obtain necessary information. Often has a visual system. In these systems, information on the distance to an object in the environment is required for detecting obstacles, for example. A method for analyzing images to obtain distance information is Structure from Motion, which uses movement information of moving images, as well as stereoscopic vision.
There is a method called.

As a technique for detecting an optical flow from a moving image, for example, Ohta N. et al. : "Image Movement D
inspection with Reliability
Indices ", IEICE Trans., E7
4, 10, pp. 3379-3388 (Oct. 199)
1), and as an example of the method to calculate the motion parameters of the camera from the optical flow and the distance to the object being photographed Naoya Ota: “From the optical flow with reliability information Shape Restoration and Its Application to Moving Object Detection ", IEICE Transactions, D-II, Vol. J76-D-II, N
o. 8, pp. 1562-1571 (August 1993). With these two methods, it is possible to calculate the distance from the moving image to the object, the motion parameters of the camera, and the variance of the estimated errors in these values. However, these methods use two frames of the moving image. A method in which distance information calculated from each frame of a moving image sequentially obtained from a camera and the motion of the camera are temporally fused and a Kalman filter is used to obtain a more accurate result is described in Matthies L. et al. M. , Kanade T .; , S
zeliski R. : "Kalman Filter
-Based Algorithms for Esti
mating Depth from Image S
"equence", Int. J. Comput. Vis
ion, 3, pp. 209-236 (1989). However, in this report, as a specific example, only the case where the motion of the camera does not include rotation is discussed, and for the general motion of the camera, Heel J. et al. : "Dynamic Motion V
Ision ”, Proc. of Image Und
erstanding Workshop '89, p
p. 702-713 (1989).

[0004]

The method proposed by Heel is to linearize the relational expression between optical flow and distance information and apply an extended Kalman filter. However, since the relationship between the optical flow and the distance information has a strong non-linearity, the movement of the camera for the correct operation of this method is limited, and there is a problem in the stability of the operation. In order to enable stable motion with respect to general camera movement, it is necessary to construct an algorithm with due consideration of the nonlinearity of this problem.

An object of the present invention is to provide a three-dimensional shape restoration method and apparatus which can temporally fuse the distance information obtained from each frame of a moving image and the motion of the camera to stably obtain more accurate information. To provide.

[0006]

According to the present invention, an estimated value of optical flow, which is movement information on an image, and its covariance matrix are calculated using two frames in a series of images forming a moving image. Optical flow calculation means, distance information calculation means for calculating the distance to the object being imaged using the estimated value and the covariance matrix of the optical flow and the estimated value of the motion of the camera and their covariance matrix, and The distance to a large number of objects obtained by continuously applying the two methods to a moving image, the motion estimation values of the camera, and their covariance matrices are temporally fused to obtain the distance to the object and the camera. In the three-dimensional shape restoration device including the highly accurate estimated value of the motion of the motion vector and the distance information fusing means for calculating the covariance matrix thereof, the distance information fusing means is configured by the distance information fusing means. It is characterized in that it constitutes an independent Kalman filter for each of the estimated value and the estimated value of the camera movement distance to the object to be calculated.

Further, according to the present invention, in addition to the estimated value of the optical flow and its covariance matrix as the input of the distance information calculation means, the magnitude of the translational motion of the camera and the variance of the noise contained therein are used for the calculation. It is characterized in that the distance information fusion means that reflects the estimated value of the distance to the object and the estimated value of the motion of the camera is adopted.

[0008]

If the extended Kalman filter is directly applied to the relationship between the optical flow having strong nonlinearity and the distance information and the motion of the camera, the operation becomes unstable. Therefore,
The distance information calculation means performs a process that sufficiently considers the non-linearity of the problem, obtains values for which the linear approximation is valid, and then applies the Kalman filter to those values. The optical flow calculation means calculates the covariance matrix of the optical flow and the error estimated for it from the moving image, and the distance information calculation means calculates the motion parameter and distance information of the camera and the error covariance matrix contained therein. These values are temporally fused by the Kalman filter of the distance information fusion means and become more accurate values. Since there is a correlation between the distance information calculated by the distance information calculation means and the motion parameter of the camera, it is possible to construct a Kalman filter by combining them, but since the dimension of the state vector is very large, It is not realistic as an actual method. Therefore, an independent Kalman filter is configured for each of the distance information and the motion parameter of the camera. It should be noted that the motion speed of the camera is provided as an input to the distance information calculation means in addition to the optical flow, for the following reason. The distance information obtained by analyzing the optical flow is only a value relative to the magnitude of the translational motion of the camera, and an absolute value cannot be known. An absolute value is necessary for the Kalman filter of the distance information fusion means to perform a correct operation, and in order to obtain this, the absolute value of the translational movement of the camera, that is, the movement speed of the camera, is obtained in the distance information calculation means. This is because it is necessary to input from a device other than the camera such as a speedometer. However, in reality, the movement of the camera is often smooth, and if the time is limited to a short time, there is often no problem even assuming that the speed is constant. In that case,
An appropriate value may be set for the measured value of the speed of the camera and the variance of the noise contained therein.

[0009]

Embodiments of the present invention will now be described with reference to the drawings. FIG. 1 is a block diagram showing an embodiment of a three-dimensional shape restoration device of the present invention. The three-dimensional shape restoration apparatus shown in FIG. 1 includes an optical flow calculation unit 10 for calculating an estimated value of an optical flow and a covariance matrix of an error estimated from the estimated value of an optical flow from a moving image, an estimated value of the optical flow, a covariance matrix, and a camera. Distance information calculating means 20 for calculating the distance information to the object being imaged, the estimated value of the motion parameter of the camera, and the covariance matrix of the error included in them, using the magnitude of the translational motion and the variance of the noise contained therein. , Constructing independent Kalman filters 30 and 40 for each of the estimated value of the distance information and the estimated value of the motion parameter of the camera, and calculating the highly accurate estimated value of the distance information and the motion parameter of the camera and its covariance matrix. The distance information fusion means 50 is provided.

First, the optical flow calculating means 10 of FIG. 1 can be used in the method of the present invention as long as it is a method of outputting the covariance matrix of the optical flow and the noise contained therein. For example, the above mentioned document Ohta N. et al. : "Image Movement D
inspection with Reliability
Indices ", IEICE Trans., E7
4, 10, pp. 3379-3388 (Oct. 199)
1) Or Naoya Ota: “Shape restoration from optical flow and its moving object detection”, Society of Instrument and Control Engineers, 25th Pattern Measurement Subcommittee, pp. 7-13 (1993/11/1
The method reported in 9) can be used. This method outputs the estimated value u _i ^* of the movement vector at the point i on the image and the covariance matrix V _ui of the error of the estimated value (more accurately, its inverse matrix V _ui ⁻¹ ).

Next, in the distance information calculation means 20, the above-mentioned document Naoya Ota: “Shape restoration from optical flow having reliability information and its application to moving object detection”, IEICE Transactions, D -II, Vol. J76-D-II, N
o. 8, pp. The method described in 1562-1571 (August 1993) is helpful, but in this method,
The case of inputting the magnitude of the translational motion of the camera is not described, and will be described below.

A central projection system with a focal length of 1 is adopted as a model of the image pickup system. When the camera moves in the stationary environment at a translational speed T and a rotational speed R, the point i (position (x
_i , y _i )) generated optical flow u _i is expressed by the following equation.

[0013]

[Equation 1]

However,

[0015]

[Equation 2]

In the above equation, the subscript i (= 1, ...,
N) is the number corresponding to each point on the image, and p _i is the reciprocal of the Z coordinate value at the point i on the image (called inverse depth).
Is. On the other hand, the speed k of the camera is given by the following equation.

K = | T | = (T _x ² + T _y ² + T _z ² ) ^1/2 (2) Here, the optical flow, which is an amount to be input to the distance information calculation means 20, and the speed of the camera are set to one. Vector y,
The translation / rotation speed of the camera and the inverse depth, which are the amounts to be calculated, are collectively shown in one vector θ.

Y = [u ₁ v ₁ u ₂ v ₂ ... u _N v _N k] ^T (3) θ = [T _x T _y T _z R _x R _y R _z p ₁ p ₂ ... p _N ] ^T vector y is a function y (θ) of θ, and details are given by equation (1).
And given in (2).

Next, the observation vector y ^* made from the optical flow calculated from the actual moving image by the optical flow calculation means 10 and the camera speed measured by a speedometer etc. has a true value y of 0 on average. Probabilistic modeling is performed assuming that multidimensional Gaussian noise n is added.

Y ^* = y (θ) + n (4) When the covariance matrix of the noise n is V _y , the probability P (y ^{*) of} observing the measured value y ^* when the parameter θ is given ^.
| Θ) is given by the following equation.

P (y ^* | θ) = (2π) ^{−N / 2} | V _y | ^−1/2 exp (−J / 2) (5) J = (y ^* −y (θ)) ^T V _y ⁻¹ (y ^* −y (θ)) The maximum likelihood estimation value θ _ML of the parameter θ when the measured value y ^* is observed is θ that maximizes Equation (5), that is, θ that minimizes J. is there. For the minimization of J, a nonlinear optimization method such as the steepest gradient method can be used. Also, reference: Naoya Ota: “Shape restoration from optical flow with reliability information and its application to moving object detection”, IEICE Transactions, D-II, Vol. J76-D-II, N
o. 8, pp. 1562-1571 (August 1993), in the minimization algorithm shown in Section 2.3, the step of normalizing the magnitude of the translational velocity | T | This algorithm can be applied as it is by changing to the size of ^* .

Next, the method of calculating the covariance matrix of the error of the estimated parameter θ _ML will be described. Due to the non-linearity of the function y (θ), it is not easy to calculate the covariance matrix exactly, so y (θ) is expanded by Taylor around θ _ML ,
Linearize by taking up to the first order term.

Y _L = y (θ _ML ) + (dy / dθ) (θ−θ _ML ) (6) where dy / dθ is the Jacobian matrix. Equation (6) is θ
Solving for

Θ = (dy / dθ) ⁻¹ (y _L −y (θ _ML )) + θ _ML (7) If the covariance matrix of the vector y _L is V _y , the covariance matrix V _h of θ is V _h = (dy / dθ) ⁻¹ V _y ((dy / dθ) ⁻¹ ) ^T (8) Since the variance information obtained by the optical flow detection method is the inverse matrix of the covariance matrix, it is more efficient to perform the calculation using the inverse matrix of V _{y in} the actual calculation. Therefore, the inverse matrix of equation (8) is created and used in actual calculation.

V _h ⁻¹ = (dy / dθ) ^T V _y ⁻¹ (dy / dθ) (9) On the other hand, in the distance information fusion means 30, the motion parameter m of the camera and each inverse depth p are calculated from the ease of calculation. _i
Since the Kalman filter is configured independently with respect to, it is necessary to generate each covariance matrix from the equation (9) for that purpose. Here, the motion parameters m is _{m = [T x T y T} z R x R y R z] T (10). So we take the corresponding submatrix from V _h ^-1 and use it as the inverse matrix of the variance for each variable. That is, with respect to the motion parameter of the camera, the submatrix of V _h ^-1 from 1st row to 1st column to 6th row to 6th column is set as the inverse matrix _Vm ^-1 of the variance of the motion parameter, and i + 6 is obtained for the i-th inverse depth. The diagonal component of row i + 6 column is the reciprocal of variance 1 / σ _pi ² . This operation is equivalent to estimating the variance of each variable by the conditional probability when the correct values of other variables are obtained.

Here, the above description will be summarized. First, by the optical flow calculation means 10,
The movement vector u _i ^* at each point on the image and the inverse matrix V _ui ⁻¹ of its covariance matrix are calculated, and the movement speed k of the camera is obtained from a speedometer or the like. The moving speed k ^* is σ _k ²
It is assumed that the error of variance of is included. At this time, it is assumed that the errors of u _i ^* and k ^* are independent. The observation vector y ^* and the inverse matrix V _y ⁻¹ of the covariance matrix are created from these and used as the input of the distance information calculation means 20. Observation vector y
^* Arranges each element in the order shown by Formula (3). Matrix V
_y ⁻¹ is the matrix V _ui ⁻¹ of 2 rows and 2 columns and its diagonal elements, namely 2 rows of 2i−1 rows and 2i−1 columns to 2i rows and 2i columns.
Arrange so as to be a submatrix of columns, and the final diagonal element 2N +
The element in the 1st row and 2N + 1th column is 1 / σ _k ² . The remaining part is 0. Equation (5) is minimized by using y ^* and _Vy ^-1 thus created, and θ _ML is calculated, and V _h ⁻¹ is obtained by equation (9). Further, referring to the equation (10), θ _ML is decomposed into m _ML , p _MLi , and V _h ⁻¹ is decomposed into V
We get _m ^-1 and 1 / σ _pi ² . The specific forms of V _m ^-1 and 1 / σ _pi ² are calculated from equations (1) and (9) and are as follows.

[0027]

[Equation 3]

In the distance information fusing means 30, as described above, the Kalman filter is constructed independently for the motion parameter m of the camera and each inverse depth p _i . Since θ _ML and the inverse matrix V _h ⁻¹ of its covariance matrix are obtained by the distance information calculating means 20, theoretically, a Kalman filter can be configured with θ as a state vector.
However, when such a Kalman filter is constructed, it is necessary to actually obtain the inverse matrix of V _h ⁻¹ by numerical calculation, and there is a problem in terms of processing amount and stability due to the large number of dimensions. In order to avoid this, θ is decomposed into a motion parameter m of the camera and each inverse depth p _i , and a Kalman filter is independently constructed.

First, the Kalman filter for the motion parameter m of the camera will be described. The translation / rotation speed of the camera is considered to be constant in a minute time, and when the state transition matrix is the unit matrix and the covariance matrix of the system noise is Q _m , the time update algorithm is as follows.

M _{t + 1} ⁻ = m _t ⁺ (12) V _{mt + 1} ⁻ = V _mt ⁺ + Q _m On the other hand, the observation update algorithm is given by the following equation.

M _t ⁺ = m _t ⁻ + K _mt (m _t ^* −m _t ⁻ ) (13) V _mt ⁺ = (I−K _mt ) V _mt ⁻ K _mt = V _mt ⁻ (V _mt ⁻ + V _mt ^{* -1 In} these equations, the vector and matrix subscripts t and t +
1 represents a time step, and + and − on the shoulder represent an observation update value and a time update value, respectively. Also, I is an identity matrix,
m _t ^* and V _mt ^* are the observed value at time t and its variance, and the inverse matrix of m _ML and V _m ⁻¹ calculated by the distance information calculation means 20 is used.

Next, a Kalman filter of inverse depth p _i is constructed. If the translation / rotation speed of the camera at that time is T _t ^* and R _t ^* , the point on the object whose three-dimensional coordinate is X _t ^* at time t is the coordinate value X after a unit time.
_{t + 1} ^* is given by the following equation.

[0033] ^{_{X t + 1 * = -T t}} * -R t * × X t * (14) X t * = (X t Y t Z t) T X t + 1 * = (X t + 1 Y t ₊₁ Z _{t + 1} ) ^T The inverse depth relationship at both times is as follows from the relationship between Z _t and Z _{t + 1 in} the above equation.

P _{it + 1} = p _it / (1 + R _yt x _it −R _xt y _it −T _zt p _it ) (15) Here, when the above formula is differentiated by p _it , the following formula is obtained.

Dp _{it + 1} / dp _it = (T _zt p _it ) / (1 + R _yt x _it −R _xt y _it −T _zt p _it ) ² + 1 / (1 + R _yt x _it −R _xt y _it −T _zt p _it ) (16) From equations (15) and (16), the time update algorithm is obtained.

P _{it + 1} ⁻ = p _it ⁺ / (1 + R _yt x _it −R _xt y _it −T _zt p _it ⁺ ) (17) σ _{pit + 1} ² − = (dp _{it + 1} / dp _it ) ² σ _pit ²⁺ + Q _p R _yt , R _xt , and T _zt use the corresponding elements of the Kalman filter estimate m _t ⁺ of the motion parameters of the camera. Further, Q _p is system noise related to inverse depth.

On the other hand, the observation update algorithm is as follows.

P _it ⁺ = p _it ⁻ + K _pt (p _it ^* −p _it ⁻ ) (18) σ _pit ²⁺ = (1−K _pt ) σ _pit ² −K _pt = σ _pit ^{2 −} / (σ _pit ^2- + σ _pit ^{2 *} ) Here, p _it ^* and σ _pit ^{2 *} correspond to the values p _MLi and σ _pi ² calculated by the distance information calculation means 20 at time t. Now, the Kalman filter for the inverse depth of the point fixed on the object is constructed as above,
The point moves on the image due to the movement of the camera. Therefore, the point corresponding to the inverse depth p _i at one time is at a different position on the image at the next time, so it is necessary to relate them by interpolation, which is achieved as follows. It When applying the observation update algorithm shown in Expression (18), the optical flow estimated using Expression (1) from the estimated value m _t ⁺ of the motion parameter of the camera at that time calculated in Expression (13). u _i
Calculate ⁺ . Predicted value p corresponding to observed value p _it ^*
_{Since it} ⁻ exists at a position separated by −u _i ⁺ on the image before the unit time, the image of p _it ⁻ is interpolated to obtain the value at that position, and It is used as a ^- p _it of. This state is shown in FIG. 2 as a one-dimensional diagram. The interpolation method includes a nearest neighbor interpolation method, a colinear interpolation method, and a cubic convolution interpolation method, and any of them can be used. Details are described in the following references.

Supervision by Mikio Takagi and Haruhisa Shimoda: Image Analysis Handbook, University of Tokyo Press, pp. 411-444 (1
991) Further, there is also a method of calculating the position of the predicted value p _it ^{− from} the estimated value m _t−1 ⁺ of the motion parameter which is a unit time before. For this, m _t-1 ⁺ is substituted into the equation (1) to estimate the optical flow u _i ⁺ at time t−1, and as shown in FIG. 3, p _it ⁻
Move the position of. By interpolating the moved position as a sample point, the value of the position of the observed value p _it ^* is obtained. This method complicates the interpolation problem because the sample points of the image to be interpolated do not exist on a regular grid. Interpolation 1 in this case
The following methods are examples. First, three sample points that are closest to and include the point to be calculated are determined. The position of these sampling points each _{(x a, y a),} (x b, y b), (x c, y c) and the value p _a, p _b, When p _c, the position (x, The value p at the point y) is given by the following equation.

[0040] _{p = (c 1 x + c} 2 y + c 3) / c 4 (19) c 1 = (y a -y b) p a + (y a -y c) p b + (y b -y a) p _{c c 2 = (x b -x} c) p a + (x c -x a) p b + (x a -x b) p c c 3 = (x c y b -x b y c) p a + _{(x a y c -x c y} a) p b + (x b y a -x a y b) p c c 4 = (x b -x c) y a + (x c -x a) y b + Like the (x _a −x _b ) y _c predicted value p _it ⁻ , _it is necessary to interpolate the variance σ _pit ^{2 −} , which is also achieved by the method described above. However, in the case of variance, the square root is calculated, the standard deviation is corrected, interpolation is then performed, and the square is used to restore the variance.

[0041]

By using the method and apparatus according to the present invention, distance information obtained from each frame of a moving image and camera movement are temporally fused to obtain more accurate information stably. be able to.

[Brief description of drawings]

FIG. 1 is a configuration diagram showing an embodiment of a three-dimensional shape restoration device of the present invention.

FIG. 2 is a diagram illustrating an inverse depth interpolation method.

FIG. 3 is a diagram illustrating an inverse depth interpolation method.

[Explanation of symbols]

 10 Optical Flow Calculation Means 20 Distance Information Calculation Means 30 Distance Information Fusion Means 40 Distance Information Kalman Filters 50 Motion Parameter Kalman Filters

Claims (7)

[Claims] 1. An optical flow calculation procedure for calculating an estimated value of optical flow, which is movement information on an image, and its covariance matrix using two frames in a series of images forming a moving image, and the optical flow calculation procedure. Distance information calculation procedure for calculating the distance to the object being imaged using the flow estimation value and the covariance matrix, the camera motion estimation value, and their covariance matrix, and these two procedures for moving images. By applying a Kalman filter independently configured to the estimated value of the distance to a large number of objects and the estimated value of the motion of the camera obtained by successively applying
A three-dimensional shape restoration method comprising a distance information fusion procedure for calculating an accurate estimation value of a distance to an object and camera motion and a covariance matrix thereof. 2. The three-dimensional shape restoration method according to claim 1, wherein in addition to the estimated value of the optical flow and its covariance matrix, the magnitude of the translational motion of the camera and the variance of the noise contained therein are used. A three-dimensional shape restoration method characterized by reflecting on an estimated value of a calculated distance to an object and an estimated value of camera motion. 3. An optical flow calculation means for calculating an estimated value of optical flow, which is movement information on an image, and its covariance matrix using two frames in a series of images forming a moving image, and the optical flow calculation means. Distance information calculation means for calculating the distance to the object being imaged using the flow estimation value and the covariance matrix, the camera movement estimation value, and their covariance matrix, and these two methods for moving images. Estimated values of distances to objects and camera movements with high accuracy by fusing the estimated values of distances to many objects and camera movements and their covariance matrices obtained temporally by continuous application. And a distance information fusion means for calculating a covariance matrix thereof, wherein the distance information fusion means is a distance to an object calculated by the distance information calculation means. A three-dimensional shape restoration device characterized in that an independent Kalman filter is configured for each of the estimated value of the motion vector and the estimated value of the motion of the camera. 4. The three-dimensional shape restoration apparatus according to claim 3, wherein, in addition to the estimated value of the optical flow and its covariance matrix as an input of the distance information calculation means, the magnitude of the translational motion of the camera and the noise contained therein A three-dimensional shape restoration device characterized by employing a distance information fusion means which reflects the estimated value of the distance to the calculated object and the estimated value of the motion of the camera using the variance. 5. An optical flow calculating means for calculating a covariance matrix of an estimated value of an optical flow and an error estimated for the optical flow from a moving image, an estimated value of the optical flow, a covariance matrix, and a magnitude of translational motion of a camera. And distance information calculation means for calculating distance information to an object being imaged, an estimated value of a motion parameter of the camera, and a covariance matrix of an error included in the motion parameter of the camera using the variance of the noise included in the motion parameter A distance information Kalman filter applied to the estimated value of the distance information and the covariance matrix, and a motion parameter Kalman filter applied to the estimated value of the motion parameter of the camera and the covariance matrix, the distance information and the camera Distance to compute a high-precision estimate of the motion parameters of the and covariance matrix of the error 3, characterized in that it comprises a distribution fusion means
Dimensional shape restoration device. 6. An artificial vision system comprising the three-dimensional shape restoration device according to claim 5 and capable of obtaining information on a distance to an object in an environment by analyzing an image obtained by a television camera. Robot to be. 7. A self-propelled vehicle that travels while understanding the road environment by itself, is equipped with the three-dimensional shape restoration device according to claim 5, and analyzes an image obtained by a television camera to determine the distance to an object in the environment. An autonomous vehicle having an artificial vision system capable of obtaining information.