JP2005078554A - Restoration method and device for fish-eye camera motion and three-dimensional information, and recording medium with program for implementing it recorded - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To simultaneously restore motion related to a camera view point and three-dimensional information of the outside world in a manner robust against noise; and to reduce a calculation cost. <P>SOLUTION: This restoration device comprises: a time-series database 1 with omnidirectional images stored; a characteristic point measurement part 2 for measuring a coordinate value of a characteristic point in each image; a measurement matrix conversion part 3 for converting the image coordinate value obtained by the measurement part to a uv image; a measurement matrix input part 4 for providing, as an initial value, a matrix including a uv coordinate value as a matrix element; and a factorization method processing part 5 for restoring motion of a camera and three-dimensional information of the outside world by a factorization method from the input data. The processing part 5 comprises: a three-dimensional information restoration part 5A by means of planar motion; a Z-axis motion restoration part 5B; and a rotary motion restoration part 5C. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention is a vehicle-mounted image or indoor image acquired using a camera equipped with a fisheye lens (fisheye camera), a marine image from a ship, general time-series images such as aerial photography, and a fisheye camera or a fisheye lens. From the time-series images acquired by manual shooting with the camera, the rotational movement around the three axes (XYZ) composed of the roll, pitch, and yaw rotation related to the viewpoint of the fisheye camera, the translational movement in the three axes (XYZ) direction In addition, the present invention relates to a method and apparatus for restoring the three-dimensional shape of the external world reflected in a time-series image, that is, the three-dimensional information constituting the appearance shape of a subject (object).

  In the computer vision field, methods for measuring or acquiring the shape of an object from time-series image data include three-dimensional analysis methods using stereo measurement and epipolar plane analysis. According to this method, it is possible to restore the three-dimensional position information related to the spatial shape or the spatial structure and the motion related to the camera viewpoint from a plurality of time-series images in which the object is photographed. However, it is difficult to seamlessly acquire images in time-series images taken while moving the camera using moving means, etc., due to the shooting environment and the minute movement of the camera. In some cases, it is difficult to accurately restore camera motion and object shape.

  In order to solve such a problem, there is a method of simultaneously and robustly restoring the camera motion and the object shape in the Euclidean space from image coordinate values that can be expressed by an orthogonal coordinate system with the image plane as a plane. For example, there is a method of restoring camera motion and three-dimensional information from image coordinate values obtained by attaching feature points to an acquired image and tracking the feature points in time series (for example, see Non-Patent Documents 1 and 2). ).

In the methods of these documents (hereinafter referred to as documents [1] and [2]), as shown in equation (A1), from time-series xy image coordinate values (left side of equation (A1)) acquired on the image, Decomposed into a matrix related to camera motion (matrix on the left side of the right side, matrix element to be converted to time series) and a matrix related to three-dimensional information (X, Y, Z) (right side matrix on the right side, matrix element invariant to time series) A mathematical method called singular value decomposition is used for this decomposition. The left side of the formula (A1) is called a measurement matrix in documents [1] and [2], the row direction represents the number of feature points, and the column direction represents the number of frames. (X _ij , y _ij ) is the image coordinate value of the j-th feature point in the i-th frame.

  In the method according to the above documents [1] and [2], the left side of the formula (A1) is only two-dimensional information obtained from the image (a two-dimensional coordinate value obtained by subtracting the barycentric coordinate value of each frame in advance). Formula (A1) is based on a projection model in which a matrix calculation (linear calculation) is performed using a matrix on the right side to obtain an image coordinate value on the left side.

  The above method also has drawbacks. In other words, when the image coordinate value obtained on the left side is represented by the right side as in the formula (A1), that is, if it is not a projection model, the camera motion can be applied by applying the literatures [1] and [2]. It is difficult to restore 3D information at the same time. Furthermore, when a camera with a finite field angle is used, it is difficult to restore the minute rotational motion of the camera, and the translation and rotational motion discrimination becomes warm even when the literatures [1] and [2] are applied. In some cases, accurate camera movement of the fisheye camera and accurate three-dimensional information cannot be restored.

  On the other hand, in order to avoid the problem of a finite field angle, imaging with an omnidirectional camera is conceivable as a means for acquiring a landscape of 360 degrees at a time. An omnidirectional camera is often used with its camera optical axis set vertically upward, and is suitable for posture estimation of an omnidirectional camera viewpoint and spatial measurement related to the outside world. Further, by using an omnidirectional camera, it is possible to restore the minute rotational motion, which is excellent in that both the translational motion and the rotational motion are accurately restored (for example, Non-Patent Document 3 and Patent Document 4). reference).

Many omnidirectional cameras use a reflection optical system in which a mirror is deformed into a hyperboloid, or a reflection optical system designed in the shape of a spindle. On the other hand, the fish-eye camera has a feature that the fish-eye lens can acquire an omnidirectional image and the optical projection relationship is linear. For this reason, in the restoration method according to Non-Patent Document 3 and Patent Document 4 (hereinafter referred to as documents [3] and [4]), the use of a fisheye camera reduces the computational cost in restoring the camera motion and three-dimensional information. Excellent in terms of reduction.
C. Tomasi & T. Kanade; “Shapeand Motion from Image Streams Under Orthography: A Factorization Method”, International Journal of Computer Vision, Vol. 9, No. 2.1992 CJPoelman & T. Kanade; “A Paraperspective Factorization Method for Shape and Motion Recovery”, IEEE Trans.Pattern Anal. & Mach.Intell, Vol.19, No.3, pp.206-218.1997 Miyakawa, Wakabayashi, Arikawa; “Motion and shape restoration from omnidirectional images by omnidirectional projection factorization”, Pattern Recognition and Media Understanding (PRMU) Study Group, June, 2002 Miyagawa, Ozawa, Wakabayashi, Arikawa: “Omnidirectional camera viewpoint movement and object shape restoration method and apparatus, omnidirectional camera viewpoint movement and object shape restoration method program, and recording medium on which the program is recorded”, Japanese Patent Application 2002-71623

  In a time-series image acquired in a shooting environment that visualizes a 360-degree landscape at a time, such as an omnidirectional camera, using moving means, the spatial information of objects in the outside world can be obtained by a measurement method that applies the principle of stereo vision. Can be acquired and restored. However, in moving shooting in various shooting environments, the camera moves minutely, so seamless time-series images cannot be acquired easily, and the influence of random noise is large. It is difficult to restore the movement and the shape of the object at the same time and with high accuracy.

  For example, in the case of applying the technique of [3] and [4] to a time-series image acquired by an omnidirectional camera (fisheye camera), as shown in FIG. 10, other than the optical axis rotation (yaw rotation) of the camera. By adding roll rotation, pitch rotation, and translational motion in the Z-axis direction, accurate motion and three-dimensional shape recovery become difficult.

  In other words, as shown in FIG. 10, the camera is often used in a posture in which the optical axis is set vertically upward. Therefore, the main purpose is to restore the planar motion with 3 degrees of freedom composed of yaw rotation and XY translational motion. It was placed.

  However, in a real environment, when roll rotation, pitch rotation, and Z-axis translational motion are included in addition to such planar motion, simple calculation by the camera cannot be used, and calculation for restoring the camera motion and the three-dimensional shape. Becomes complicated, and minute camera posture changes related to the omnidirectional camera as described above are mixed with noise, and cannot be accurately and robustly restored. This will be described using a projection model of a fisheye camera as an example.

  When taking a picture of the outside world using a camera equipped with a fisheye lens and trying to restore the camera viewpoint movement and 3D information from the obtained image sequence, the fisheye camera movement and 3D coordinate values of the object Are related by equation (A2) in the situation of FIG.

(X ′ _ij , y ′ _ij , z ′ _ij ) in Expression (A2) is a unit spherical coordinate value with the viewpoint as the origin. (X _ij , y _ij ) shown in Expression (B1) can be observed as the image coordinate values of the object (feature point) obtained by being projected onto the image of the fisheye camera. f is the focal length of the fisheye lens, D is a constant determined by D = πf / 2R (R is the radius of the image circle), and (Cx, Cy) is the image coordinate value of the projection center C.

On the left side of the formula (A2), there is λ _ij that cannot be obtained directly from the image coordinate values, but a component that depends on the time series (suffix is i) and a component that does not depend on the time series (suffix is j) Since it cannot be separated, it is on the left side. Although the right side is composed of four matrices, the three rotation matrices can be expanded into one matrix, which can be a matrix corresponding to the camera movement on the right side of Equation (A1), but from Equation (A2) itself As can be seen, the information obtained from the image coordinate values only on the left side as in equation (A1), the information on the right side corresponding to camera motion (suffix i), and the information corresponding to three-dimensional information (suffix j It is not easy to perform the linear matrix calculation as shown in the equation (A1) to construct the image coordinate value on the left side because the projection model is too complex. In other words, it is impossible to restore the camera motion and the three-dimensional information by applying the above-mentioned documents [1], [2], [3], and [4] to the formula (A2).

  Furthermore, the application form of an omnidirectional camera has been limited in conventional applications such as monitoring cameras and image measurement. In general, an omnidirectional camera (fisheye camera) is often used in a posture in which the optical axis is vertically upward as shown in FIG. In this figure, as a line segment that is substantially parallel to the optical axis, a vertical line segment in the height direction of the building outdoors (a vertical line segment from the floor to the ceiling in the room) is extracted from the image (see FIG. 11). This line segment has a coordinate value (Dx, Rx) proportional to roll rotation / pitch rotation by utilizing the property that roll rotation and pitch rotation intersect at a certain point (intersection point A) with a certain degree of angular range limitation. Dy) can be obtained.

  However, when a fish-eye camera is used outdoors, as shown in FIG. In this figure, it is difficult to detect the line segment for obtaining the intersection point in the entire image, and if noise is added in the line segment extraction, the detection of the intersection point is also affected. For example, since roadside trees, traffic lights, etc. block the camera's viewpoint outdoors, it is difficult to automatically extract only the straight line that contributes to the desired intersection. In particular, when shooting in urban areas, such a shield cannot be ignored.

  As described above, the problem to be solved by the present invention is to restore the camera motion and the three-dimensional information at the same time and robustly using the documents [3] and [4] in the projection model shown in the formula (A2). 11 and the intersection point A reflecting the roll rotation and the pitch rotation as shown in FIG. 11 is not obtained by an image processing method such as straight line detection, and the time of the feature points set in the image (including the feature points on the shield) It is to obtain a roll rotation and a pitch rotation from only the target movement, and to simultaneously restore the motion related to the camera viewpoint and the three-dimensional information of the outside world and robust to noise.

  Also, from the viewpoint of calculation algorithm, the present invention is to solve the problem of high calculation cost for restoring motion and three-dimensional information from a time-series image.

  In the present invention, on the premise that the documents [1], [2], [3], and [4] are applied, the camera motion is limited in the equation (A2), and imaging is performed with the conditional camera motion. The projected model is assumed. That is, in a camera equipped with a fisheye lens, when roll rotation and pitch rotation are very small in a projection model in which matrix expansion is performed in formula (A2) with i = 1, 2,... That is, when cos ψ≈1, sin ψ≈ψ, cos ω≈1, sin ω≈ω,

It becomes. (U _ij , v _ij ) in the equation (A3) are uv coordinate values defined in the present invention, and u _ij = x _ij / z _ij and v _ij = y _ij / z _ij . In this projection model, as can be seen from the right side of Expression (A3), the matrix on the left side of the right side is information that depends on the time series (information on camera motion), and the matrix on the right side of the right side is invariant that does not depend on the time series. It is information (three-dimensional information of the outside world). That is, the left side of equation (A3) is related to information about camera motion and three-dimensional information in addition to the information obtained from the image. Even when the above camera motion is limited, the projection model is too complicated and the document [1]. , [2], [3], [4] cannot be easily applied to restore camera motion and three-dimensional information at the same time.

In the present invention, roll rotation ω _i , pitch rotation ψ _i , yaw rotation θ _i , XYZ translational motion (Tx _i , Ty _i , Tz _i ); i = 1, 2,. , F (camera parameters restored by the present invention, which may be simply referred to as camera motion), and external three-dimensional information (X _j , Y _j , Z _j ); j = 1, 2,. Hereinafter, the three-dimensional information is a restoration processing method for restoring (three-dimensional coordinate values related to P feature points). However, the documents [1], [2], [3], and [4] are not forcibly applied to the projection model of Expression (A3).

In the present invention, in order to solve the equation (A3), the stepwise restoration is repeatedly performed to approximately restore the camera motion and the three-dimensional information based on the projection model of the equation (A3). Here, since it is difficult to easily restore camera motion and three-dimensional information with the projection model of equation (A3), initialization is performed in equation (A3), and camera motion and three-dimensional information are used using the simplified model. To restore. First, ignoring roll rotation and pitch rotation, ε _ij = 0, i = 1, 2,..., F, j = 1, 2,..., P, and (ζ _i , η _i ) = (0, 0 ), I = 1, 2,... Then

And can be simplified. However, in the left side of the formula (A4), there are Z-axis translational components Tz _i ; i = 1, 2,..., F and three-dimensional information height information Z _j ; include. Accordingly, information corresponding to camera motion and three-dimensional information cannot be obtained from only the measurement matrix composed of the uv coordinate values as in the right side of Expression (A4).

  In the planar motion / three-dimensional information restoration method of the present invention, in Expression (A4), first, a process of restoring motion and three-dimensional information is performed using a projection model that is regarded as having no Z-axis translational motion. In other words, in order to apply documents [3] and [4], the projection model is

And Expression (A5) becomes matrix decomposition data to be processed to restore plane motion and three-dimensional information when initialization of Expression (6) described later is performed in Expression (5b) described later. This matrix data (measurement matrix) is obtained in the measurement matrix measurement step. From the measurement matrix consisting of the uv coordinate value (information obtained from the image) on the left side of equation (A5), based on the projection model of equation (A5), the right side of equation (A5) is applied to documents [3] and [4]. The matrix on the left side of the camera motion (formula (A5)) is the yaw rotation in each frame, that is, the rotation θ around the Z axis (optical axis) in FIG. 10, and the XY translational motion (Tx, Ty) in FIG. This motion with 3 degrees of freedom is referred to as plane motion), and three-dimensional information can be restored.

  Expression (A5) is a projection model used in documents [3] and [4]. In this case, it is assumed that the camera performs a planar motion. That is, the projection model is based on the assumption that the camera does not perform roll rotation, pitch rotation, or Z-axis translation with respect to a certain surface.

  The present invention uses the documents [3] and [4] to restore the plane motion and the three-dimensional information, restore the Z-axis translational motion from the error between the equations (A5) and (A4), The roll rotation and the pitch rotation are restored from the error between the camera motion and the three-dimensional information restored so far and the projection model of the formula (A3). In repeating such processing, the camera motion of the fisheye camera (roll rotation, pitch rotation, FIG. 10) according to the projection model of the formula (A3) using the documents [3] and [4] mainly based on linear operations. Yaw rotation, XYZ translation) and the external three-dimensional information (XYZ) are restored simultaneously and robustly.

  As described above, the present invention is characterized by the following method, apparatus, and recording medium.

(Invention of method)
(1) In a time-series omnidirectional image or a wide-field image acquired by an image input device such as a fish-eye camera, a temporal change amount of an image coordinate value related to a feature point arranged in a target image. A method for restoring the three-dimensional information constituting the movement of the camera viewpoint and the object shape of the outside world,
The value obtained by dividing the orthogonal direction component of the moving direction of the camera by the height direction component in the coordinate value obtained by projecting the three-dimensional coordinate value of the outside world onto the spherical surface with the viewpoint as the center, and the moving direction component of the camera as the height direction component In the feature point coordinate system set as a time-series omnidirectional image or wide-field image, with the values divided by u as the u coordinate and v coordinate, the image coordinate values of the feature points in each image are obtained, and the image coordinates A measurement matrix measurement step for obtaining a uv coordinate value of the feature point from the phase angle and the elevation angle calculated from the values;
Using all the uv coordinate values obtained in the measurement matrix measurement step, matrix decomposition data for restoration processing is generated, the matrix decomposition data is subjected to singular value decomposition, noise is removed, and matrix data representing motion information is obtained. Matrix data representing three-dimensional information is obtained, and in the motion information, a transformation matrix that satisfies the conditions set for defining the motion is obtained, and this transformation matrix is applied to the matrix data representing the motion information to relate to the camera viewpoint. The rotational motion around the Z-axis that becomes the optical axis and the translational motion in the XY-axis directions of the two axes are restored, and the inverse matrix of this transformation matrix is applied to matrix data representing three-dimensional information to form the object shape 3 Planar motion / three-dimensional information restoration step for restoring dimensional information;
The reprojection coordinate value in the uv coordinate system for each feature point is obtained from the plane motion and the three-dimensional information obtained in the plane motion / three-dimensional information restoration step, and each reprojection coordinate value and each of the measurement matrix measurement steps obtained in the measurement matrix measurement step are obtained. The Z axis that restores the translational motion of the camera viewpoint in the Z axis direction using the error between the uv coordinate value of the feature point and the Z coordinate value of each feature point obtained in the plane motion / three-dimensional information restoration step A motion recovery step;
From the matrix decomposition data, components that contribute to rotational motion around axes other than the optical axis are extracted, and those components and motion restored in the plane motion / three-dimensional information restoration step and Z-axis motion restoration step, In addition, using the three-dimensional information, a rotational motion restoring step for restoring rotational motion around another axis other than the optical axis;
It is determined whether or not the error obtained in the Z-axis motion restoration step has converged to a certain value or less. If not, the camera viewpoint XY obtained in the planar motion and three-dimensional information restoration step is determined. Using the translation component, the three-dimensional information, and the rotation angle around the XY axis other than the optical axis, a coefficient ε for deforming the uv coordinate value is obtained, and each feature point obtained in the plane motion and the three-dimensional information restoration step The coefficient δ for scaling the uv coordinate value is obtained using the height information of the image and the position on the Z axis of each viewpoint obtained in the Z axis translational motion restoration step, and the coefficient ε is obtained in the measurement matrix measurement step. Obtained by multiplying the coefficient δ by the value obtained by multiplying each matrix element of the measurement matrix and each coordinate value obtained by subtracting the intersection coordinate value from each matrix element of the measurement matrix obtained in the measurement matrix measurement step. New value combined Generates matrix decomposition data to column element, returns to the information restoration step, iteratively until the error converges below a predetermined value, characterized in that to restore the camera motion and three-dimensional information.

(2) From a temporal change amount of image coordinate values regarding feature points arranged in a target image in a time-series omnidirectional image or a wide-field image acquired by an image input device such as a fisheye camera, A method for restoring the three-dimensional information constituting the movement of the camera viewpoint and the object shape of the outside world,
With respect to the omnidirectional image or wide-field image acquired by the image input device, each image coordinate value is converted into a uv coordinate value by using a phase angle and an elevation angle obtained from each image coordinate value, and each pixel value is converted into its uv coordinate value. A uv measurement matrix measurement step of generating a time series uv image so as to be associated with each other, and measuring a uv coordinate value of a feature point in each image in a feature point coordinate system set in the uv image;
Matrix decomposition data for restoration processing is generated for each uv coordinate value obtained in the uv measurement matrix measurement step, the matrix decomposition data is subjected to singular value decomposition, noise is removed, and matrix data representing motion information is generated. And obtaining matrix data representing 3D information, obtaining a transformation matrix satisfying the conditions set for defining the motion in the motion information, and applying this transformation matrix to the matrix data representing the motion information, 3D information that constitutes the object shape by applying the inverse matrix of this transformation matrix to the matrix data representing the 3D information and the rotational motion around the Z axis that is the optical axis and the translational motion in the X and Y directions of the 2 axes Plane motion and 3D information restoration step to restore
A reprojection value in the uv coordinate system for each feature point is obtained from the plane motion and the three-dimensional information obtained in the plane motion / three-dimensional information restoration step, and each feature obtained in the reprojection value and the uv measurement matrix measurement step is obtained. A Z-axis motion that restores the translational motion of the camera viewpoint in the Z-axis direction using the error between the uv coordinate value of the point and the Z-coordinate value of each feature point obtained in the plane motion / three-dimensional information restoration step. A restore step,
From the matrix decomposition data, components that contribute to rotational motion around other axes other than the optical axis are extracted, and those components and motion restored in the plane motion / three-dimensional information restoration step and Z-axis motion restoration step. And a rotational motion restoring step for restoring rotational motion around another axis other than the optical axis using the three-dimensional information,
It is determined whether or not the error obtained in the Z-axis motion restoration step has converged to a certain value or less. If not, the camera viewpoint obtained in the planar motion and the three-dimensional information restoration step is determined. Using the XY translation component, the three-dimensional information, and the rotation angle around the XY axis other than the optical axis, a coefficient ε for deforming the uv coordinate value is obtained, and each feature obtained in the plane motion and the three-dimensional information restoration step Using the point height information and the position of each viewpoint on the Z-axis obtained in the Z-axis translational motion restoration step, a coefficient δ for scaling the uv coordinate value is obtained, and the coefficient ε is obtained in the measurement matrix measurement step. The value obtained by multiplying each matrix element of the measured matrix and the coordinate value obtained by subtracting the intersection coordinate value from each matrix element of the measurement matrix obtained in the measurement matrix measurement step are multiplied by the coefficient δ, respectively. Combine the obtained values Generates matrix decomposition data as a new matrix elements back to the information restoration step, iteratively until the error converges below a predetermined value, characterized in that to restore the camera motion and three-dimensional information.

  (3) In the measurement matrix measurement step, in a singular value component obtained by singular value decomposition of the measurement matrix, a determination value representing motion is calculated from the component, and if the determination value is less than a certain value, planar motion If the judgment value is greater than a certain value, a process of restoring the three-dimensional information with three degrees of freedom consisting of rotation around the optical axis and movement on a plane perpendicular to the optical axis is performed. It is characterized by restoring the motion of the degree of freedom 6 consisting of rotation and translational motion and the three-dimensional information of the outside world.

  (4) In the uv measurement matrix measurement step, in a singular value component obtained by singular value decomposition of the measurement matrix, a determination value representing motion is calculated from the component, and if the determination value is less than a certain value, a plane If the judgment value is more than a certain value, the processing is to restore the 3D freedom of motion and 3D freedom, consisting of rotation around the optical axis and motion on a plane perpendicular to the optical axis. It is characterized by restoring motion with a degree of freedom of 6 consisting of rotation and translational motion and three-dimensional information of the outside world.

(Invention of the device)
(5) From a temporal change amount of image coordinate values regarding feature points arranged in a target image in a time-series omnidirectional image or a wide-field image acquired by an image input device such as a fisheye camera, A device that restores the movement of the camera viewpoint and the three-dimensional information constituting the object shape of the outside world,
The value obtained by dividing the orthogonal direction component of the moving direction of the camera by the height direction component in the coordinate value obtained by projecting the three-dimensional coordinate value of the outside world onto the spherical surface with the viewpoint as the center, and the moving direction component of the camera as the height direction component In the feature point coordinate system set as a time-series omnidirectional image or wide-field image, with the values divided by u as the u coordinate and v coordinate, the image coordinate values of the feature points in each image are obtained, and the image coordinates A measurement matrix measuring means for obtaining a uv coordinate value of the feature point from a phase angle and an elevation angle calculated from the values;
Using all uv coordinate values obtained by the measurement matrix measurement means, matrix decomposition data for restoration processing is generated, and this matrix decomposition data is subjected to singular value decomposition, noise is removed, and matrix data representing motion information is obtained. Matrix data representing three-dimensional information is obtained, and in the motion information, a transformation matrix that satisfies the conditions set for defining the motion is obtained, and this transformation matrix is applied to the matrix data representing the motion information to relate to the camera viewpoint. The rotational motion around the Z-axis that becomes the optical axis and the translational motion in the XY-axis directions of the two axes are restored, and the inverse matrix of this transformation matrix is applied to matrix data representing three-dimensional information to form the object shape 3 Plane motion / three-dimensional information restoring means for restoring dimensional information;
The reprojection coordinate value in the uv coordinate system for each feature point is obtained from the plane motion obtained by the plane motion / three-dimensional information restoration means and the three-dimensional information, and the reprojection coordinate value and each measurement matrix measurement means obtained by the measurement matrix measurement means are obtained. The Z axis that restores the translational motion of the camera viewpoint in the Z axis direction using the error between the uv coordinate value of the feature point and the Z coordinate value of each feature point obtained by the plane motion / three-dimensional information restoration means Movement recovery means,
From the matrix decomposition data, components that contribute to rotational motion around axes other than the optical axis are extracted, and those components and motion restored by the planar motion / three-dimensional information restoring means and Z-axis motion restoring means, And rotational motion restoring means for restoring rotational motion around another axis other than the optical axis using three-dimensional information,
It is determined whether or not the error obtained by the Z-axis motion restoring means has converged to a certain value or less. If not converged, XY of the camera viewpoint obtained by the planar motion and three-dimensional information restoring means is determined. Using the translation component, the three-dimensional information, and the rotation angle around the XY axis other than the optical axis, a coefficient ε for deforming the uv coordinate value is obtained, and each feature point obtained by the plane motion and the three-dimensional information restoration means The coefficient δ for scaling the uv coordinate value is obtained using the height information of the image and the position on the Z-axis of each viewpoint obtained by the Z-axis translational movement restoring means, and the coefficient ε is obtained by the measurement matrix measuring means. Obtained by multiplying the value obtained by multiplying each matrix element of the measurement matrix by the coefficient δ by each coordinate value obtained by subtracting the intersection coordinate value from each matrix element of the measurement matrix obtained by the measurement matrix measurement means Matrix decomposition data with new matrix elements combined with values It generates data, return to the information restoring means, iteratively until the error converges below a predetermined value, characterized in that to restore the camera motion and three-dimensional information.

(6) From a temporal change amount of image coordinate values regarding feature points arranged in a target image in a time-series omnidirectional image or a wide-field image acquired by an image input device such as a fish-eye camera, A device that restores the movement of the camera viewpoint and the three-dimensional information constituting the object shape of the outside world,
With respect to the omnidirectional image or wide-field image acquired by the image input device, each image coordinate value is converted into a uv coordinate value by using a phase angle and an elevation angle obtained from each image coordinate value, and each pixel value is converted into its uv coordinate value. A uv measurement matrix measuring means for generating a time series uv image so as to be associated with each other and measuring the uv coordinate value of the feature point in each image in the feature point coordinate system set in the uv image;
For each uv coordinate value obtained by the uv measurement matrix measurement means, matrix decomposition data for restoration processing is generated, the matrix decomposition data is subjected to singular value decomposition, noise is removed, and matrix data representing motion information is generated. And obtaining matrix data representing 3D information, obtaining a transformation matrix satisfying the conditions set for defining the motion in the motion information, and applying this transformation matrix to the matrix data representing the motion information, 3D information that constitutes the object shape by applying the inverse matrix of this transformation matrix to the matrix data representing the 3D information and the rotational motion around the Z axis that is the optical axis and the translational motion in the X and Y directions of the 2 axes Plane motion / three-dimensional information restoration means for restoring
A reprojection value in the uv coordinate system for each feature point is obtained from the plane motion obtained by the plane motion / three-dimensional information restoration means and the three-dimensional information, and the reprojection value and each feature obtained by the uv measurement matrix measurement means. Z-axis motion that restores the translational motion of the camera viewpoint in the Z-axis direction using the error between the uv coordinate value of the point and the Z-coordinate value of each feature point obtained by the plane motion / three-dimensional information restoration means Recovery means,
From the matrix decomposition data, components that contribute to rotational motion around axes other than the optical axis are extracted, and those components and motion restored by the planar motion / three-dimensional information restoring means and the Z-axis motion restoring means. , As well as rotational motion restoring means for restoring rotational motion around another axis other than the optical axis using three-dimensional information;
It is determined whether or not the error obtained by the Z-axis motion restoration means has converged to a certain value or less. If not, the camera viewpoint obtained by the planar motion and the three-dimensional information restoration means is determined. Using the XY translation component, the three-dimensional information, and the rotation angle around the XY axis other than the optical axis, a coefficient ε for deforming the uv coordinate value is obtained, and each feature obtained by the plane motion and the three-dimensional information restoration means Using the point height information and the position of each viewpoint on the Z-axis obtained by the Z-axis translational motion restoration means, a coefficient δ for scaling the uv coordinate value is obtained, and the coefficient ε is obtained by the measurement matrix measurement means. The value obtained by multiplying each matrix element of the measured matrix and the coordinate value obtained by subtracting the intersection coordinate value from each matrix element of the measurement matrix obtained by the measurement matrix measuring means are multiplied by the coefficient δ, respectively. A new matrix element combining the obtained values Generate separation data, return to the information restoring means, iteratively until the error converges below a predetermined value, characterized in that to restore the camera motion and three-dimensional information.

  (7) In the measurement matrix measurement means, in a singular value component obtained by singular value decomposition of the measurement matrix, a determination value representing motion is calculated from the component, and if the determination value is less than a certain value, plane motion If the judgment value is greater than a certain value, a process of restoring the three-dimensional information with three degrees of freedom consisting of rotation around the optical axis and movement on a plane perpendicular to the optical axis is performed. It is characterized by restoring the motion of the degree of freedom 6 consisting of rotation and translational motion and the three-dimensional information of the outside world.

  (8) In the uv measurement matrix measurement means, in a singular value component obtained by singular value decomposition of the measurement matrix, a determination value representing motion is calculated from the component, and if the determination value is less than a certain value, a plane If the judgment value is more than a certain value, the processing is to restore the 3D freedom of motion and 3D freedom, consisting of rotation around the optical axis and motion on a plane perpendicular to the optical axis. It is characterized by restoring motion with a degree of freedom of 6 consisting of rotation and translational motion and three-dimensional information of the outside world.

(Invention of recording medium)
(9) A program in which the fisheye camera motion and the three-dimensional information restoration method according to any one of (1) to (4) above are recorded so as to be executable by a computer is recorded.

  According to the present invention, all time-series images acquired using a fish-eye camera (vehicle-mounted images, marine images, aerial images, indoor images taken using moving means, and further equipped with a fish-eye camera or fish-eye lens) It is possible to acquire and restore the object shape related to the object with high accuracy from the time-series images acquired by manual shooting of the camera. In addition, high-accuracy three-dimensional measurement equivalent to conventional surveying techniques is possible.

  In addition, the present invention is such that in-vehicle cameras and the like are robust to noise even in the presence of vibration while traveling, with minute fluctuations in camera posture (rotation around three axes), and accuracy of interpolation of remote sensors such as GPS. It becomes possible to accurately measure the translational motion in three axes.

  Further, in the present invention, most of the calculation processing to be used is composed of linear operations, so that implementation in a computer language is easy.

  In the present invention, in an application such as image measurement using an omnidirectional camera, the posture with the optical axis of the fisheye camera vertically upward is not assumed. That is, it can also be applied to a time-series image acquired when the fisheye camera is used without being directed vertically upward (FIG. 13). The camera motion (rotation around three axes and translational motion in three axes) and 3 Dimensional information can be restored simultaneously and robustly.

(Embodiment 1)
FIG. 1 is a basic configuration diagram relating to the first aspect of the invention. In the present embodiment, a fisheye camera (hereinafter referred to as a camera) using a fisheye lens optically designed by equidistant projection (projection proportional to the incident angle from the optical axis) will be described. The present invention can also be applied to a time-series image acquired by a fish-eye camera imaging device using a fish-eye lens designed with a three-dimensional angle projection or an equal solid angle projection.

In FIG. 10, the point P (X _j , Y _j , Z _j ) in the space to be restored in the present invention and the movement of the camera, that is, roll rotation (ω _i ), pitch rotation (ψ _i ), yaw rotation (Θ _i ) and translational movement T _i (Tx _i , Ty _i , Tz _i ) will be described. FIG. 10 shows the positional relationship between the camera and the object (subject). For convenience of explanation, the axis on which the camera travels is the Y axis. The camera rotates in roll rotation (ω _i ), pitch rotation (ψ _i ), and yaw rotation (θ _i ) while moving in translational motion (Tx _i , Ty _i , Tz _i ). At this time, the camera optical axis is the Z-axis direction.

The position of the projection center C (viewpoint position T _i ) is the center of motion, and the translational motion in the i-th frame is the position of T _i (Tx _i , Ty _i , Tz _i ). Furthermore, the rotation around the traveling axis is roll rotation (rotating around the Y axis), the rotation around the camera optical axis is the yaw rotation (rotating around the Z axis), and around the normal of the surface stretched between the camera optical axis and the traveling axis. The rotation is referred to as pitch rotation (rotation around the X axis).

On the other hand, a point P (X _j , Y _j , Z _j ) in space is projected onto an image coordinate value (x _ij , y _ij ) on the image plane in the image circle. Equation (1) is an optical projection called equidistant projection, and the phase angle ρ _{ij in} FIG. 10 can be obtained from the image coordinate values (x _ij , y _ij ) using Equation (2). In Equation (1), f is the focal length of the fisheye lens, and D is a constant determined by D = πf / (2R). Here, R is the radius of the image circle shown in FIG. In FIG. 10, the image coordinate value of the projection center C on the image plane is (Cx, Cy).

  Next, the basic configuration of FIG. 1 will be described. The present invention includes a time-series database 1 that stores omnidirectional images, a feature point measurement unit 2 that measures the coordinate values of feature points in each image, and a measurement matrix conversion that converts image coordinate values obtained by the measurement unit into uv images. Unit 3, a measurement matrix input unit 4 that gives a matrix having uv coordinate values as matrix elements as initial values, and a factorization method processing unit 5 that restores camera motion and three-dimensional information of the external world from these input data by a factorization method Consists of The factorization processing unit 5 includes a plane motion / three-dimensional information restoration unit 5A, a Z-axis motion restoration unit 5B, and a rotational motion restoration unit 5C.

  In this configuration, the time-series image database 1 may be in the form of using a recording medium such as a hard disk, a RAID device, a CD-ROM, or a remote data resource via a network. Furthermore, FIG. 1a is a processing block diagram in the case where the image input unit 1A is provided for processing in real time, and the present invention does not necessarily require a storage device such as each database unit.

  First, the feature point measurement unit 2 takes out an omnidirectional image obtained by photographing a landscape from the time-series image database 1 and sets a feature point on the omnidirectional image. For this feature point setting, feature points are automatically detected by image processing techniques such as pixel detection (edge detection, etc.), Canny operator, Hough transform, etc., where the gradient value increases due to one-dimensional or two-dimensional changes in shading. To generate. Alternatively, the feature points may be arranged on the initial image by appropriately marking the feature points with an operator intervening.

At this time, the number of feature points to be arranged is P (j = 1, 2,..., P), and the coordinates from the origin in the image coordinate system in FIG. The values (x _1j , y _1j ); j = 1, 2,..., P are recorded. Next, the time-series images following the initial image are read from the database one by one, the feature points arranged in the initial image are changed to a method that focuses on the change in shading between the time-series images, and the pixel patterns centered on the feature points are Tracking between frames (for example, KLT method: Kanade-Lucas-Tomasi method) or a method that takes into account projection distortion peculiar to omnidirectional images, and tracking each time-series image (i. The two-dimensional coordinate values (x _1j , y _1j ) from the origin in the image coordinate system in FIG. 10 are recorded as the image coordinate values of the feature points of the (th image). When the time-series image is continuously read out, when the feature point disappears between images due to disappearance or occlusion, the tracking is stopped and the feature point tracking is ended. When the feature point tracking is completed, the number of time-series images read out is F including the initial image (i = 1, 2,..., F). The feature point measuring unit 2 arranges the amount of change in temporal image coordinate arrangement of feature points in each time-series image in the data format of Expression (3), and holds the measurement matrix [A].

Next, in the measurement matrix conversion unit 3, the image coordinate value (x _ij) of the j-th feature point in the i-th frame is obtained from the measurement matrix [A] having the image coordinate value acquired by the feature point measurement unit 2 as an element. , Y _ij ). Further, the image is temporarily converted into relative image coordinate values with the projection center (Cx, Cy) as the origin, and the phase angle ρ _ij and the elevation angle φ _ij are obtained using equations (1) and (2). From this phase angle and elevation angle, coordinate transformation is performed by transformation of equation (4) to obtain uv coordinate values (u _ij , v _ij ).

  Here, the physical meaning of the uv transformation and the uv coordinate value in the present invention will be described. When the three-dimensional coordinate values (X, Y, Z) of the outside world are projected onto a hemisphere having a radius 1 centered on the viewpoint Ti, (X ′, Y ′, Z ′) = (cos (Φ) cos (ρ), cos (Φ) sin (ρ), sin (Φ)). However, in the spherical coordinate system, the angle from the Z-axis in FIG. 10 is expressed as (sin (Φ) cos (ρ), sin (Φ) sin (ρ), cos (Φ)) with Φ, but in this embodiment, Assuming that the elevation angle when looking up the three-dimensional coordinate value (X, Y, Z) from the viewpoint in FIG. 10 is Φ, (sin (π / 2−Φ) cos (ρ), sin (π / 2−Φ) sin (Ρ), cos (π / 2−Φ)) = (cos (Φ) cos (ρ), cos (Φ) sin (ρ), sin (Φ)).

  The uv coordinate value is obtained from this unit spherical coordinate value (X ′, Y ′, Z ′) by (u, v) = (X ′ / Z ′, Y ′ / Z ′). FIG. 14 shows an example of an image captured by a fisheye camera. FIG. 15 shows an example in which this is converted into uv coordinate values according to FIG. 4 and the uv image shown in FIG. 5 is generated. As shown in FIG. 15, in this coordinate conversion, the measurement error is emphasized and converted to a point that moves away from the v-axis as the feature point (point closer to the bottom of the building) with a smaller elevation angle Φ. That is, when feature points are measured by the same operation, errors that are difficult to understand on the xy image plane can be grasped more accurately on the uv plane. Therefore, if the motion of the camera viewpoint and the three-dimensional coordinate value are obtained so that the emphasized error is minimized, the measurement can be performed with high accuracy. Further, in the uv plane, the roll rotation movement can be converted into the u-direction coordinate movement, the pitch rotation movement can be converted into the v-direction coordinate movement, and the Z-axis translation movement can be simply converted into the scale conversion in the uv coordinate system. Determination of rotation, pitch rotation, and Z-axis translational movement is relatively easy. By utilizing this ease of discrimination, the present invention repeatedly deforms the uv coordinate value using the deformation coefficient, and continues to reconstruct the camera motion and the three-dimensional coordinate value repeatedly, so that the camera motion with higher accuracy can be obtained. It is possible to restore the three-dimensional coordinate values.

  As described above, the omnidirectional image is converted into an image represented by the uv coordinate value in response to the problem that the motion and the three-dimensional shape cannot be accurately restored due to the influence of noise in the omnidirectional image. It has the effect of raising the effect of noise more clearly, and by performing this conversion without being buried in the noise, it is easy to restore the camera movement including the rotational movement of the minute camera and the three-dimensional information. It works suitably.

  The above uv conversion is performed on all frames and all feature points. As a data format after conversion, a matrix [B] in an element format shown in Expression (5a) is held as an initial value of the matrix [B] in the measurement matrix input unit. The matrix format (arrangement of matrix elements) of the matrix [B] is a matrix called a measurement matrix used in documents [3] and [4].

Next, an outline of processing in the factorization method processing unit 5 will be described. This processing unit includes three processing units (planar motion / three-dimensional information restoring unit 5A, Z-axis motion restoring unit 5B, and rotational motion restoring unit 5C). FIG. 2 shows a processing flow in this processing unit. First, the coefficients ε _ij and δ _ij are initialized according to the equation (6), and are initialized as (ζ _i , η _i ) = (0, 0), i = 1, 2,. Next, matrix decomposition data shown in Expression (5b) is generated from the matrix [B] of the initial value, and this is again held as the matrix [B] (held in the measurement matrix input unit 4). The initial value matrix [B] is not overwritten and held until the processing in FIG. 2 is completed). This is the matrix decomposition data in FIG.

  In the plane motion / three-dimensional information restoration step, singular value decomposition performs matrix decomposition into three matrices [U], [W], and [V] shown in Expression (7). Here, [U] is a 2F × P size matrix, [W] is a P × P size diagonal matrix, and [V] is a P × P size matrix.

  Further, in the noise removal of FIG. 2, as shown in the second term of Expression (8), the components of each matrix of rank 4 or higher are deleted.

  At the time of this deletion, the matrix [U] is taken out, the fourth to Pth columns are deleted from the elements of this matrix, the matrix consisting of the remaining components is held, the matrix [W] is taken out, The fourth to Pth rows and the fourth to Pth columns are deleted in the element, the matrix composed of the remaining components is retained, the matrix [V] is extracted, and the fourth to Pth rows are extracted from the elements of this matrix. Delete up to the eye and hold the matrix composed of the remaining components (Equation (9)).

  Next, from the matrix obtained by taking the square of the diagonal elements of the matrix [W] from which the 4th to Pth rows and the 4th to Pth columns are deleted, the matrix [U shown in Equations (10) and (11) is used. '] And matrix [V'] are obtained.

  In calculation of the transformation matrix in the next process of FIG. 2, the held matrix [U ′] is taken out, and the matrix [D] represented by the equation (13) having values obtained by the equations (12a) to (12c) as matrix elements. And the calculation shown in the matrix [D] and the equation (14) is performed to obtain the values a, b, c, d, e, and f. Note that the last matrix on the right side of Expression (14) is a 3F × 1 column vector in which 2 values from the top are 1F, and F values from 0 are subsequently arranged.

  A matrix [C] in which the above values a, b, c, d, e, and f are put in the elements shown in Expression (15) is prepared, and this matrix [C] is subjected to eigenvalue decomposition as shown in Expression (16). . Here, a matrix [C ′] of Expression (17) is generated from the square of the eigenvalue matrix and the eigenvalue matrix, and a matrix [Q] having this matrix element as a component is calculated according to Expression (18).

  Next, based on the projection model formula of Formula (A5), the matrix on the right side of Formula (A5) is changed to the matrix [M ′] obtained by Formula (19), and the matrix on the right side of Formula (A5) is Formula (22) ) To obtain the matrix [S ′] obtained. By this association, the yaw rotation angle θ can be restored from the matrix component of Equation (19) by the calculation of Equation (20), and the XY translational motion can be restored from Equation (21).

On the other hand, the three-dimensional information (X, Y, Z) in the Euclidean space can be restored according to the equation (22) by associating the matrix on the right side of the equation (A5) with the matrix [S ′]. That is, the matrix [M ′] is calculated from the obtained matrix [Q] and the stored matrix [U ′] by the matrix operation of Expression (19). The matrix element (m _ix , n _ix ) or (m _iy , n _iy ) of each frame (i-th frame) is extracted from the matrix [M ′], and the yaw rotation θ _i , i = 1 is obtained using equation (20). , 2, ..., F are restored. Further, the matrix element (T _iu , T _iv ) of each frame (i-th frame) is extracted from the matrix [M ′]. From this (T _iu , T _iv ), XY translational motion (Tx _i , Ty _i ), i = 1, 2,..., F in the Euclidean space in the i-th frame is calculated using equation (21).

On the other hand, in the three-dimensional information restoration, the matrix operation shown in Expression (22) is performed from the matrix [V ′] held in advance and the matrix [Q] obtained by the transformation matrix calculation, and the matrix [S ′ ]. Next, the calculation shown in Expression (23) is performed on the elements of the matrix [S ′], and a matrix having these elements as [P]. The column vector of the matrix [P] is a three-dimensional coordinate straight (X _j , Y _j , Z _j ) in the Euclidean space of the jth feature point.

In the processing so far, since the plane motion and the three-dimensional information are restored using the projection model of the formula (A5), compared with the case of processing with the projection model of the formula (A4), the Z-axis translational motion component ( Tz _i ) is regarded as a noise component at the stage of the processing of equation (9) and deleted, that is, roll rotation, pitch rotation, and Z-axis translation are “not considered to exist”, only plane motion and three-dimensional information. Is restoring.

In the next Z-axis motion restoration step, ε _ij , i = 1, 2,..., F; j = 1, 2,..., P and (ζ _i , η _i ) = (0, 0), i = 1 , 2,..., F, ignoring roll rotation and pitch rotation, and using the projection model of equation (A4), restore the Z-axis translation (Tz _i ) that has not yet been restored. Therefore, the components after rank 4 deleted in equation (9), that is,

In this step, the Z axis translational motion is restored by returning to the projection model of the formula (A4). The principle that this can be achieved will be described below with mathematical formulas.

  If the plane motion and the three-dimensional information are restored using only the equation (A5) (ignoring the Z-axis translational motion), errors of the components shown in the equations (24) and (25) occur. From this error, the translation component of the Z axis in the formula (A4) is extracted.

  In formula (A4), if only the projection model related to the jth feature point of the i frame is extracted,

It is expressed. That is, the component relating to the Z-axis translational motion Tz _i , i = 1, 2,..., F appears as an error on the right side of the equation (A7). This can be further transformed into the following equation.

  When formula (A8) is expanded into a matrix for i = 1, 2,..., F;

It becomes. Therefore, when the plane motion and the three-dimensional information are restored using only the formula (A5), the simultaneous equation shown in the formula (A9) is calculated by the formula (26) using the error shown in the formula (24). Z-axis translational motion can be restored.

Therefore, in the processing flow of the Z-axis motion restoration step, the errors shown in equations (24) and (25) are calculated for all frames and all feature points (reprojection error in the uv coordinate system). Using these reprojection errors and the previously obtained Z coordinate values (Z _j , j = 1, 2,..., P) of each feature point, the calculation of Expression (26) is performed, and each frame (i-th frame) is calculated. ), The Z axis translational motion Tz _i , i = 1, 2,..., F can be restored.

  At the stage of the Z-axis motion restoration step, roll rotation and pitch rotation have not yet been restored. That is, the motion and the three-dimensional information are not restored based on the projection model of the formula (A3), and the motion and the three-dimensional information are restored according to the projection model of the formula (A4).

  In the next rotational motion restoration step, in the formula (A3), the camera motion restored so far, the matrix component consisting of three-dimensional information, and the matrix component associated with the roll rotation and pitch rotation that have not been restored yet are separated. Only components related to roll rotation and pitch rotation are extracted, and roll rotation and pitch rotation are restored from these components. The principle for realizing this will be described below with mathematical expressions.

  For simplicity of explanation, if only the projection model related to the jth feature point of the i frame is extracted from the equation (A3),

It becomes. Here, when transforming into an equation with unknown roll rotation and pitch rotation that have not yet been restored,

Is obtained. here,

And expand to all feature points (j = 1, 2,..., P),

It becomes. The errors ΔU _ij and ΔV _ij shown in the equation (A13) are errors generated when the projection model of the equation (A4) is restored, and these errors are extracted by the calculation of the equation (A13) and are simultaneous for all feature points. The roll rotation angle and the pitch rotation can be obtained by solving the equation (formula (A14)) according to formulas (27) to (32). The processing of Expression (32) is performed for each frame, and the roll rotation angles ω _i (i = 1, 2,..., F) and pitch rotations ψ _i (i = 1, 2,..., F) in all time series. Can be restored.

Therefore, in the rotational motion restoration step, the error shown in Equation (27) is calculated from the matrix decomposition data in the data format shown in Equation (5a), the coefficient δ _ij , the planar motion restored as described above, and the three-dimensional information. . Further, the values of the equations (28) to (31) are calculated, and from these values and the error shown in the equation (27), the roll rotation (ω _i ) and pitch rotation (ψ _i ) restore. This is obtained over all frames (j = 1, 2,..., F). At this time, at the same time, (ζ _i , η _i ), i = 1, 2,..., F are obtained for each frame by the calculation of equation (33), and the previous (ζ _i , η _i ), i = 1, 2, ..., F are updated.

  According to the above processing flow, roll rotation, pitch rotation, yaw rotation, XYZ translation, and 3D information can be restored. However, stepwise processing (planar motion / 3D information) is performed on the projection model of equation (A3). These information can be restored by performing a restoration step, a Z-axis motion restoration step, and a rotational motion restoration step. That is, plane motion and three-dimensional information restoration based on equation (A5), Z-axis translation restoration based on equation (A4), roll rotation based on equation (A3), and pitch rotation restoration are sequentially performed. .

Here, the uv coordinate value observed from the image, the camera motion and the three-dimensional information restored by the processing so far, and the deformation coefficients (ε _ij , δ _ij ) at this stage are used to express the model adaptability. When the equation (34) is calculated to obtain the error shown in the equation (35), the error is large. Because, in the initial stage (the previous stage), in equation (A3), ε _ij = 0 and (ζ _i , η _i ) = (0, 0); i = 1, 2,. This is because the processing so far has been executed. That is, the plane motion / three-dimensional information restoration unit approximately approximates the uv coordinate value (in the actual omnidirectional image) projected by the camera motion including roll rotation, pitch rotation, yaw rotation, and XYZ translational motion. The plane motion and the three-dimensional information are restored by regarding the uv coordinate value projected by the camera motion of the plane motion.

Therefore, in the processing flow following the rotational motion restoration step, ε _ij = 0 and (ζ _i , η _i ) = (0, 0); i = 1, 2,. The same value is updated using the information obtained in the previous processing, and the same processing is repeated. That is, ε _ij is _expressed by the roll rotation angle ω _i , pitch rotation ψ _i , three-dimensional information (X _j , Y _j , Z _j ), and XY translational motion (Tx _i , Ty _i ) restored so far. Substitute and update in the first expression of 36). Similarly, δ _ij is updated by substituting the Z-axis translational movement Tz _i and the height Z _j of the three-dimensional information into the second expression of Expression (36). Further, a new matrix [B] to be matrix decomposition data is prepared according to the equation (5b) using the updated deformation coefficient (ε _ij , δ _ij ). Thereafter, as in the past, from the newly obtained measurement matrix [B], yaw rotation, XY translation, and 3D information are restored in the plane motion / 3D information restoration step, and then the Z axis In the motion restoring step, the Z-axis translational motion is restored, and in the rotational motion restoring step, roll rotation and pitch rotation are restored.

When all camera movements and 3D information can be restored, the error obtained in Equation (34) and Equation (35) at this stage is obtained, and this error changes compared to the error in the previous stage. When it is determined that there is no error (the error is constant) or less than the allowable value, the processing is stopped, and the camera motion and three-dimensional information at this stage are output. Otherwise, the deformation coefficient (ε _ij , δ _ij ) is updated from the camera motion and 3D information obtained in the previous processing, and the plane motion / 3D information restoration step, Z-axis motion restoration step, rotational motion Repeat the process in the restore step. By performing such an iterative process, even if the projection model shown in the formula (A3) is processed stepwise in the plane motion / three-dimensional information restoration step, the Z-axis motion restoration step, and the rotational motion restoration step, an error is generated. Evenly distributed, there is no bias in the accuracy of camera motion and 3D information, and noise can be restored robustly. In other words, with respect to the projection model formula (A3) of the fisheye camera, the present invention can simultaneously restore camera motion and three-dimensional information with high accuracy and robustness against noise.

  As described above, according to the embodiment of the present invention, from the temporal movement of the feature points in the image series, the movement of the camera viewpoint, that is, the rotational movement around the three axes and the translational movement in the three axial directions, and the three-dimensional that constitutes the object shape Information can be restored.

(Embodiment 2)
FIG. 3 is a basic configuration diagram relating to claim 2 and the like. This embodiment is different from the first embodiment in that a deformed image (uv image) is generated by the image conversion processing unit 6 with respect to an image extracted from the time-series image database, and this processing unit will be described.

  The present embodiment includes a time-series database 1 in which omnidirectional images are stored, an image conversion processing unit 6 that converts each image into a uv image, a feature point measurement unit 2 that measures coordinate values of feature points in each image, and uv coordinate values. A measurement matrix input unit 4 that gives a matrix having a matrix element as an initial value, and a factorization processing unit 5 that restores the three-dimensional information of the camera motion and the external world from these input data by a factorization method.

  In this configuration, the time-series image database 1 may be in the form of using a recording medium such as a hard disk, a RAID device, a CD-ROM, or a remote data resource via a network. Furthermore, FIG. 3A is a processing configuration diagram in the case of processing in real time, and the present invention does not necessarily require a storage device such as each database unit.

  In the present embodiment, an uv image is generated in the image conversion processing unit in FIG. 3 for an image extracted from the time-series image database. In order to generate a uv image, an uv coordinate value is designated, an xy coordinate value corresponding to this is calculated, and a pixel value of the coordinate value is extracted to be a uv pixel (see FIG. 5).

This processing flow is shown in FIG. First, weighting per unit pixel in the u direction and the v direction in the uv image is performed. Assume that the image size of the uv image is U × V pixels, and the elevation angle φ is determined to be in a range of φ _min ≦ φ ≦ 90 degrees. Thereby, the weighting per unit pixel in the u direction and the v direction is determined according to the equation (37). Further, the uv coordinate value range of the uv image is set from the center of the image to the origin (0, 0), from −U / 2 to + U / 2 in the u direction, and from −V / 2 to + V / 2 in the v direction. Integer m, n representing each coordinate value in u direction and v direction (-M ≦ m ≦ M, −N ≦ n ≦ N, M: integer, N: integer, M and N are the same, that is, even if the uv image is square. The uv coordinate value is obtained according to the equation (37). While changing the integers m and n in the range of −M ≦ m ≦ M and −N ≦ n ≦ N, the phase angle ρ and the elevation angle φ corresponding to the (u, v) coordinate value are obtained according to the equation (38). Next, an image coordinate value is calculated by Expression (39). A value obtained by rounding the image coordinate value to an integer is (x, y) (Formula 40).

  Next, a pixel value corresponding to the image coordinate value (x, y) is extracted, and the pixel value is embedded in a coordinate value corresponding to the uv coordinate value on the uv image. By performing this operation for all uv coordinate values, a uv image is generated. Further, all the time series images to be processed are sequentially converted to generate a time series uv image. From this point onward, according to the same processing procedure as in the first embodiment, from the temporal movement of the feature points in the image sequence, the movement of the camera viewpoint, that is, the rotational movement around the three axes and the translational movement in the three axial directions, and the object shape are obtained. The three-dimensional information to be configured can be restored.

(Embodiment 3)
FIG. 6 is a basic configuration diagram relating to the third aspect of the present invention. In the present embodiment, in contrast to the first embodiment, the image coordinate value of the feature point obtained by the feature point measurement unit 2 is converted into the uv coordinate value in the measurement matrix conversion unit 3 with respect to the image extracted from the time series image database. In this case, since the restoration process determination unit 7 determines the process for restoring the camera motion and the three-dimensional information, only this point will be described.

  In this embodiment, the time-series image database 1 may use either a hard disk, a RAID device, a CD-ROM or other recording medium, or a remote data resource via a network. Furthermore, FIG. 6A is a processing configuration diagram in the case of processing in real time, and the present invention does not necessarily require a storage device such as each database unit.

  The image coordinate value of the feature point obtained by the feature point measurement unit 2 in FIG. 6 is converted into the uv coordinate value in the measurement matrix conversion unit 3 in FIG. 6 to hold the measurement matrix [B]. This measurement matrix [B] is in the form of equation (5a).

Next, processing in the restoration processing determination unit 7 in FIG. 6 is performed. The processing flow in this processing unit is shown in FIG. Singular value decomposition is performed on this matrix [B] as shown in Equation (8) to obtain a singular value matrix [W]. The singular value matrix [W] is a diagonal element, and the singular values that are the respective elements are arranged in ascending / descending order and are all positive real numbers. The singular values of the matrix element W ₃₃ and the matrix element W ₄₄ are extracted from the singular values. In rank detection, the calculation shown in Formula (41a) or Formula (41b) is performed to obtain the determination amount Ew.

Determining the amount or Ew is less than a certain tolerance value [delta] _W is or determines whether determination amount Ew allowable value [delta] _W or more. This allowable value δ _W is a specific constant value, and the operator can also set the value sequentially. If less than the allowable value [delta] _W there is determination amount Ew is camera motion is determined to have a planar motion proceeds to the process A, the case of more than the allowable value [delta] _W there is determination amount Ew, the camera motion is generally It is determined that the movement (other than the planar movement) has been performed, and the process proceeds to process B. The process B performs the process of the first embodiment, and from the temporal movement of the feature points in the image sequence, the motion of the camera viewpoint, that is, the rotational motion around the three axes, the translational motion in the three axial directions, and the object shape. This is a process for restoring the three-dimensional information to be configured. Therefore, hereinafter, the process A will be described.

  The process in the factorization method processing unit in FIG. 6 (substantially the same as the processing content of the plane motion / three-dimensional information restoration unit in the first embodiment) is referred to as processing A, and this processing flow is shown in FIG. Each matrix data obtained by matrix decomposition into the three matrices [U], [W], and [V] of Expression (7) by singular value decomposition performed in the restoration processing determination unit in FIG. 6 is input. Here, [U] is a 2F × P size matrix, [W] is a P × P size diagonal matrix, and [V] is a P × P size matrix.

  First, in the noise removal of FIG. 8, the components of each matrix of rank 4 or higher are deleted as shown in the second term of equation (8). At the time of this deletion, the matrix [U] is taken out, the fourth to Pth columns are deleted from the elements of this matrix, the matrix consisting of the remaining components is held, the matrix [W] is taken out, The fourth to Pth rows and the fourth to Pth columns are deleted in the element, the matrix composed of the remaining components is retained, the matrix [V] is extracted, and the fourth to Pth rows are extracted from the elements of this matrix. Delete up to the eye and hold the matrix composed of the remaining components (Equation (9)).

  Next, from the matrix obtained by taking the square of the diagonal elements of the matrix [W] from which the 4th to Pth rows and the 4th to Pth columns are deleted, the matrix [U shown in Equations (10) and (11) is used. '] And matrix [V'] are obtained.

  In calculation of the transformation matrix in the next process of FIG. 2, the held matrix [U ′] is taken out, and the matrix [D] represented by the equation (13) having values obtained by the equations (12a) to (12c) as matrix elements. And the calculation shown in the matrix [D] and the equation (14) is performed to obtain the values a, b, c, d, e, and f. Note that the last matrix on the right side of Expression (14) is a 3F × 1 column vector in which 2 values from the top are 1F, and F values from 0 are subsequently arranged. A matrix [C] in which the values a, b, c, d, e, and f are put in the elements shown in Expression (15) is prepared, and this matrix [C] is subjected to eigenvalue decomposition as shown in Expression (16). Here, a matrix [C ′] of Expression (17) is generated from the square of the eigenvalue matrix and the eigenvalue matrix, and a matrix [Q] having this matrix element as a component is calculated according to Expression (18).

  Furthermore, let the matrix on the left side of the right side of Expression (A5) be the matrix [M] obtained by Expression (19), and the matrix on the right side of Expression (A5) be the matrix [S ′] obtained by Expression (22). By this association, the yaw rotation angle θ can be restored from the matrix component of Equation (19) by the calculation of Equation (20), and the XY translational motion can be restored from Equation (21).

On the other hand, the three-dimensional information (x, y, z) in the Euclidean space can be restored according to the equation (22) by associating the matrix on the right side of the equation (A5) with the matrix [S ′]. That is, the matrix [M ′] is calculated from the obtained matrix [Q] and the stored matrix [U ′] by the matrix operation of Expression (19). The matrix element (m _ix , n _ix ) or (m _iy , n _iy ) of each frame (i-th frame) is extracted from the matrix [M ′], and the yaw rotation θ _i , i = 1 is obtained using equation (20). , 2, ..., F are restored. Further, the matrix element (T _iu , T _iv ) of each frame (i-th frame) is extracted from the matrix [M ′]. From this (T _iu , T _iv ), XY translational motion (Tx _i , Ty _i ), i = 1, 2,..., F in the Euclidean space in the i-th frame is calculated using equation (21).

On the other hand, in the three-dimensional information restoration, the matrix operation shown in Expression (22) is performed from the previously held matrix [V ′] and the matrix [Q] obtained by the transformation matrix calculation, and the matrix [S ′ ]. Next, the calculation shown in Expression (23) is performed on the elements of the matrix [S ′], and the matrix having these elements as [P]. The column vector of the matrix [P] is a three-dimensional coordinate value (X _j , Y _j , Z _j ) in the Euclidean space of the jth feature point.

  Since the process A is performed by determining the planar motion in the restoration processing establishment unit in FIG. 6, from the temporal movement of the feature points in the image sequence, the camera viewpoint motion, that is, rotation around the optical axis and biaxial direction The translational motion of (XY) and the three-dimensional information constituting the object shape are restored.

  When the restoration processing determination unit in FIG. 6 determines that the motion is a general motion, the camera viewpoint motion, that is, the rotational motion around the three axes, from the temporal motion of the feature points in the image series, as in the first embodiment. It is possible to restore three-dimensional translational motion and three-dimensional information constituting the object shape.

(Embodiment 4)
FIG. 9 is a basic structural view relating to the invention of claim 4. In the present embodiment, a restoration process is performed on the uv image input from the image conversion processing unit 6 by using matrix decomposition data composed of uv coordinate values of the feature points obtained by the feature point measurement unit 2 in the uv image input from the image conversion processing unit 6. Since the determination unit 7 differs in the point of determining the camera motion and the process of restoring the three-dimensional information, only this point will be described.

  In this embodiment, the time-series image database 1 may use either a hard disk, a RAID device, a CD-ROM or other recording medium, or a remote data resource via a network. Further, FIG. 9A is a processing configuration diagram in the case of processing in real time, and the present invention does not necessarily require a storage device such as each database unit.

A measurement matrix [B] having the uv coordinate values of the feature points obtained by the feature point measurement unit in FIG. 9 as matrix elements is held. This measurement matrix [B] is in the form of equation (5a). Next, processing in the restoration processing determination unit in FIG. 6 is performed. The processing flow in this processing unit is shown in FIG. Singular value decomposition is performed on this matrix [B] as shown in Equation (8) to obtain a singular value matrix [W]. The singular value matrix [W] is a diagonal matrix, and the singular values that are each element thereof are arranged in ascending / descending order and are all positive real numbers. Singular values of the matrix element W ₃₃ and the matrix element W ₄₄ are extracted from the singular values. In rank detection, the calculation shown in Formula (41a) or Formula (41b) is performed to obtain the determination amount E _W. Determining the amount of E or _W is less than a certain tolerance value [delta] _W is or determines whether determination amount E _W allowable value [delta] _W or more. This allowable value δ _W is a specific constant value, and the operator can also set the value sequentially. If the determination amount E _W is less than a certain allowable value δ _W , it is determined that the camera motion has moved in a plane, and the process proceeds to processing A. If the determination amount E _W is greater than the certain allowable value δ _W , the camera motion Is determined to have performed general motion (other than planar motion), and the process proceeds to process B.

  The process B is the process of the first embodiment. From the temporal movement of the feature points in the image series, the camera view, the movement of the points, that is, the rotation around the three axes and the translational movement in the three axes, and the object shape Is a process for restoring the three-dimensional information constituting the. Note that the processing A is the same as the processing A of the third embodiment, and thus description thereof is omitted.

A basic configuration diagram of an image storage type according to claim 1 or the like. A basic configuration diagram of a real-time processing type according to claim 1 or the like. The processing flowchart of Claim 1 grade | etc.,. The image storage type copy block diagram of Claim 2 grade | etc.,. A basic configuration diagram of a real-time processing type according to claim 2 or the like. The process flow of uv image conversion. The figure explaining uv image generation. 4. A basic configuration diagram of an image storage type according to claim 3. A basic configuration diagram of a real-time processing type according to claim 3 or the like. The processing flow in a restoration process determination part. Process flow of process A. 5. A basic configuration diagram of an image storage type according to claim 4. A basic configuration diagram of a real-time processing type according to claim 4 or the like. The figure which shows the omnidirectional camera, the spatial information to restore | restore, and a camera coordinate system. The figure explaining detecting a perpendicular line segment from an image and calculating intersection A. The figure which shows that the intersection A is not calculated | required by the vertical line segment detected from the image by obstructions, such as a roadside tree. Usage mode in which the camera optical axis is not oriented vertically when restoring camera motion and 3D information. The figure which shows the edge line accompanying the building in the outdoor image | video which the fisheye camera imaged. The figure which shows the result of having converted the edge of FIG. 14 as a uv image.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Time-sequential image database 1A Image input part 2 Feature point measurement part 3 Measurement matrix conversion part 4 Measurement matrix input part 5 Factor decomposition method processing part 5A Plane motion / three-dimensional information restoration part 5B Z-axis movement restoration part 5C Rotation movement restoration part 6 Image conversion processing unit 7 Restoration processing determination unit

Claims (9)

From the temporal change in image coordinate values for feature points placed in the target image in a time-series omnidirectional image or wide-field image acquired by an image input device such as a fisheye camera, the time-series camera viewpoint A method for restoring motion and 3D information constituting the object shape of the outside world,
The value obtained by dividing the orthogonal direction component of the moving direction of the camera by the height direction component in the coordinate value obtained by projecting the three-dimensional coordinate value of the outside world onto the spherical surface with the viewpoint as the center, and the moving direction component of the camera as the height direction component In the feature point coordinate system set as a time-series omnidirectional image or wide-field image, with the values divided by u as the u coordinate and v coordinate, the image coordinate values of the feature points in each image are obtained, and the image coordinates A measurement matrix measurement step for obtaining a uv coordinate value of the feature point from the phase angle and the elevation angle calculated from the values;
Using all the uv coordinate values obtained in the measurement matrix measurement step, matrix decomposition data for restoration processing is generated, the matrix decomposition data is subjected to singular value decomposition, noise is removed, and matrix data representing motion information is obtained. Matrix data representing three-dimensional information is obtained, and in the motion information, a transformation matrix that satisfies the conditions set for defining the motion is obtained, and this transformation matrix is applied to the matrix data representing the motion information to relate to the camera viewpoint. The rotational motion around the Z-axis that becomes the optical axis and the translational motion in the XY-axis directions of the two axes are restored, and the inverse matrix of this transformation matrix is applied to matrix data representing three-dimensional information to form the object shape 3 Planar motion / three-dimensional information restoration step for restoring dimensional information;
The reprojection coordinate value in the uv coordinate system for each feature point is obtained from the plane motion and the three-dimensional information obtained in the plane motion / three-dimensional information restoration step, and each reprojection coordinate value and each of the measurement matrix measurement steps obtained in the measurement matrix measurement step are obtained. The Z axis that restores the translational motion of the camera viewpoint in the Z axis direction using the error between the uv coordinate value of the feature point and the Z coordinate value of each feature point obtained in the plane motion / three-dimensional information restoration step A motion recovery step;
From the matrix decomposition data, components that contribute to rotational motion around axes other than the optical axis are extracted, and those components and motion restored in the plane motion / three-dimensional information restoration step and Z-axis motion restoration step, In addition, using the three-dimensional information, a rotational motion restoring step for restoring rotational motion around another axis other than the optical axis;
It is determined whether or not the error obtained in the Z-axis motion restoration step has converged to a certain value or less. If not, the camera viewpoint XY obtained in the planar motion and three-dimensional information restoration step is determined. Using the translation component, the three-dimensional information, and the rotation angle around the XY axis other than the optical axis, a coefficient ε for deforming the uv coordinate value is obtained, and each feature point obtained in the plane motion and the three-dimensional information restoration step The coefficient δ for scaling the uv coordinate value is obtained using the height information of the image and the position on the Z axis of each viewpoint obtained in the Z axis translational motion restoration step, and the coefficient ε is obtained in the measurement matrix measurement step. Obtained by multiplying the coefficient δ by the value obtained by multiplying each matrix element of the measurement matrix and each coordinate value obtained by subtracting the intersection coordinate value from each matrix element of the measurement matrix obtained in the measurement matrix measurement step. New value combined Generating a matrix decomposition data as column elements, returning to the information restoration step, and repeating until the error converges to a certain value or less to restore the camera movement and the three-dimensional information; A method for restoring three-dimensional information. From the temporal change in image coordinate values for feature points placed in the target image in a time-series omnidirectional image or wide-field image acquired by an image input device such as a fisheye camera, the time-series camera viewpoint A method for restoring motion and 3D information constituting the object shape of the outside world,
With respect to the omnidirectional image or wide-field image acquired by the image input device, each image coordinate value is converted into a uv coordinate value by using a phase angle and an elevation angle obtained from each image coordinate value, and each pixel value is converted into its uv coordinate value. A uv measurement matrix measurement step of generating a time series uv image so as to be associated with each other, and measuring a uv coordinate value of a feature point in each image in a feature point coordinate system set in the uv image;
Matrix decomposition data for restoration processing is generated for each uv coordinate value obtained in the uv measurement matrix measurement step, the matrix decomposition data is subjected to singular value decomposition, noise is removed, and matrix data representing motion information is generated. And obtaining matrix data representing 3D information, obtaining a transformation matrix satisfying the conditions set for defining the motion in the motion information, and applying this transformation matrix to the matrix data representing the motion information, 3D information that constitutes the object shape by applying the inverse matrix of this transformation matrix to the matrix data representing the 3D information and the rotational motion around the Z axis that is the optical axis and the translational motion in the X and Y directions of the 2 axes Plane motion and 3D information restoration step to restore
A reprojection value in the uv coordinate system for each feature point is obtained from the plane motion and the three-dimensional information obtained in the plane motion / three-dimensional information restoration step, and each feature obtained in the reprojection value and the uv measurement matrix measurement step is obtained. A Z-axis motion that restores the translational motion of the camera viewpoint in the Z-axis direction using the error between the uv coordinate value of the point and the Z-coordinate value of each feature point obtained in the plane motion / three-dimensional information restoration step. A restore step,
From the matrix decomposition data, components that contribute to rotational motion around other axes other than the optical axis are extracted, and those components and motion restored in the plane motion / three-dimensional information restoration step and Z-axis motion restoration step. And a rotational motion restoring step for restoring rotational motion around another axis other than the optical axis using the three-dimensional information,
It is determined whether or not the error obtained in the Z-axis motion restoration step has converged to a certain value or less. If not, the camera viewpoint obtained in the planar motion and the three-dimensional information restoration step is determined. Using the XY translation component, the three-dimensional information, and the rotation angle around the XY axis other than the optical axis, a coefficient ε for deforming the uv coordinate value is obtained, and each feature obtained in the plane motion and the three-dimensional information restoration step Using the point height information and the position of each viewpoint on the Z-axis obtained in the Z-axis translational motion restoration step, a coefficient δ for scaling the uv coordinate value is obtained, and the coefficient ε is obtained in the measurement matrix measurement step. The value obtained by multiplying each matrix element of the measured matrix and the coordinate value obtained by subtracting the intersection coordinate value from each matrix element of the measurement matrix obtained in the measurement matrix measurement step are multiplied by the coefficient δ, respectively. Combine the obtained values A fish-eye camera which generates matrix decomposition data as new matrix elements, returns to the information restoration step, and repeats until the error converges to a certain value or less to restore camera motion and three-dimensional information How to restore motion and 3D information.   In the measurement matrix measurement step, in the singular value component obtained by singular value decomposition of the measurement matrix, a determination value representing motion is calculated from the component, and if the determination value is less than a certain value, it is regarded as planar motion. Perform 3 degrees of freedom of plane motion consisting of rotation around the optical axis and motion on a plane perpendicular to the optical axis, and processing to restore 3D information. If the judgment value is above a certain value, 3. The method of restoring fisheye camera motion and 3D information according to claim 1 or 2, wherein the motion of 6 degrees of freedom consisting of translational motion and the external 3D information are restored.   In the uv measurement matrix measurement step, in a singular value component obtained by singular value decomposition of the measurement matrix, a determination value representing motion is calculated from the component, and if the determination value is less than a certain value, it is regarded as plane motion. If the judgment value is greater than a certain value, it performs a rotation about the optical axis and a plane motion with 3 degrees of freedom consisting of a rotation around the optical axis and a motion on a plane perpendicular to the optical axis. 3. The method of restoring fisheye camera motion and three-dimensional information according to claim 1 or 2, wherein the motion with a degree of freedom of 6 comprising translational motion and the three-dimensional information of the outside world are restored. From the temporal change in image coordinate values for feature points placed in the target image in a time-series omnidirectional image or wide-field image acquired by an image input device such as a fisheye camera, the time-series camera viewpoint A device for restoring motion and three-dimensional information constituting the object shape of the outside world,
The value obtained by dividing the orthogonal direction component of the moving direction of the camera by the height direction component in the coordinate value obtained by projecting the three-dimensional coordinate value of the outside world onto the spherical surface with the viewpoint as the center, and the moving direction component of the camera as the height direction component In the feature point coordinate system set as a time-series omnidirectional image or wide-field image, with the values divided by u as the u coordinate and v coordinate, the image coordinate values of the feature points in each image are obtained, and the image coordinates A measurement matrix measuring means for obtaining a uv coordinate value of the feature point from a phase angle and an elevation angle calculated from the values;
Using all uv coordinate values obtained by the measurement matrix measurement means, matrix decomposition data for restoration processing is generated, and this matrix decomposition data is subjected to singular value decomposition, noise is removed, and matrix data representing motion information is obtained. Matrix data representing three-dimensional information is obtained, and in the motion information, a transformation matrix that satisfies the conditions set for defining the motion is obtained, and this transformation matrix is applied to the matrix data representing the motion information to relate to the camera viewpoint. The rotational motion around the Z-axis that becomes the optical axis and the translational motion in the XY-axis directions of the two axes are restored, and the inverse matrix of this transformation matrix is applied to matrix data representing three-dimensional information to form the object shape 3 Plane motion / three-dimensional information restoring means for restoring dimensional information;
The reprojection coordinate value in the uv coordinate system for each feature point is obtained from the plane motion obtained by the plane motion / three-dimensional information restoration means and the three-dimensional information, and the reprojection coordinate value and each measurement matrix measurement means obtained by the measurement matrix measurement means are obtained. The Z axis that restores the translational motion of the camera viewpoint in the Z axis direction using the error between the uv coordinate value of the feature point and the Z coordinate value of each feature point obtained by the plane motion / three-dimensional information restoration means Motion recovery means,
From the matrix decomposition data, components that contribute to rotational motion around axes other than the optical axis are extracted, and those components and motion restored by the planar motion / three-dimensional information restoring means and Z-axis motion restoring means, And rotational motion restoring means for restoring rotational motion around another axis other than the optical axis using three-dimensional information,
It is determined whether or not the error obtained by the Z-axis motion restoration means has converged to a certain value or less. If not, the camera viewpoint XY obtained by the planar motion and the three-dimensional information restoration means is determined. Using the translation component, the three-dimensional information, and the rotation angle around the XY axis other than the optical axis, a coefficient ε for deforming the uv coordinate value is obtained, and each feature point obtained by the plane motion and the three-dimensional information restoration means The coefficient δ for scaling the uv coordinate value is obtained using the height information of the image and the position on the Z-axis of each viewpoint obtained by the Z-axis translational movement restoring means, and the coefficient ε is obtained by the measurement matrix measuring means. Obtained by multiplying the value obtained by multiplying each matrix element of the measurement matrix by the coefficient δ by each coordinate value obtained by subtracting the intersection coordinate value from each matrix element of the measurement matrix obtained by the measurement matrix measurement means Matrix decomposition data with new matrix elements combined with values A reconstructing device for fisheye camera motion and three-dimensional information, wherein the camera motion and three-dimensional information are reconstructed by returning to the information restoring means and repeating until the error converges to a certain value or less. . From the temporal change in image coordinate values for feature points placed in the target image in a time-series omnidirectional image or wide-field image acquired by an image input device such as a fisheye camera, the time-series camera viewpoint A device for restoring motion and three-dimensional information constituting the object shape of the outside world,
With respect to the omnidirectional image or wide-field image acquired by the image input device, each image coordinate value is converted into a uv coordinate value by using a phase angle and an elevation angle obtained from each image coordinate value, and each pixel value is converted into its uv coordinate value. A uv measurement matrix measuring means for generating a time series uv image so as to be associated with each other and measuring the uv coordinate value of the feature point in each image in the feature point coordinate system set in the uv image;
For each uv coordinate value obtained by the uv measurement matrix measurement means, matrix decomposition data for restoration processing is generated, the matrix decomposition data is subjected to singular value decomposition, noise is removed, and matrix data representing motion information is generated. And obtaining matrix data representing 3D information, obtaining a transformation matrix satisfying the conditions set for defining the motion in the motion information, and applying this transformation matrix to the matrix data representing the motion information, 3D information that constitutes the object shape by applying the inverse matrix of this transformation matrix to the matrix data representing the 3D information and the rotational motion around the Z axis that is the optical axis and the translational motion in the X and Y directions of the 2 axes Plane motion / three-dimensional information restoration means for restoring
A reprojection value in the uv coordinate system for each feature point is obtained from the plane motion obtained by the plane motion / three-dimensional information restoration means and the three-dimensional information, and the reprojection value and each feature obtained by the uv measurement matrix measurement means. Z-axis motion that restores the translational motion of the camera viewpoint in the Z-axis direction using the error between the uv coordinate value of the point and the Z-coordinate value of each feature point obtained by the plane motion / three-dimensional information restoration means Recovery means,
From the matrix decomposition data, components that contribute to rotational motion around axes other than the optical axis are extracted, and those components and motion restored by the planar motion / three-dimensional information restoring means and the Z-axis motion restoring means. , As well as rotational motion restoring means for restoring rotational motion around another axis other than the optical axis using three-dimensional information;
It is determined whether or not the error obtained by the Z-axis motion restoration means has converged to a certain value or less. If not, the camera viewpoint obtained by the planar motion and the three-dimensional information restoration means is determined. Using the XY translation component, the three-dimensional information, and the rotation angle around the XY axis other than the optical axis, a coefficient ε for deforming the uv coordinate value is obtained, and each feature obtained by the plane motion and the three-dimensional information restoration means Using the point height information and the position of each viewpoint on the Z-axis obtained by the Z-axis translational motion restoration means, a coefficient δ for scaling the uv coordinate value is obtained, and the coefficient ε is obtained by the measurement matrix measurement means. The value obtained by multiplying each matrix element of the measured matrix and the coordinate value obtained by subtracting the intersection coordinate value from each matrix element of the measurement matrix obtained by the measurement matrix measuring means are multiplied by the coefficient δ, respectively. A new matrix element combining the obtained values Reconstructing the fisheye camera motion and 3D information, generating decomposed data, returning to the information restoring means, and repeating until the error converges below a certain value to restore the camera motion and 3D information apparatus.   In the measurement matrix measurement means, in the singular value component obtained by singular value decomposition of the measurement matrix, a determination value representing motion is calculated from the component, and if the determination value is less than a certain value, it is regarded as planar motion. Perform 3 degrees of freedom of plane motion consisting of rotation around the optical axis and motion on a plane perpendicular to the optical axis, and processing to restore 3D information. If the judgment value is above a certain value, The apparatus for restoring fisheye camera motion and three-dimensional information according to claim 5 or 6, wherein the motion with six degrees of freedom consisting of translational motion and the three-dimensional information of the outside world are restored.   In the uv measurement matrix measurement means, in a singular value component obtained by singular value decomposition of the measurement matrix, a determination value representing motion is calculated from the component, and if the determination value is less than a certain value, it is regarded as plane motion. If the judgment value is greater than a certain value, it performs a rotation about the optical axis and a plane motion with 3 degrees of freedom consisting of a rotation around the optical axis and a motion on a plane perpendicular to the optical axis. 7. The apparatus for restoring fisheye camera motion and three-dimensional information according to claim 5 or 6, wherein the motion with a degree of freedom of 6 consisting of translational motion and the three-dimensional information of the outside world are restored.   5. A recording medium having recorded thereon a program that can execute the fisheye camera motion and the three-dimensional information restoration method according to any one of claims 1 to 4 by a computer.

Images