JP2006172026A - Camera motion, and device and method for restoring three-dimensional information, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To stably and robustly restor camera motion and three-dimensional information at the same time. <P>SOLUTION: This restoring device is composed of a time-series image database 1 for storing time-series images, an asymptotic matrix generating part 2 for extracting an image series from the database 1, observing image coordinate values of a characteristic point between full frames and generating asymptotic matrix data to be a source for restoring camera motion and three-dimensional information, a plane motion/three-dimensional information restoring part 3 for restoring the plane motion and the three-dimensional information from the asymptotic matrix data, and a stabilization processing part 4 for restoring rotational motion other than optical axial rotation and optical axial translational motion, determining whether to require the next iteration to asymptotically make (stabilize) camera motion to plane motion and transferring information required to generate asymptotic matrix data to the asymptotic matrix generating part 2 when the iteration is necessary. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention generally relates to time-series images such as an in-vehicle image or indoor image acquired using a camera, a marine image from a ship, an aerial image, an omnidirectional image captured by an omnidirectional camera, and a walking image captured while walking. Three-dimensional (XYZ) composed of roll, pitch, and yaw rotation in a camera coordinate system set with the camera viewpoint as the origin from a time-series image A device that restores the three-dimensional information that constitutes the rotational shape of the surroundings, the translational motion in the three-axis (XYZ) direction, and the three-dimensional shape of the external environment reflected in the time-series image, that is, the appearance shape of the subject (object), Regarding the method.

  In the computer vision field, methods for measuring or acquiring the shape of an object from time-series images include a three-dimensional analysis method using stereo measurement and epipolar surface analysis. According to this method, it is possible to restore the three-dimensional position information related to the spatial shape or the spatial structure and the motion related to the camera viewpoint from a plurality of time-series images in which the object is photographed. However, it is difficult to seamlessly acquire images in time-series images taken while moving the camera using moving means, etc., due to the shooting environment and the minute movement of the camera. In some cases, it is difficult to accurately restore camera motion and object shape.

  In order to solve such a problem, there is a technique for simultaneously and robustly restoring the camera motion and the object shape in the Euclidean space from the video scene photographed by the camera. For example, there is a factorization method as a typical method for restoring camera motion and three-dimensional information from image coordinate values obtained by attaching feature points to an acquired image and tracking the feature points in time series (for example, (See Patent Document 1).

In this factorization method, as shown in equation (A1), a camera motion matrix (matrix on the left side of the right side ((m _ix )) is obtained from time-series xy image coordinate values (left side of equation (A1)) acquired on the image. , _{M iy} , m _iz ) and (n _ix , n _iy , n _iz ) represent the components of the camera motion according to the projection model in the i-th frame) and a matrix (X, Y, Z) relating to the three-dimensional information (X, Y, Z) This is a method of decomposing the matrix on the right side of the right side, (X _j , Y _j , Z _j ) is the j-th three-dimensional coordinate value. The left side of equation (A1) is called a measurement matrix, the row direction represents the number of feature points, the column direction represents the frame order, and (x _ij , y _ij ) represents the jth feature point of the i-th frame. Image coordinate values.

  In the factorization method, the left side of the equation (A1) is only two-dimensional information obtained from the image (a two-dimensional coordinate value obtained by subtracting the barycentric coordinate value of each frame in advance). This is based on a projection model in which an image coordinate value on the left side is obtained by performing a matrix operation (linear operation). However, the projection models that can be decomposed into the formula (A1) are limited to orthographic projection, weak perspective projection, and parallel perspective projection models. These projection models are approximate forms of actual perspective projection models, and depending on the relationship between the subject and the camera, the respective conditions may approximately hold. When factorization is performed as in (A1), the accuracy of camera motion and three-dimensional shape is poor.

On the other hand, when the perspective projection type factorization method is used, the factorization method of the formula (A1) is repeatedly used to asymptotically approach the perspective projection model, and the camera motion and the three-dimensional shape are obtained under the conditions of the perspective projection image. Can be restored at the same time. This method can obtain camera motion and three-dimensional shape with higher accuracy than orthographic, weak perspective, and parallel perspective projection models. However, it is necessary to consider the sign at the time of decomposition in each iteration. It was difficult to obtain accurate camera motion and 3D shape.
C. Tomasi and T. Kanade; "Shape and Motion from Image Streams UnderOrthography: A Factorization Method", International Journal of Computer Vision, Vol. 9, No. 2,1992.

  When a camera is mounted on a moving means such as a vehicle and moving and observed in an urban area, camera vibration (camera shake) occurs due to the relationship between the road surface and the vehicle. In addition, when a subject is photographed with a commercially available camera such as a mobile phone with a camera or a digital camera, there is a problem that a camera shake occurs and the video scene is shaken by a slight vibration during the photographing.

  The factorization method is effective when trying to restore the camera motion and 3D shape in a video scene with camera vibration and camera shake. However, when using the perspective projection type factorization method, the camera vibration and camera shake are effective. Therefore, there is a problem in that the camera vibration cannot be accurately obtained, and the three-dimensional shape is distorted. This problem is caused by the fact that the camera motion that must be restored as translational motion is converted into rotational motion.

  On the other hand, some commercially available cameras have camera vibration and camera shake correction functions, and can shoot seamless video scenes with reduced camera vibration and camera shake during shooting. In general, camera shake correction is an optical camera shake correction type that detects camera vibration or camera shake vibration using an angle sensor or gyro sensor, and moves the condensing lens accordingly, and CCD that detects optical flow in the screen. It can be roughly classified into an electronic image stabilization type that shifts pixels on the surface to reduce shaking. However, in both correction methods, it is difficult to formulate a projection model for restoring the camera motion and the three-dimensional shape, and many parameters are involved in obtaining an accurate projection model. Is difficult to apply.

  Therefore, it is important to restore the camera motion and 3D shape at the same time with high accuracy and robustness even in video scenes with camera vibrations and camera shakes during movement observation. Therefore, it is necessary to obtain the camera motion and the three-dimensional shape stably without considering the sign.

  When the camera is used to image the outside world and the motion of the camera viewpoint and the three-dimensional information are to be restored from the obtained image sequence, the motion related to the camera viewpoint and the three-dimensional coordinate value of the object are as shown in FIG. , (A2).

(X ′ _ij , y ′ _ij , z ′ _ij ) in Expression (A2) is a unit hemispherical coordinate value with the viewpoint as the origin, and the image coordinate value of the object (feature point) obtained by being projected onto the image. Can be observed as (x _ij , y _ij ) = (x ′ _ij / z ′ _ij , y ′ _ij / z ′ _ij ). However, since λ _ij cannot be obtained directly from the image coordinate values and cannot be separated into a component that depends on time series (suffix i) and a component that does not depend on time series (suffix j), A2) On the left side. Although the right side of Expression (A2) is composed of four matrices, the three rotation matrices can be expanded into one matrix, which can be a matrix corresponding to the camera movement on the right side of Expression (A1). As can be seen from A2) itself, information obtained from image coordinate values on the left side, information corresponding to camera motion (suffix i) and information corresponding to three-dimensional information are shown on the right side as in equation (A1). It is difficult to transform into (suffix j). In other words, the camera motion and the three-dimensional information cannot be restored by applying the factorization method based on the formula (A2).

  The problem to be solved by the present invention is that the projection model shown in the formula (A2) uses the method of the following patent document, and does not take into account the ambiguity such as the sign of the solution and stably The problem is to restore the dimension information simultaneously and robustly.

  Patent document “Japanese Patent Laid-Open No. 2003-271925, Miyagawa, Ozawa, Wakabayashi, Arikawa:“ Omnidirectional camera viewpoint movement and object shape restoration method and apparatus, omnidirectional camera viewpoint movement and object shape restoration method program, and recording the program recoding media""

(Principle explanation)
The present invention presupposes a projection model in which camera vibration is a small change in rotational motion, the camera motion is limited in equation (A2), and an image is captured with the camera motion of the limited rotational motion. That is, in the projection model in which the matrix expansion is performed with i = 1, 2,..., F; j = 1, 2,..., P in equation (A2), when the roll rotation and pitch rotation are very small, that is, cos ψ≈1, When sin ψ≈ψ, cos ω≈1, sin ω≈ω,

It becomes. In this projection model, as can be seen from the right side of Expression (A3), the matrix on the left side of the right side is information that depends on the time series (information on camera motion), and the matrix on the right side of the right side is invariant that does not depend on the time series. It is information (three-dimensional information of the outside world). This is in the form of plane motion and three-dimensional information factorized in the above-mentioned patent document. On the other hand, the left side of equation (A3) is related to information about camera motion and three-dimensional information in addition to the information obtained from the image. Even when the above-mentioned camera motion is limited, the projection model is too complicated and the above-mentioned patent document. Cannot be restored easily to restore camera motion and 3D information at the same time. However, if the image coordinate values (x _ij , y _ij ) observed in the image can be converted as in the left side of equation (A3), the camera motion and the three-dimensional shape can be restored by the method of the above-mentioned patent document.

In the present invention, roll rotation ω _i , pitch rotation ψ _i , yaw rotation θ _i , XYZ translational motion (Tx _i , Ty _i , Tz _i ); i = 1, 2,. , F, and the external three-dimensional information (X _j , Y _j , Z _j ); j = 1, 2,..., P (hereinafter, the three-dimensional information is a three-dimensional coordinate value for P feature points. Restore). In order to solve the equation (A3), the matrix element on the left side of the equation (A3) is repeatedly generated in the asymptotic matrix input step in the present invention, and the method of the above-mentioned patent document is performed in the plane motion / three-dimensional information restoration step in the present invention. To restore plane motion and 3D information.

  The present invention stabilizes the camera motion that is shaken by the camera vibration to the planar motion by making the camera motion asymptotic to the planar motion. That is, in the asymptotic matrix input step in the present invention, the matrix element on the left side of the formula (A3) in the next iteration uses a pitch other than plane motion, roll rotation (rotation other than optical axis rotation) and optical axis translational motion, The observed image coordinate values are converted into image coordinate values projected by plane motion. This transformation is equivalent to stabilizing the camera motion to a planar motion. In the stabilization processing step according to the present invention, necessary conversion coefficients for stabilization, pitch other than plane motion, roll rotation (rotation other than optical axis rotation) and optical axis translational motion are obtained, and stabilization proceeds. Thus, the plane motion and the three-dimensional information can be obtained with high accuracy in the plane motion / three-dimensional information restoration step of the present invention.

  As described above, according to the present invention, the reconstruction process is repeatedly performed stepwise using the method of the above-described patent document, so that the camera motion based on the projection model of Expression (A3) is approximately 3 It is possible to restore the dimension information.

  As described above, the present invention is characterized by the following apparatuses, methods, and programs.

(Invention of the device)
(1) In a time-series image, the movement of the camera viewpoint in the time series and the three-dimensional information constituting the object shape of the outside world are obtained from the temporal change amount of the image coordinate values regarding the feature points arranged in the target image. A device to restore,
In the feature point coordinate system set for the time series image, the image coordinate value (observation coordinate value) of the feature point in each frame image is input, and the rotation coordinate of the camera, the optical axis coordinate value, the translational motion, and Asymptotic matrix generation means for generating asymptotic matrix data whose elements are coordinate values multiplied by a coefficient consisting of three-dimensional information;
The asymptotic matrix data is subjected to singular value decomposition, noise removal is performed to obtain matrix data representing motion information and matrix data representing three-dimensional information, and the motion information satisfies the conditions set for defining the motion. A transformation matrix is obtained, and the transformation matrix is applied to the matrix data representing the motion information, so that the rotational motion around the optical axis with respect to the camera viewpoint and the translational motion on a plane perpendicular to the optical axis (a plane with three degrees of freedom consisting of these components). A plane motion / three-dimensional information restoring means for restoring the three-dimensional information constituting the object shape by applying an inverse matrix of the transformation matrix to the matrix data representing the three-dimensional information.
The plane motion obtained by the plane motion / three-dimensional information restoration means, the reprojection error calculated from the three-dimensional information, and the coordinates converted by the coefficient ε and the optical axis coordinate values into the observed coordinate values obtained by the asymptotic matrix generation means Matrix data having values as matrix elements is obtained, matrix data obtained by singular value decomposition of the matrix data to remove noise, and the Z coordinate value (Z direction) of each feature point obtained by the plane motion / three-dimensional information restoration means Optical axis motion that restores the translational motion in the optical axis direction of the camera viewpoint from the matrix whose element is the height of the feature point position when the is vertically oriented), and updates the coefficient δ by the restored optical axis translational motion Recovery means,
An error is obtained between the observed coordinate value obtained by the asymptotic matrix generation means, the plane motion obtained by the plane motion / three-dimensional information restoration means, and the re-projection error obtained from the three-dimensional information by the coefficient δ. From the matrix data having the error as a matrix element, the plane motion obtained by the plane motion / three-dimensional information restoring means, the three-dimensional information, and the optical axis translation obtained by the optical axis motion restoring means, the optical axis Rotational motion restoration means for obtaining rotational motion around mutually orthogonal axes other than and updating the coefficient ε and the optical axis coordinate value by the restored rotational motion,
The observed coordinate values obtained by the asymptotic matrix generating means are converted into the conversion coefficient (coefficient ε) and the optical axis coordinate value updated by the rotational motion restoring means, and the conversion coefficient (coefficient δ) updated by the optical axis motion restoring means. ), The asymptotic value to the plane motion between the plane motion obtained by the plane motion / three-dimensional information restoration means and the reprojected coordinate value of the feature point for each frame image from the three-dimensional information. An asymptotic error is calculated, and processing for reducing the asymptotic error (camera motion asymptotically approaches planar motion) by switching between processing by the optical axis translational motion restoring means and processing in the rotational motion restoring means by increasing or decreasing the asymptotic error. Repetitive stabilization means,
It is characterized by having.

  (2) In the feature point coordinate system set for the omnidirectional image in the above (1), means for inputting the image coordinate value of the feature point in each frame image, and an azimuth angle from a reference axis based on the image coordinate value ( (Phase angle), means for obtaining an angle (elevation angle) from the optical axis direction obtained according to the projection method used in the omnidirectional camera, and coordinate values obtained by coordinate conversion using the phase angle and elevation angle It has the means to obtain as, It is characterized by the above-mentioned.

  (3) In the feature point coordinate system set in the time-series image in (1) above, means for inputting image coordinate values of feature points in each frame image, and a matrix having the observed coordinate values as matrix elements are singular values. A means for decomposing, a means for calculating a determination value representing the degree of freedom of movement from the component of this singular value, and if this determination value is less than a certain value, the rotation around the optical axis and its light Means for restoring three-dimensional information with three degrees of freedom consisting of movement on a plane perpendicular to the axis, and a degree of freedom of six consisting of rotational movement and translational movement of the camera if the determination value exceeds a certain value. And a means for restoring three-dimensional information.

(4) In the feature point coordinate system set in the omnidirectional image in (1) above, means for inputting the image coordinate value of the feature point in each frame image, and an azimuth angle from a reference axis (from the image coordinate value) (Phase angle), means for obtaining an angle (elevation angle) from the optical axis direction obtained according to the projection method used in the omnidirectional camera, and coordinate values obtained by coordinate conversion using the phase angle and elevation angle As a means to seek
Means for singular value decomposition of a matrix having the observed coordinate values as matrix elements, means for calculating a determination value representing the degree of freedom of movement from the component of this singular value, and when the determination value is less than a certain value, If the judgment value is greater than a certain value, it is considered to be a planar motion, means for restoring three-dimensional information with three degrees of freedom consisting of rotation around the optical axis and motion on a plane perpendicular to the optical axis. A means for restoring the three-dimensional information and the six-degree-of-freedom motion comprising the rotational motion and translational motion of the camera;
It is characterized by having.

(Invention of method)
(5) In the time-series image, the movement of the camera viewpoint in the time series and the three-dimensional information constituting the object shape of the outside world are obtained from the temporal change amount of the image coordinate value regarding the feature point arranged in the target image. A method of restoring,
In the feature point coordinate system set for the time series image, the image coordinate value (observation coordinate value) of the feature point in each frame image is input, and the rotation coordinate of the camera, the optical axis coordinate value, the translational motion, and An asymptotic matrix generating step for generating asymptotic matrix data having a coordinate value multiplied by a coefficient consisting of three-dimensional information as an element;
The asymptotic matrix data is subjected to singular value decomposition, noise removal is performed to obtain matrix data representing motion information and matrix data representing three-dimensional information, and the motion information satisfies the conditions set for defining the motion. A transformation matrix is obtained, and the transformation matrix is applied to the matrix data representing the motion information, so that the rotational motion around the optical axis with respect to the camera viewpoint and the translational motion on a plane perpendicular to the optical axis (a plane with three degrees of freedom consisting of these components). A plane motion / three-dimensional information restoring step for restoring the three-dimensional information constituting the object shape by applying an inverse matrix of the transformation matrix to the matrix data representing the three-dimensional information.
Coordinates obtained by converting the plane motion obtained in the plane motion / three-dimensional information restoration step, the reprojection error calculated from the three-dimensional information, and the observed coordinate value obtained in the asymptotic matrix generation step using the coefficient ε and the optical axis coordinate value. Matrix data having values as matrix elements is obtained, the matrix data obtained by singular value decomposition of the matrix data to remove noise, and the Z coordinate value (Z direction) of each feature point obtained in the plane motion / three-dimensional information restoration step. Optical axis motion that restores the translational motion in the optical axis direction of the camera viewpoint from the matrix whose element is the height of the feature point position when the is vertically oriented), and updates the coefficient δ by the restored optical axis translational motion A restore step,
An error between the observed coordinate value obtained in the asymptotic matrix generation step, the plane value obtained in the plane motion / three-dimensional information restoration step, and the re-projection error obtained from the three-dimensional information is obtained by a coefficient δ. From the matrix data having the error as a matrix element, the plane motion obtained in the plane motion / three-dimensional information restoration step, the three-dimensional information, and the optical axis translation obtained in the optical axis motion restoration step, the optical axis A rotational motion restoring step for obtaining rotational motion around mutually orthogonal axes other than and updating the coefficient ε and the optical axis coordinate value by the restored rotational motion,
The observed coordinate values obtained in the asymptotic matrix generation step are converted into the conversion coefficient (coefficient ε) and the optical axis coordinate value updated in the rotational motion restoration step, and the conversion coefficient (coefficient δ) updated in the optical axis motion restoration step. ), The asymptotic value to the plane motion between the plane motion obtained in the plane motion / three-dimensional information restoration step and the reprojected coordinate value of the feature point for each frame image from the three-dimensional information. An asymptotic error representing the following is obtained, and processing to reduce the asymptotic error (camera motion is asymptotic to planar motion) by switching the processing in the optical axis translational motion restoration step and the processing in the rotational motion restoration step by increasing or decreasing the asymptotic error. Repeated stabilization steps;
It is characterized by having.

  (6) In the feature point coordinate system set in the omnidirectional image in (5) above, the step of inputting the image coordinate value of the feature point in each frame image, and the azimuth angle from a reference axis based on the image coordinate value ( Phase angle), a step of obtaining an angle (elevation angle) from the optical axis direction obtained according to the projection method used for the omnidirectional camera, and a coordinate value obtained by performing coordinate transformation using the phase angle and the elevation angle. And a step of obtaining as follows.

  (7) In the feature point coordinate system set in the time series image in (5) above, a step of inputting image coordinate values of feature points in each frame image, and a matrix having the observed coordinate values as matrix elements are singular values. A step of decomposing, a step of calculating a determination value representing the degree of freedom of movement from the component of this singular value, and if this determination value is less than a certain value, the rotation around the optical axis and the light A step of restoring three-dimensional information with three degrees of freedom consisting of movement on a plane perpendicular to the axis, and a degree of freedom of six consisting of rotational movement and translational movement of the camera if the determination value is greater than a certain value. And a step of restoring three-dimensional information.

(8) In the feature point coordinate system set in the omnidirectional image in (5) above, the step of inputting the image coordinate value of the feature point in each frame image, and the azimuth angle from a certain reference axis from the image coordinate value ( Phase angle), a step of obtaining an angle (elevation angle) from the optical axis direction obtained according to the projection method used for the omnidirectional camera, and a coordinate value obtained by performing coordinate transformation using the phase angle and the elevation angle. And asking for steps
A step of singular value decomposition of a matrix having the observed coordinate values as matrix elements, a step of calculating a determination value representing the degree of freedom of movement from the component of the singular value, and the determination value is less than a certain value, A step of restoring three-dimensional information with three degrees of freedom consisting of rotation around the optical axis and movement on a plane perpendicular to the optical axis as if it were a plane movement, Reconstructing the three-dimensional information and the six-degree-of-freedom motion comprising the rotational motion and translational motion of the camera;
It is characterized by having.

(Invention of the program)
The processing procedure in the camera motion and three-dimensional information restoration method described in any one of (5) to (8) above is configured to be executable by a computer.

  As described above, according to the present invention, all time-series images acquired using a camera (vehicle-mounted images, sea images, aerial images, indoor images, fish-eye cameras, omnidirectional cameras, Alternatively, it is possible to acquire and restore the motion of the camera (rotational motion and translational motion) and the object shape related to the object with high accuracy from the manually captured image.

  Since manual shooting has camera shake and the on-vehicle camera has vibration during traveling, the present invention is robust to noise even in such camera vibration, and can perform minute camera posture fluctuations (rotation around three axes). Can be restored. In particular, when the present invention is applied to the mobile observation with the camera shown in FIG. 15, it is possible to accurately measure the translational motion in the three-axis directions with the accuracy of interpolating a remote sensor such as GPS.

  Since most of the calculations used in the present invention are composed of linear operations, implementation in a computer language is easy.

(Embodiment 1)
FIG. 1 is a basic configuration diagram relating to claim 1 and the like, FIG. 2 is a processing configuration diagram in the case of processing in real time that does not require a storage device such as a time-series image database unit, and FIG. 2 is a processing flow of a wavy line block portion in FIG.

  This embodiment will be described with reference to FIGS. In this embodiment, a time-series image database unit 1 for storing time-series images, an image sequence is extracted from the database unit 1, image coordinate values of feature points between all frames are observed, and camera motion and three-dimensional information are restored. An asymptotic matrix generator 2 for generating matrix data (hereinafter referred to as asymptotic matrix data) to be a base for performing plane motion and three-dimensional information to restore plane motion and three-dimensional information from the asymptotic matrix data using the method of the above-mentioned patent document 3D information restoration unit 3, restores rotational motion other than optical axis rotation and optical axis translational motion, and determines whether the next iteration is necessary to make the camera motion asymptotic (stabilized) to planar motion. Is required, the stabilization processing unit 4 passes information necessary for generating asymptotic matrix data to the asymptotic matrix generation unit 2. In this configuration, the time series image database unit 1 may be in a form using a recording medium such as a hard disk, a RAID device, a CD-ROM, or a form using remote data resources via a network. Also, the image input unit 1A in FIG. 2 obtains data resources in real time instead of the time-series image database unit 1.

In FIG. 14, the point P _j (X _j , Y _j , Z _j ) in the space to be restored in the present invention and the camera motion, that is, roll rotation (ω _i ), pitch rotation (ψ _i ), yaw The rotation (θ _i ) and the translational movement T _i (Tx _i , Ty _i , Tz _i ) will be described. FIG. 14 shows the positional relationship between the camera and the object (subject), the center of motion is the viewpoint, and the camera coordinate system XYZ with the viewpoint as the origin and the world coordinate system XwYwZw with the origin O are set. For convenience of explanation, the camera optical axis is the Z-axis direction, and the plane perpendicular to the optical axis is the XY plane. In this coordinate system, the camera rotates by roll rotation (ω _i ), pitch rotation (ψ _i ), and yaw rotation (θ _i ) while moving in translational motion (Tx _i , Ty _i , Tz _i ). Observe the point P _j . The position (viewpoint position T _i ) of the projection center (principal point) on which the image is projected is the center of the camera movement, and is the position of the translational movement T _i (Tx _i , Ty _i , Tz _i ) in the i-th frame. . It is assumed that the point P _j (X _j , Y _j , Z _j ) of the object is projected by the camera onto the image coordinate value (x _ij , y _ij ) with the projection center as the origin on the image plane. Note that the viewpoint in the initial frame and O coincide with each other, the optical axis is parallel to the Zw axis, and θ _i is the angle formed by the X and Xw axes, but the generality is not impaired.

First, in the asymptotic matrix generation unit 2 of FIG. 1 or FIG. 2, an image sequence of the number of frames F is extracted from the time-series image database unit 1 as a time-series image obtained by photographing the object. Feature point tracking is performed on the extracted image series. The feature points are extracted by the following procedure as used conventionally. In the region of the initial image (image 1), (1) a 2 × 2 Hessian matrix for each pixel is obtained. Next, (2) it is determined whether or not the point is a local maximum in the 3 × 3 neighborhood of each point, and points other than the local maximum are deleted (non-maxima suppression). Further, (3) eigenvalues σ _l and σ _s (σ _s ≦ σ _l ) of the obtained Hessian matrix are obtained, and points where σ _s is equal to or greater than a predetermined allowable value σ _p are extracted. Finally, (4) sorting is performed in the order of the size of σ _s of the extracted points, and the point (p _h ) higher than the point (p _l ) in order from the upper point is the distance within the predetermined number of pixels σ _d If it exists, the lower point p ₁ is deleted. Further, the extracted feature points (j = 1, 2,..., P) are tracked over the image i (i = 2,..., F) by the KLT method (Kanade-Lucas-Tomasi), and the image coordinate values (x _ij , y _ij ) is observed. 2F × P matrix data (matrix data [A]) in which the image coordinate values of the features obtained in this way are arranged in the array shown in Expression (1) is prepared.

The matrix data [A] is transferred to the generation of asymptotic matrix data (S2) from the input of observed coordinate values (S1) in FIG. In the generation of asymptotic matrix data (S2), coefficients ε _ij = 1, δ _ij = 1, (ζ _i , η _i ) = (0, 0), Tz _i = 0 are initialized, and converted coordinates according to equation (2) A value (x ′ _ij , y ′ _ij ) is obtained. Asymptotic matrix data [B] of 2F × P expression (1a) having the converted coordinate values (x ′ _ij , y ′ _ij ) as matrix elements is generated. At this time, as an initial setting, the stabilization mode is set to the “rotation mode”.

  Next, when asymptotic matrix data is given to the planar motion / three-dimensional information restoration unit 3 of FIG. 1 or FIG. 2, three matrices [U], shown in the equation (3) in the singular value decomposition (S3) of FIG. Matrix decomposition into [W] and [V].

  Here, [U] is a 2F × P size matrix, [W] is a P × P size diagonal matrix, and [V] is a P × P size matrix. Further, in the noise removal (S4) of FIG. 3, as shown in the second term of the equation (4), the components of each matrix of rank 4 or higher are deleted.

  At the time of this deletion, the matrix [U] is taken out, the fourth to Pth columns are deleted from the elements of this matrix, the matrix consisting of the remaining components is held, the matrix [W] is taken out, The fourth to Pth rows and the fourth to Pth columns are deleted in the element, the matrix composed of the remaining components is retained, the matrix [V] is extracted, and the fourth to Pth rows are extracted from the elements of this matrix. Delete up to the eyes and keep the matrix of the remaining components. This noise removal is as shown in equation (5).

  Next, from the matrix obtained by taking the square of the diagonal elements of the matrix [W] from which the 4th to Pth rows and the 4th to Pth columns have been deleted, the matrix shown in Equations (6) and (7) [ U ′] and matrix [V ′] are obtained.

  In the transformation matrix calculation (S5) of FIG. 3, the held matrix [U ′] is taken out, and the matrix [D] of Expression (11) having values obtained by Expressions (8) to (10) as matrix elements is obtained. Prepare and perform the calculation shown in the matrix [D] and the equation (12) to obtain the values a, b, c, d, e, and f. Note that the last matrix on the right side of Expression (12) is a 3F × 1 column vector in which 2 values from the top are 1F, and F values from 0 are subsequently arranged. A matrix [C] in which the values a, b, c, d, e, and f are put in the elements shown in the equation (13) is prepared, and the matrix [C] is subjected to eigenvalue decomposition as shown in the equation (14). Here, a matrix [C ′] of Expression (15) is generated from the square of the eigenvalue matrix and the eigenvalue matrix, and a matrix [Q] having these matrix elements as components is calculated according to Expression (16).

Next, in the plane motion restoration (S6) of FIG. 3, based on the projection model formula of the formula (A3), the matrix on the left side of the formula (A3) is changed to the matrix [M ′] obtained by the formula (17). The matrix on the right side of the right side of equation (A3) is defined as the matrix [S ′] obtained by equation (20). By this association, the rotation (yaw rotation angle) θ _i around the optical axis and the XY translational motion are restored from the matrix component of Expression (17), and the three-dimensional information (X in the Euclidean space (X , Y, Z). From the obtained matrix [Q] and the retained matrix [U ′], the matrix [M ′] is calculated by the matrix operation of Expression (17). The matrix element (m _ix , n _ix ) or (m _iy , n _iy ) of each frame (i-th frame) is extracted from the matrix [M ′], and the yaw rotation θ _i , i = 1 is obtained using equation (18). , 2, ..., F are restored. Further, the matrix element (T _iu , T _iv ) of each frame (i-th frame) is extracted from the matrix [M ′]. From this (T _iu , T _iv ), XY translational motion (T _xi , T _yi ), i = 1, 2,..., F in the Euclidean space in the i-th frame is calculated using equation (19).

On the other hand, in the three-dimensional information restoration (S7) in FIG. 3, the matrix [V ′] previously held and the matrix [Q] obtained by the transformation matrix calculation (S5) are shown in Expression (20). Matrix operation is performed to obtain a matrix [S ′]. Next, the calculation shown in Expression (21) is performed on the elements of the matrix [S ′], and the matrix having these elements as [P]. The column vector of the matrix [P] is a three-dimensional coordinate value (X _j , Y _j , Z _j ) in the Euclidean space of the jth feature point.

  Next, the stabilization processing unit 4 shown in FIG. 1 or 2 checks whether the stabilization mode is the rotation mode or the optical axis translation mode by performing the check (S8) in FIG. 3, and performs different processing. If the stabilization mode is the rotation mode, the process proceeds to rotational motion stabilization (S9) in FIG. 3, and if the stabilization mode is the optical axis translation mode, the process proceeds to optical axis translational motion stabilization (S10) in FIG. .

In the rotational motion stabilization (S9), pitch rotation and roll rotation are obtained assuming that the coefficient δ _ij is known. In this processing, an error between the image coordinate values (x _ij , y _ij ) observed from the image series and the reprojection coordinate values (U _ij , V _ij ) obtained by the plane motion / three-dimensional information restoration unit 3 is The error is caused by pitch rotation and roll rotation. If this is expressed in mathematical formulas,

It becomes. This can be expressed by the matrix element on the left side of equation (A3).

It becomes. When the left side of Expression (A8) is expanded using Expressions (A4) and (A6),

It becomes. The matrices [R _i ] and [A _i ] are the expressions (22), (22a) to (22d), and (23). Therefore, the pitch rotation ψ _i and the roll rotation ω _i are obtained by the calculation of the equation (24) from the relationship of the equation (A9).

FIG. 4 and FIG. 5 show detailed processing flows of the stabilization processing (S9) and (S10) for the above calculation. In the rotational motion stabilization (S9) shown in FIG. 4, each matrix element of the matrix data [A] in the data format shown in Expression (1), the coefficient δ _ij , and the planar motion / three-dimensional information restoration restored above. The error (Δx _ij , Δy _ij ) shown in the above equation (A7) is calculated from the reprojection coordinate value calculation (S22) and the three-dimensional information restoration (S23) by the unit 3 (S21). Further, the values of the equations (22a) to (22d) are calculated (S24), and the matrix [R _i ] of the equation (22) having these as elements is prepared (S25). On the other hand, a matrix [A _i ] of Expression (23) having errors (Δx _ij , Δy _ij ) as matrix elements is prepared (S26), and the calculation of Expression (24) is performed to perform roll rotation ω _i and pitch rotation ψ. _i is restored (S27). This is obtained over all frames (i = 1, 2,..., F). At the same time, (ζ _i , η _i ), i = 1, 2,..., F are obtained for each frame by the calculation of equation (25), and the previous (ζ _i , η _i ), i = 1, 2,..., F are updated, and the coefficient ε _ij is updated by substituting the pitch rotation ψ _i and roll rotation ω _i obtained by the equation (A6) (S28).

In the optical axis translational motion stabilization S10 shown in FIG. 3, the optical axis translational motion Tz _i is obtained on the assumption that the coefficient ε _ij and the optical axis coordinate values (ζ _i , η _i ) are known. At this time, the matrix element on the left side of Expression (A3) is

It becomes. Expression (A10) is a format in which (1-Tz _i / Z _j ) times the coordinate value (u ′ _ij , v ′ _ij ) is the reprojected coordinate value (u _ij , v _ij ). Using this relationship, the optical axis translational motion Tz _i is obtained. Therefore, the equation (A10) is

And matrix expansion for all frames i = 1, 2,..., F and all feature points j = 1, 2,.

(Where Δw _ij is Equation (26)). Therefore, the optical axis translational motion can be restored by obtaining simultaneous equations shown in Formula (A13).

In the detailed processing flow of optical axis translational stabilization (S10) shown in FIG. 5, first, coordinate values (x _ij , y _ij ) are given from the input of observed coordinate values, and coefficient ε _ij and optical axis coordinate values ( The coordinate values (u ′ _ij , v ′ _ij ) of the equation (A11) are generated using ζ _i , η _i ). From the coordinate values (u ′ _ij , v ′ _ij ) and the re-projection coordinate values (u _ij , v _ij ) obtained from the plane motion and the three-dimensional information from the plane motion / three-dimensional information restoration unit 3, Δw _ij is calculated (S31, S32), and a matrix [ΔW] of Formula (27) of F × P using this as a matrix element is prepared (S33). At this time, the rank is 1 as can be seen from the left side of the formula (A13). On the other hand, the matrix [ΔW] on the right side of Expression (A13) is F × P. Therefore, singular value decomposition is performed as shown in Expression (28), and decomposition into three matrices, that is, F × P [U _w ], P × P [W _w ], and P × P [V _w ]. (S34).

Next, in noise removal (S35), the components of each matrix of rank 1 or higher are deleted. At the time of this deletion, the matrix [U _w ] is taken out, the second to Pth columns are deleted from the elements of this matrix, the matrix consisting of the remaining components is held (matrix [U ′ _w ]), and the matrix [W _w ] is taken out, the 2nd to Pth rows and the 2nd to Pth columns are deleted from the elements of this matrix, and a matrix consisting of the remaining components is held (matrix [W ′ _w ]). The matrix [V _w ] is taken out, the second to P-th rows are deleted from the elements of this matrix, and the matrices composed of the remaining components are held (matrix [V ′ _w ]), respectively, as shown in equation (28a) Matrix operation is performed and held as a matrix [ΔW] (S36). Further, a matrix [1 / Z] of Expression (30) consisting of the reciprocal of the Z value of the feature point is prepared from the three-dimensional information restoration (S37), and the calculation of Expression (29) is performed to calculate the optical axis translational motion Tz _i . Obtain (S38). Here, the coefficient δ _ij is updated using the optical axis translation obtained by the equation (A6) (S39).

  As described above, depending on whether the stabilization mode is the rotation mode or the optical axis translation mode, the pitch rotation, the roll rotation, and the optical axis translational motion are obtained by distributing to the above processing.

Returning to FIG. 3, using the coefficient ε _ij , the optical axis coordinate values (ζ _i , η _i ), and the coefficient δ _ij at this time, the coordinate values (x ′ _ij , y ′ _ij ) are calculated by the equation (2). ) To obtain an asymptotic error ΔE shown in equation (31) (S11). When this asymptotic error ΔE is smaller than the previous ΔE, the process returns to the generation of asymptotic matrix data, and is obtained using the updated coefficient ε _ij , optical axis coordinate values (ζ _i , η _i ), and coefficient δ _ij . An asymptotic matrix [B] having the coordinate values (x ′ _ij , y ′ _ij ) of Expression (2) as a new matrix element is obtained, and the process of plane motion / three-dimensional information restoration is continued (S12).

  On the other hand, when this asymptotic error ΔE increases from the previous ΔE, the stabilization mode is switched (S13, S14). That is, the process proceeds to generation of asymptotic matrix data by switching from the rotation mode to the optical axis translation mode or from the optical axis translation mode to the rotation mode. When the number of times of switching the stabilization mode exceeds N times from the beginning, it is determined that the asymptotic error ΔE has converged, and the iterative process is stopped, and the camera motion and three-dimensional information at that time are output. The process is terminated.

  As described above, according to the present embodiment, from the temporal movement of the feature points in the image series, the movement of the camera viewpoint, that is, the rotational movement around the three axes and the translational movement in the three axial directions, and the three-dimensional information constituting the object shape are obtained. Can be restored.

(Embodiment 2)
The omnidirectional camera is used in robot vision and mobile observation, and is installed on the roof of a vehicle as shown in FIG. 15 and installed with the optical axis facing the sky. FIG. 15 is a schematic diagram at the time of moving observation by a vehicle, and an urban landscape such as a building wall surface is imaged by an omnidirectional camera. In the moving observation, the omnidirectional camera is fluctuated due to the vehicle vibration, and the present embodiment restores the camera motion and the three-dimensional information from the image sequence shaken by the vehicle vibration.

  FIG. 6 is a basic configuration diagram relating to claim 2 and the like. In the present embodiment, the camera is an omnidirectional camera. Compared with the first embodiment, the image coordinates of the feature points obtained by the feature point observation unit 5 in the image extracted from the time-series image database unit 1 are described. Since only the point where the coordinate value obtained by converting the value by the coordinate conversion unit 6 is the image coordinate value of the feature point in the first embodiment is different, only this portion will be described.

  In this embodiment, the time-series image database unit may use a recording medium such as a hard disk, a RAID device, a CD-ROM, or a remote data resource via a network. Furthermore, FIG. 7 is a processing configuration diagram in the case of processing in real time, and in this embodiment, a storage device such as each database unit 1 is not necessarily required.

Since the omnidirectional camera is designed to capture a wide field of view unlike a commercially available camera, the present embodiment cannot be applied with the image coordinate values of the feature points in the omnidirectional image. Therefore, as shown in FIG. 16, the phase angle ρ _ij and the elevation angle φ _ij are used as the image coordinate values indicating the feature points. It is assumed that a point P _j (X _j , Y _j , Z _j ) in space is projected onto an image coordinate value (x _ij , y _ij ) on the image plane. Expression (32a) is optical projection with a fish-eye lens called equidistance projection with the focal length f, and the phase angle ρ _{ij in} FIG. 16 is obtained from the image coordinate values (x _ij , y _ij ) from Expression (33). Can be obtained using However, FIG. 16 is an example in which an image is captured by a camera equipped with a fisheye lens, R is the radius of the image circle, and the origin is the image coordinate value of the projection center on the image plane. As an omnidirectional camera other than this, in the case of projection reflected by a parabolic mirror, the elevation angle φ _ij is obtained by the equation (32b) (h is the radius of the paraboloid in the xy plane), and all reflected by the hyperbolic mirror. In the case of the azimuth camera, the elevation angle φ _ij is obtained by the equation (32c) (b and c are hyperbolic parameters, and f is a focal length). In these omnidirectional cameras, since the image plane and the XY plane can be considered parallel, the phase angle ρ _ij can be obtained from the general omnidirectional camera by Expression (33).

  When the phase angle and the elevation angle are obtained from the image coordinate value of the feature point in the feature point observation unit 5 in FIG.

To obtain coordinate values (p _ij , q _ij ).

  The effect of this coordinate transformation will be described with reference to FIGS. FIG. 17 is an example of an image acquired when moving and observing with an omnidirectional camera (fisheye or the like). As shown in this figure, in the omnidirectional image, the horizontal and vertical edges located on the wall of the building are projected as curves where the horizontal and vertical edges should be projected as straight lines. On the other hand, if the coordinates are converted by the equation (A14), horizontal and vertical edges can be obtained as line segments as shown in FIG. This coordinate transformation is a coordinate transformation from omnidirectional projection to perspective projection.

The coordinate conversion unit 6 regards (p _ij , q _ij ) obtained by the equation (A14) as (x _ij , y _ij ), and uses this (x _ij , y _ij ) as the image coordinate value (observed coordinate value) of the feature point. ) In the asymptotic matrix generation unit 2 in FIG.

  As described above, according to the embodiment of the present invention, the motion of the camera viewpoint, that is, the rotational motion around the three axes and the translational motion in the three axial directions, and the object shape are configured from the temporal motion of the feature points in the omnidirectional image sequence. Three-dimensional information can be restored.

(Embodiment 3)
FIG. 8 is a process configuration diagram of the present embodiment. Since the process of the restoration process determination unit 7 is different from that of the first embodiment, only this part will be described below. In this configuration, the time series image database unit 1 may be in a form using a recording medium such as a hard disk, a RAID device, a CD-ROM, or a form using remote data resources via a network. Furthermore, FIG. 9 is a processing configuration diagram in the case of processing in real time, and in this embodiment, a storage device such as the time-series image database unit 1 is not necessarily required.

In the restoration processing determination unit in FIG. 8, the image coordinate values of the feature points are observed for the image series extracted from the time-series image database unit 1, and held as a matrix [A] in the form of equation (1). FIG. 10 shows the subsequent processing flow, which will be described according to this processing flow. The matrix [A] is subjected to matrix decomposition into three matrices [U], [W], and [V] shown in Expression (3). The singular value matrix [W] is a diagonal element, and the singular values that are the respective elements are arranged in ascending / descending order and are all positive real numbers. From this singular value matrix, the singular values of the matrix element W ₃₃ of (3, 3) and the matrix element W ₄₄ of (4, 4) shown in Expression (4) are extracted (S41, S42). In rank detection, the calculation shown in Formula (34a) or Formula (34b) is performed to obtain the determination amount E _w (S43). It is determined whether this determination amount E _w is less than the allowable value or whether the determination amount E _w is greater than or equal to the allowable value (S44). This allowable value is a specific constant value, and the user (or worker) can also set the value sequentially. If less than the allowable value is determined amount E _w is camera motion proceeds to the process it is determined that the planar motion A (S45, S46), when exceeding the allowable value is determined amount E _w is the camera It is determined that the motion is a general motion (other than planar motion), and the process proceeds to process B (S47, S48).

  The process B is the process of the first embodiment, and from the temporal movement of the feature points in the image series by the process B, the movement of the camera viewpoint, that is, the rotational movement around the three axes and the translational movement in the three axial directions (6 degrees of freedom). And the three-dimensional information constituting the object shape are restored. On the other hand, when it is determined as the process A, the plane motion and the three-dimensional information are restored according to FIG. Since this processing is equivalent to the processing content of the planar motion / three-dimensional information restoration unit of the first embodiment, a description thereof will be omitted.

  With the above processing, when the camera motion is a plane motion, the calculation cost is greatly reduced by calculating only the plane motion without going to the processing of the first embodiment, and the camera motion and the three-dimensional information are calculated. Can be restored.

(Embodiment 4)
FIG. 12 is a basic configuration diagram relating to claim 4 and the like. In the present embodiment, only the process of the restoration process determination unit 7 is different from the second embodiment. In the present embodiment, the time-series image database unit 1 may use a recording medium such as a hard disk, a RAID device, a CD-ROM, or a remote data resource via a network. Furthermore, FIG. 13 is a processing configuration diagram in the case of processing in real time, and this embodiment does not necessarily require a storage device such as each database unit 1.

In the restoration processing determination unit 7 of FIG. 12, the image coordinate values of the feature points are observed for the image series extracted from the time-series image database unit 1 and held as a matrix [A] in the form of equation (1). Next, the phase angle and the elevation angle are obtained from the image coordinate values of each feature point according to the projection of the omnidirectional camera, the coordinate transformation of equation (A14) is performed, and the coordinate values (p _ij , q _ij ) are obtained. This (p _ij , q _ij ) is regarded as (x _ij , y _ij ), and this (x _ij , y _ij ) is used as the image coordinate value (observation coordinate value) of the feature point to the restoration processing determination unit 7 in FIG. Handle. Since the process of the restoration process determination unit 7 is the same as the process of the third embodiment, a description thereof will be omitted.

  By the above processing, when the omnidirectional camera motion is a plane motion, the calculation cost is greatly reduced by calculating only the plane motion without going to the processing of the first embodiment, and the camera motion and 3 Dimensional information can be restored.

  In the present invention, some or all of the processing functions of the methods shown in FIGS. 3 to 5 and the like can be configured as a program and can be executed using a computer. It is also possible to provide a recording medium recording this program through a network.

The basic block diagram of the image storage type decompression | restoration apparatus which concerns on Claim 1 grade | etc.,. A basic configuration diagram of a real-time processing type restoration apparatus according to claim 1 and the like. A processing flow of the restoration method according to claim 1 and the like. Processing flow for rotation stabilization. Processing flow for optical axis translation stabilization. The basic block diagram of the image storage type decompression | restoration apparatus which concerns on Claim 2 grade | etc.,. A basic configuration diagram of a real-time processing type restoration apparatus according to claim 2 or the like. The basic block diagram of the image storage type decompression | restoration apparatus which concerns on Claim 3 grade | etc.,. A basic configuration diagram of a real-time processing type restoration apparatus according to claim 3 and the like. The processing flow in a restoration process determination part. Process flow of process A. The basic block diagram of the image storage type decompression | restoration apparatus based on Claim 4 grade | etc.,. A basic configuration diagram of a real-time processing type restoration apparatus according to claim 4 and the like. The figure which shows the positional relationship of a camera, a to-be-photographed object, and a camera coordinate system. The figure in the case of carrying out mobile observation, fixing an omnidirectional camera to the ceiling of a vehicle. The figure which shows the positional relationship of an omnidirectional camera and a to-be-photographed object in FIG. 15, and a camera coordinate system. The figure which shows the edge line of the building wall surface which the omnidirectional camera imaged. The figure which shows the result of having coordinate-transformed the edge of FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Time series image database part 1A Image input part 2 Asymptotic matrix production | generation part 3 Plane motion and three-dimensional information restoration part 4 Stabilization process part 5 Feature point observation part 6 Coordinate conversion part 7 Restoration process determination part

Claims (9)

An apparatus for restoring the movement of the camera viewpoint in the time series and the three-dimensional information constituting the object shape of the outside world from the temporal change amount of the image coordinate value regarding the feature point arranged in the target image in the time series image Because
In the feature point coordinate system set for the time series image, the image coordinate value (observation coordinate value) of the feature point in each frame image is input, and the rotation coordinate of the camera, the optical axis coordinate value, the translational motion, and Asymptotic matrix generation means for generating asymptotic matrix data whose elements are coordinate values multiplied by a coefficient consisting of three-dimensional information;
The asymptotic matrix data is subjected to singular value decomposition, noise removal is performed to obtain matrix data representing motion information and matrix data representing three-dimensional information, and the motion information satisfies the conditions set for defining the motion. A transformation matrix is obtained, and the transformation matrix is applied to the matrix data representing the motion information, so that the rotational motion around the optical axis with respect to the camera viewpoint and the translational motion on a plane perpendicular to the optical axis (a plane with three degrees of freedom consisting of these components). A plane motion / three-dimensional information restoring means for restoring the three-dimensional information constituting the object shape by applying an inverse matrix of the transformation matrix to the matrix data representing the three-dimensional information.
The plane motion obtained by the plane motion / three-dimensional information restoration means, the reprojection error calculated from the three-dimensional information, and the coordinates converted by the coefficient ε and the optical axis coordinate values into the observed coordinate values obtained by the asymptotic matrix generation means Matrix data having values as matrix elements is obtained, matrix data obtained by singular value decomposition of the matrix data to remove noise, and the Z coordinate value (Z direction) of each feature point obtained by the plane motion / three-dimensional information restoration means Optical axis motion that restores the translational motion in the optical axis direction of the camera viewpoint from the matrix whose element is the height of the feature point position when the is vertically oriented), and updates the coefficient δ by the restored optical axis translational motion Recovery means,
An error is obtained between the observed coordinate value obtained by the asymptotic matrix generation means, the plane motion obtained by the plane motion / three-dimensional information restoration means, and the re-projection error obtained from the three-dimensional information by the coefficient δ. From the matrix data having the error as a matrix element, the plane motion obtained by the plane motion / three-dimensional information restoring means, the three-dimensional information, and the optical axis translation obtained by the optical axis motion restoring means, the optical axis Rotational motion restoration means for obtaining rotational motion around mutually orthogonal axes other than and updating the coefficient ε and the optical axis coordinate value by the restored rotational motion,
The observed coordinate values obtained by the asymptotic matrix generating means are converted into the conversion coefficient (coefficient ε) and the optical axis coordinate value updated by the rotational motion restoring means, and the conversion coefficient (coefficient δ) updated by the optical axis motion restoring means. ), The asymptotic value to the plane motion between the plane motion obtained by the plane motion / three-dimensional information restoration means and the reprojected coordinate value of the feature point for each frame image from the three-dimensional information. An asymptotic error is calculated, and processing for reducing the asymptotic error (camera motion asymptotically approaches planar motion) by switching between processing by the optical axis translational motion restoring means and processing in the rotational motion restoring means by increasing or decreasing the asymptotic error. Repetitive stabilization means,
A device for restoring camera motion and three-dimensional information, comprising: In claim 1,
In the feature point coordinate system set for the omnidirectional image, the means for inputting the image coordinate value of the feature point in each frame image, the azimuth angle (phase angle) from a reference axis from the image coordinate value, and the omnidirectional camera Means for obtaining an angle (elevation angle) from the optical axis direction obtained according to the projection method used; means for obtaining a coordinate value obtained by coordinate conversion using the phase angle and elevation angle as the observation coordinate value;
A device for restoring camera motion and three-dimensional information, comprising: In claim 1,
In the feature point coordinate system set for the time series image, means for inputting the image coordinate value of the feature point in each frame image, means for decomposing a matrix having the observed coordinate value as a matrix element, and the singular value A means for calculating a judgment value representing the degree of freedom of movement from the component, and if this judgment value is less than a certain value, it is regarded as a plane motion, and from rotation around the optical axis and movement on a plane perpendicular to the optical axis. Means for restoring three-dimensional information with three degrees of freedom, and if the determination value is greater than a certain value, restores six-degree-of-freedom movement and three-dimensional information consisting of rotational and translational motions of the camera. Means,
A device for restoring camera motion and three-dimensional information, comprising: In claim 1,
In the feature point coordinate system set for the omnidirectional image, the means for inputting the image coordinate value of the feature point in each frame image, the azimuth angle (phase angle) from a reference axis from the image coordinate value, and the omnidirectional camera Means for obtaining an angle (elevation angle) from the optical axis direction obtained according to the projection method used; means for obtaining a coordinate value obtained by coordinate conversion using the phase angle and elevation angle as the observation coordinate value;
Means for singular value decomposition of a matrix having the observed coordinate values as matrix elements, means for calculating a determination value representing the degree of freedom of movement from the component of this singular value, and when the determination value is less than a certain value, If the judgment value is greater than a certain value, it is considered to be a planar motion, means for restoring three-dimensional information with three degrees of freedom consisting of rotation around the optical axis and motion on a plane perpendicular to the optical axis. A means for restoring the three-dimensional information and the six-degree-of-freedom motion comprising the rotational motion and translational motion of the camera;
A device for restoring camera motion and three-dimensional information, comprising: A method for reconstructing three-dimensional information constituting the motion of the camera viewpoint in the time series and the object shape of the outside world from the temporal change amount of the image coordinate value regarding the feature point arranged in the target image in the time series image Because
In the feature point coordinate system set for the time series image, the image coordinate value (observation coordinate value) of the feature point in each frame image is input, and the rotation coordinate of the camera, the optical axis coordinate value, the translational motion, and An asymptotic matrix generating step for generating asymptotic matrix data having a coordinate value multiplied by a coefficient consisting of three-dimensional information as an element;
The asymptotic matrix data is subjected to singular value decomposition, noise removal is performed to obtain matrix data representing motion information and matrix data representing three-dimensional information, and the motion information satisfies the conditions set for defining the motion. A transformation matrix is obtained, and the transformation matrix is applied to the matrix data representing the motion information, so that the rotational motion around the optical axis with respect to the camera viewpoint and the translational motion on a plane perpendicular to the optical axis (a plane with three degrees of freedom consisting of these components). A plane motion / three-dimensional information restoring step for restoring the three-dimensional information constituting the object shape by applying an inverse matrix of the transformation matrix to the matrix data representing the three-dimensional information.
Coordinates obtained by converting the plane motion obtained in the plane motion / three-dimensional information restoration step, the reprojection error calculated from the three-dimensional information, and the observed coordinate value obtained in the asymptotic matrix generation step using the coefficient ε and the optical axis coordinate value. Matrix data having values as matrix elements is obtained, the matrix data obtained by singular value decomposition of the matrix data to remove noise, and the Z coordinate value (Z direction) of each feature point obtained in the plane motion / three-dimensional information restoration step. Optical axis motion that restores the translational motion in the optical axis direction of the camera viewpoint from the matrix whose element is the height of the feature point position when the is vertically oriented), and updates the coefficient δ by the restored optical axis translational motion A restore step,
An error between the observed coordinate value obtained in the asymptotic matrix generation step, the plane value obtained in the plane motion / three-dimensional information restoration step, and the re-projection error obtained from the three-dimensional information is obtained by a coefficient δ. From the matrix data having the error as a matrix element, the plane motion obtained in the plane motion / three-dimensional information restoration step, the three-dimensional information, and the optical axis translation obtained in the optical axis motion restoration step, the optical axis A rotational motion restoring step for obtaining rotational motion around mutually orthogonal axes other than and updating the coefficient ε and the optical axis coordinate value by the restored rotational motion,
The observed coordinate values obtained in the asymptotic matrix generation step are converted into the conversion coefficient (coefficient ε) and optical axis coordinate value updated in the rotational motion restoration step, and the conversion coefficient (coefficient δ) updated in the optical axis motion restoration step. ), The asymptotic value to the plane motion between the plane motion obtained in the plane motion / three-dimensional information restoration step and the reprojected coordinate value of the feature point for each frame image from the three-dimensional information. An asymptotic error representing the following is obtained, and processing for reducing the asymptotic error (camera motion asymptotically approaches planar motion) by switching the processing in the optical axis translational motion restoration step and the processing in the rotational motion restoration step by increasing or decreasing the asymptotic error. Repeated stabilization steps;
A method for restoring camera motion and three-dimensional information, comprising: In claim 5,
In the feature point coordinate system set for the omnidirectional image, the step of inputting the image coordinate value of the feature point in each frame image, the azimuth angle (phase angle) from a certain reference axis from the image coordinate value, and the omnidirectional camera Obtaining an angle (elevation angle) from the optical axis direction obtained according to the projection method used, obtaining a coordinate value obtained by coordinate conversion using the phase angle and the elevation angle, as the observed coordinate value;
A method for restoring camera motion and three-dimensional information, comprising: In claim 5,
In the feature point coordinate system set for the time series image, a step of inputting image coordinate values of feature points in each frame image, a step of singular value decomposition of a matrix having the observed coordinate values as matrix elements, and A step of calculating a judgment value representing the degree of freedom of movement from the component, and if this judgment value is less than a certain value, it is regarded as a plane motion, and is determined from a rotation around the optical axis and a movement on a plane perpendicular to the optical axis. If the determination value is greater than or equal to a certain value, the motion with the degree of freedom 6 consisting of the rotational motion and the translational motion and the 3D information are restored. Steps,
A method for restoring camera motion and three-dimensional information, comprising: In claim 5,
In the feature point coordinate system set for the omnidirectional image, the step of inputting the image coordinate value of the feature point in each frame image, the azimuth angle (phase angle) from a certain reference axis from the image coordinate value, and the omnidirectional camera Obtaining an angle (elevation angle) from the optical axis direction obtained according to the projection method used, obtaining a coordinate value obtained by coordinate conversion using the phase angle and the elevation angle, as the observed coordinate value;
A step of singular value decomposition of a matrix having the observed coordinate values as matrix elements, a step of calculating a determination value representing the degree of freedom of movement from the component of the singular value, and the determination value is less than a certain value, A step of restoring three-dimensional information with three degrees of freedom consisting of rotation around the optical axis and movement on a plane perpendicular to the optical axis as if it were a plane movement, Reconstructing the three-dimensional information and the six-degree-of-freedom motion comprising the rotational motion and translational motion of the camera;
A method for restoring camera motion and three-dimensional information, comprising: 9. A program configured to execute a processing procedure in the camera motion and three-dimensional information restoration method according to claim 5 by a computer.

Images