JP2005063012A - Full azimuth camera motion and method and device for restoring three-dimensional information and program and recording medium with the same recorded - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To highly precisely acquire and restore the shape of an object concerning an object from time series images acquired from a full azimuth camera. <P>SOLUTION: This device is configured to restore full azimuth camera view point motions in time series and three-dimensional information constituting the shape of an object in an external field from the secular change amounts of image coordinate values concerning featured points arranged in the target image in full azimuth images(or wide-angle view images) in time series acquired from the full azimuth camera. This device is provided with a featured point measuring part 11 to measure the coordinate values of the featured points in each image, a measurement matrix converting part 12 to convert the image coordinate values acquired by the featured point measuring part 11 into a uv image, an intersection coordinate measuring part 13 to calculate intersection coordinate values in the neighborhood of the center of projection in each image, a measurement matrix inputting part 14 to apply a matrix with the uv coordinate values as matrix elements as an initial value and a factorization processing part 15 to restore the motions of the camera and the three-dimensional information of the external field by factorization from data supplied from the intersection coordinate measuring part 13 and the measurement matrix inputting part 14. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention is used for vehicle-mounted images or indoor images acquired using a camera equipped with a fish-eye lens (hereinafter referred to as an omnidirectional camera), general time-series images such as sea images from a ship, aerial photography, etc. The three-dimensional shape of the outside world, that is, the subject (object), moving from the time-series image acquired by the omnidirectional camera to the omnidirectional camera viewpoint related to the roll, pitch, and yaw rotation, the distribution motion, and the time-series video. ) To restore the three-dimensional information constituting the external shape.

  In the field of computer vision, there are three-dimensional analysis methods using stereo measurement and epipolar plane analysis as methods for measuring or acquiring the shape of an object from time-series image data. According to this method, it is possible to restore the three-dimensional position information related to the spatial shape or the spatial structure and the motion related to the camera viewpoint from a plurality of time-series images in which the object is photographed.

  However, it is difficult to seamlessly acquire images in time-series images taken while moving the camera using moving means etc. due to the shooting environment and subtle movements of the camera. In some cases, it is difficult to accurately restore camera motion and object shape.

  Furthermore, the camera motion and object shape restoration in the Euclidean space can be performed using a factorization method (for example, Non-Patent Document 1 and Non-Patent Document 2) from image coordinate values that can be expressed in an orthogonal coordinate system with a flat image surface. Although it is possible, when a camera with a finite field angle is used, it is difficult to restore the delicate rotational motion of the camera, and the translation and rotational motion may not be accurately discriminated.

On the other hand, in order to avoid a finite field angle, imaging with an omnidirectional camera is conceivable as means for acquiring a landscape of 360 degrees at a time. An omnidirectional camera is often used with its camera optical axis set vertically upward, and is suitable for posture estimation of an omnidirectional camera viewpoint and space measurement related to the outside world (for example, Non-Patent Document 3). In addition, by using an omnidirectional camera, it is possible to restore a minute rotational motion, which is excellent in that both translational motion and rotational motion can be accurately restored.
C. tomasi and T.M. Kanade; “Shape and Motion from Image Stream Under Orthography: A Factorization Method”, International Journal of Computer Vision, Vol. 9, no. 2, 1992. C. J. et al. Poelman and T.W. Kanade; "A Paraperspective Factor Method for Shape and Motion Recovery", IEEE trans. Pattern Anal. and Mach. Intel, Vol. 19, no. 3, pp. 206-218, 1997. Miyakawa, Wakabayashi, Arikawa; “Motion and shape restoration from omnidirectional images by omnidirectional projection factorization”, Pattern Recognition and Media Understanding (PRMU) Study Group, June, 2002

  Acquire spatial information of objects in the outside world using a measuring method that applies the principle of stereo vision in a time-series image acquired in a shooting environment that visualizes a 360-degree landscape at once, such as an omnidirectional camera using a moving means. However, in moving shooting in various shooting environments, since the camera moves minutely, a seamless time-series image cannot be easily acquired. For this reason, the influence of random noise is large, and it is difficult to restore the camera movement and the shape of the object simultaneously and with high accuracy at all times.

  For example, when the method of Non-Patent Document 3 is applied, as shown in FIG. 13, in addition to the optical axis rotation (yaw rotation) of the omnidirectional camera, roll rotation, pitch rotation, and translational motion in the Z-axis direction are applied. Therefore, it was difficult to restore accurate motion and three-dimensional shape. That is, as shown in FIG. 13, the omnidirectional camera is often used in a posture in which the optical axis is set vertically upward, and therefore, a plane motion with 3 degrees of freedom consisting of yaw rotation and XY translational motion is restored. The main focus was placed.

  However, in the real environment, when roll rotation, pitch rotation, and Z-axis translational motion are included in addition to such planar motion, simple calculations using an omnidirectional camera cannot be used, and the camera motion and three-dimensional shape are restored. The calculation of the above becomes complicated, and the minute camera posture change related to the omnidirectional camera as described above is mixed with noise, and there is a problem that it cannot be accurately and robustly restored.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide an omnidirectional camera motion and three-dimensional image that can acquire and restore an object shape related to an object with high accuracy from a time-series image acquired from an omnidirectional camera. The present invention provides an information restoration method, an apparatus and a program thereof, and a recording medium on which the information is recorded.

  In order to solve the above-described problems, the omnidirectional camera motion and three-dimensional information restoration method, apparatus and program thereof, and the recording medium on which the omnidirectional camera is recorded are the coordinates of the feature points in the omnidirectional image acquired by the omnidirectional camera. Using the factorization method, the rotational motion around the optical axis (of the viewpoint) and the translational motion on the XY plane (plane motion) of the omnidirectional camera are obtained from the temporal change of After calculating the rotational motion around the axis other than the optical axis (of the viewpoint) of the omnidirectional camera from the image coordinate value (intersection coordinate value) of the intersection point formed by the line segment near the projection center in each image Is used to determine the translational motion of the omnidirectional camera (viewpoint) in the Z-axis direction, further to calculate the three-dimensional shape of the target object, and repeatedly calculate until these errors are less than or equal to a predetermined value. 3rd order of the object And restoring the information and (viewpoint) 6 freedom of movement of the omnidirectional camera.

That is, the invention according to claim 1 is based on the temporal change amount of the image coordinate value related to the feature point arranged in the target image in the time-series omnidirectional image (or wide-field image) acquired from the omnidirectional camera. , The movement of the omnidirectional camera viewpoint in time series, and the method of restoring the three-dimensional information constituting the object shape of the outside world,
In the feature point coordinate system set for the acquired time-series omnidirectional image (or wide-field image), the image coordinate value indicating the temporal change amount of the feature point in each image and the line segment near the projection center in each image A step (measurement matrix measurement step) of measuring an image coordinate value (intersection coordinate value) of an intersection formed by
Different coordinate values (uv coordinate values) are obtained from the phase angle and elevation angle calculated from the image coordinate values of each feature point obtained in this measurement matrix measurement step, and the uv coordinate values and intersection coordinate values of these feature points are obtained. Is used to generate matrix data (matrix decomposition data) for restoration processing, singular value decomposition of this matrix decomposition data, and noise removal to obtain matrix data representing motion information and matrix data representing three-dimensional information From the motion information component, a transformation matrix that satisfies the conditions set to define the motion is obtained, and this transformation matrix is applied to the matrix data that is motion information to rotate around the optical axis of the omnidirectional camera viewpoint. Restoring the motion and translational motion on the plane perpendicular to it (XY plane) (this motion with 3 degrees of freedom is called plane motion), and using the rotation angle around the optical axis and the intersection coordinate value, Restore rotational movement around the axis And, further, a step (restoration step of planar motion and three-dimensional information) to restore the three-dimensional information that constitutes the object shape by applying an inverse matrix of the transformation matrix matrix data representing the three-dimensional information,
The re-projection coordinate value in the uv coordinate system for each feature point is obtained from the plane interlock and the three-dimensional information obtained in the plane motion and the three-dimensional information restoration step, and the reprojected coordinate value and matrix elements constituting the matrix decomposition data Using this error and the Z coordinate value (component of 3D information) of the feature point obtained in the plane motion and 3D information restoration step, the translation of the camera viewpoint in the Z-axis direction is calculated. A step of restoring motion (Z-axis translational motion restoration step);
It is determined whether or not the error obtained by the Z-axis translational motion recovery snubber has converged to a certain value or less. If not, the planar motion obtained by the planar motion and the three-dimensional information restoration step and 3 Determining the deformation motion from the dimensional information and the rotational motion around the axis other than the optical axis, combining this deformation coefficient with the uv coordinate value of each feature point to determine a new uv coordinate value, and using this as a matrix element The method includes generating decomposition data, returning to the plane motion and three-dimensional information restoration step, and repeating until the error converges to a predetermined value or less to restore the omnidirectional camera motion and the three-dimensional information.

According to the second aspect of the present invention, in the time-series omnidirectional image (or wide-field image) acquired from the omnidirectional camera, the temporal change amount of the image coordinate value regarding the feature point arranged in the target image is This is a method for restoring the omnidirectional camera viewpoint movement in the sequence and the three-dimensional information constituting the object shape of the outside world. For each time-series omnidirectional image (or wide-field image) acquired, each image coordinate value A time series image (uv image) is generated by converting each image coordinate value into a uv coordinate value by the phase angle and the elevation angle obtained from the above, and associating each pixel with the uv coordinate value, and the feature points set in this uv image In the coordinate system, a step of measuring a uv coordinate value indicating a temporal change amount of a feature point in each image and a uv coordinate value (intersection coordinate value) of an intersection formed by a line segment in the vicinity of the projection center in each uv image (uv) Measurement matrix measurement step),
Generate matrix data (matrix decomposition data) for restoration processing using the uv coordinate value of each feature point obtained in the uv measurement matrix measurement step and the intersection coordinate value, and singular value decomposition of the matrix decomposition data, The matrix data representing the motion information and the matrix data representing the three-dimensional information are obtained by performing noise removal, and a transformation matrix satisfying the conditions set for defining the motion is obtained from the motion information components, and the motion information and This transformation matrix is applied to the matrix data to restore the rotational motion around the optical axis of the omnidirectional camera viewpoint and the translational motion on this plane (XY plane) (this motion with 3 degrees of freedom is called planar motion). Then, using the rotation angle around the optical axis and the coordinate value of the intersection point, the rotational motion around the other axis other than the optical axis is restored, and the inverse matrix of this transformation matrix is applied to the matrix data representing the three-dimensional information. Configure object shape And step (restoration step of planar motion and three-dimensional information) to restore the three-dimensional information that,
A reprojection coordinate value in the uv coordinate system for each feature point is obtained from the plane motion and 3D information obtained in the plane motion and 3D information restoration step, and the reprojected coordinate value and matrix elements constituting matrix decomposition data are obtained. Using this error and the Z coordinate value (component of 3D information) of the feature point obtained in the plane motion and 3D information restoration step, the translational motion of the camera viewpoint in the Z-axis direction is calculated. Restoring (Z-axis translational motion restoring step),
It is determined whether or not the error obtained in the Z-axis translational motion restoration step has converged to a certain value or less. If the error has not converged, the planar motion obtained in the planar motion and three-dimensional information restoration step and 3 Deformation coefficient is obtained from dimensional information and rotational motion around an axis other than the optical axis, and this deformation coefficient is combined with the uv coordinate value of each feature point to obtain a new uv coordinate value, and matrix decomposition using this as a matrix element The method includes generating data, returning to the plane motion and three-dimensional information restoration step, and repeating until the error converges to a certain value or less to restore the omnidirectional camera motion and the three-dimensional information.

The invention according to claim 3 is the omnidirectional camera motion and three-dimensional information restoration method according to claim 1, wherein, in the measurement matrix measurement step, the degree of freedom of motion from the singular value component obtained by singular value decomposition of the measurement matrix. A judgment value representing
When the determination value is less than a certain value, the camera motion is regarded as a plane motion, and in the step of restoring the omnidirectional camera motion and the three-dimensional information, the rotation around the optical axis and the plane perpendicular to the optical axis are performed. Perform 3 degrees of freedom of plane motion consisting of the above motion and processing to restore 3D information,
If the judgment value is greater than a certain value, the camera motion is regarded as a general motion other than a plane motion, and the omnidirectional camera motion and the three-dimensional information are restored, and the free motion consisting of rotation and translational motion about the camera A step of performing a process of restoring the degree 6 motion and the external three-dimensional information.

According to a fourth aspect of the present invention, in the omnidirectional camera motion and three-dimensional information restoration method according to the second aspect, in the uv measurement matrix measurement step, the motion is expressed from a singular value component obtained by singular value decomposition. Calculate the judgment value,
When the determination value is less than a certain value, the camera motion is considered to be a plane motion, and in the step of restoring the omnidirectional camera motion and the three-dimensional information, the rotation around the optical axis and the direction perpendicular to the optical axis are performed. Perform 3 degrees of freedom of plane motion consisting of motion on a plane and processing to restore 3D information,
If the judgment value is greater than a certain value, the camera motion is regarded as a general motion other than a plane motion, and the omnidirectional camera motion and the three-dimensional information are restored, and the free motion consisting of rotation and translational motion about the camera A step of performing a process of restoring the degree 6 motion and the external three-dimensional information.

According to the fifth aspect of the present invention, in the time-series omnidirectional image (or wide-field image) acquired from the omnidirectional camera, the temporal change amount of the image coordinate value regarding the feature point arranged in the target image is A device that restores the movement of the omnidirectional camera viewpoint in the series and the three-dimensional information constituting the object shape of the outside world,
A feature point measurement unit that measures coordinate values of feature points in each image;
A measurement matrix conversion unit that converts the image coordinate values obtained by the feature point measurement unit into a uv image;
An intersection coordinate measurement unit for obtaining an intersection coordinate value near the projection center in each image;
a measurement matrix input unit that gives a matrix having uv coordinate values as matrix elements as an initial value;
A factorization processing unit that restores the motion of the camera and the three-dimensional information of the outside world from the data supplied from the intersection coordinate measurement unit and the measurement matrix input unit by a factorization method;

According to the sixth aspect of the present invention, in the time-series omnidirectional image (or wide-field image) acquired from the omnidirectional camera, the temporal change amount of the image coordinate value related to the feature point arranged in the target image is A device that restores the movement of the omnidirectional camera viewpoint in the series and the three-dimensional information constituting the object shape of the outside world,
An image conversion processing unit for converting each image into a uv image;
A feature point measurement unit that measures coordinate values of feature points in each image;
An intersection coordinate measurement unit for obtaining an intersection coordinate value near the projection center in each uv image;
a measurement matrix input unit that gives a matrix having uv coordinate values as matrix elements as an initial value;
A factorization processing unit that restores the motion of the camera and the three-dimensional information of the outside world from the data supplied from the intersection coordinate measurement unit and the measurement matrix input unit by a factorization method;

According to a seventh aspect of the present invention, in the omnidirectional camera motion and three-dimensional information restoration device according to the fifth aspect, the motion is performed from the singular value component obtained by singular value decomposition of the uv image supplied from the measurement matrix converter. A restoration processing determination unit that calculates a determination value representing the degree of freedom of
When the determination value is less than a certain value, it is considered that the camera motion is a plane motion, and the factorization processing unit has a degree of freedom 3 consisting of a rotation around the optical axis and a motion on a plane perpendicular to the optical axis. To restore the plane motion and 3D information of
If the determination value is equal to or greater than a certain value, it is assumed that the camera motion is a general motion other than a plane motion, and the factorization processing unit performs a motion of 6 degrees of freedom consisting of rotation and translational motion related to the camera and 3 of the external world. A process for restoring the dimension information is performed.

According to an eighth aspect of the present invention, in the omnidirectional camera motion and three-dimensional information restoration device according to the sixth aspect, the motion is performed from a singular value component obtained by singular value decomposition of the uv image supplied from the feature point measurement unit. Including a restoration process determination unit that calculates a determination value representing
When the judgment value is less than a certain value, it is considered that the camera motion is a plane motion, and the factorization processing unit has a degree of freedom comprising a rotation around the optical axis and a motion on a plane perpendicular to the optical axis. 3 to restore the plane motion and 3D information,
If the determination value is equal to or greater than a certain value, it is assumed that the camera motion is a general motion other than a plane motion, and the factorization processing unit performs a motion of 6 degrees of freedom consisting of rotation and translational motion related to the camera and 3 of the external world. A process for restoring the dimension information is performed.

  In the above invention, the wide-field image is a technical term synonymous with the omnidirectional image (hereinafter, the same applies to embodiments of the invention described later).

  The uv coordinate value is a value obtained by dividing a three-dimensional coordinate value of the external world onto a spherical surface centered on the viewpoint, and a value obtained by dividing the orthogonal direction component of the moving direction of the camera by the height direction component, and the moving direction component of the camera Is obtained by dividing the height component by the height direction component, and obtained as the u coordinate and the v coordinate, respectively.

  The deformation coefficient represents the degree of deformation when the translational motion in the Z-axis direction (of the viewpoint) of the omnidirectional camera and the rotational motion around other axes other than the optical axis are expressed by deformation of the uv coordinate value equivalent to them. It is a coefficient.

  The omnidirectional camera motion and three-dimensional information restoration method and apparatus of the present invention can be realized by a computer and a program, and the program can be recorded on a computer-readable recording medium or provided through a network. . Examples of the recording medium include FD (floppy disk (registered trademark)), MO, ROM, memory card, CD, DVD, and removable disk.

  According to the omnidirectional camera motion and three-dimensional information restoration method, apparatus and program thereof, and the recording medium on which the omnidirectional camera movement of the present invention is recorded, all time-series images acquired using the omnidirectional camera (photographed using moving means) The object shape related to the object can be obtained and restored with high accuracy from the in-vehicle image, the sea image, the aerial image, the indoor image, and the like. In addition, high-accuracy three-dimensional stereoscopic viewing similar to conventional surveying techniques is possible.

  In particular, when applied to a vehicle-mounted camera, the camera vibrates with the traveling of the car. However, even in such a situation, the present invention is robust against noise, minute camera posture fluctuations (rotation around three axes), and It becomes possible to accurately measure translational motion with accuracy for interpolating a remote sensor such as GPS. In addition, since most of the operations used in the present invention are composed of linear operations, they can be easily implemented in a computer language.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(Embodiment 1)
1a and 1b are schematic configuration diagrams showing an embodiment of an apparatus for carrying out the invention of claim 1. FIG. In the present embodiment, a fish-eye camera (hereinafter referred to as a camera) using a fish-eye lens optically designed by equidistant projection (projection proportional to the incident angle from the optical axis) may be abbreviated as a camera. (Referred to as an omnidirectional camera). Note that the present invention can also be applied to a time-series image acquired by a fisheye camera lens imaging apparatus using a fisheye lens designed by another projection (solid angle projection, equal solid angle projection).

Referring to FIG. 13, the point P (X _j , Y _j , Z _j ) in the space to be restored in this embodiment and the movement of the omnidirectional camera, that is, roll rotation (ω _i ), pitch rotation (φ _i ), yaw rotation (θ _i ) and translational movement T _i (Tx _i , Ty _i , Tz _i ) will be described.

This figure shows the positional relationship between the camera and the object (subject). For convenience of explanation, the axis along which the camera travels is the Y axis. The camera rotates by roll rotation (ω _i ), pitch rotation (φ _i ), and yaw rotation (θ _i ) while moving by translational motion (Tx _i , Ty _i , Tz _i ). At this time, the camera optical axis is the Z-axis direction. The position of the projection center C (viewpoint position T _i ) is the center of motion, and the translational motion in the i-th frame is the position of Ti (Tx _i , Ty _i , Tz _i ). Furthermore, the rotation around the traveling axis is roll rotation (rotating around the Y axis), the rotation around the camera optical axis is the yaw rotation (rotating around the Z axis), and around the normal of the surface stretched between the camera optical axis and the traveling axis. The rotation is referred to as pitch rotation (rotation around the X axis). On the other hand, the point P (X _j , Y _j , Z _j ) in the space is projected onto the image coordinate value (x _ij , y _ij ) on the image plane in the image circle. Expression (1) is an optical projection called equidistant projection, and the phase angle ρ _ij shown in FIG. 13 can be obtained by calculation from the image coordinate values (x _ij , y _ij ) to Expression (2). . In equation (1), f is the focal length of the fisheye lens, and D is a constant determined by D = πf / (2R). Here, R is the radius of the image circle shown in FIG. In FIG. 13, the image coordinate value of the projection center C on the image plane is (C _x , C _y ).

  The restoration device shown in FIG. 1a includes a time-series database 10 in which omnidirectional images are accumulated, a feature point measurement unit 11 that measures coordinate values of feature points in each image, and image coordinate values obtained by the measurement unit 11 Is a measurement matrix conversion unit 12 that converts a uv image into a uv image, an intersection coordinate measurement unit 13 that obtains an intersection coordinate value near the projection center in each image, and a measurement matrix that provides a matrix having uv coordinate values as matrix elements as initial values An input unit 14 and a factorization method processing unit 15 that restores camera motion and three-dimensional information of the external world from these input data by a factorization method are provided. The time-series image database 10 may use a recording medium such as a hard disk, a RAID device, a CD-ROM, or a remote data resource via a network.

  FIG. 1b is a schematic configuration diagram showing an example of a restoration device in the case of processing in real time. The restoration apparatus includes a data image input unit 16 instead of the time-series image database 10. Therefore, the restoration apparatus of this embodiment does not necessarily require storage means such as each database unit.

  Hereinafter, each component of the present embodiment will be described.

The feature point measurement unit 11 takes out an omnidirectional image capturing a landscape from the time-series image database 10 and sets a feature point on the omnidirectional screen. For this feature point setting, pixels with a large gradient value due to a one-dimensional or two-dimensional change in shading are detected (edge detection, etc.), and feature points are automatically generated by image processing techniques such as Canny operator and Hough transform. I am letting. Further, the feature points may be arranged on the initial image by appropriately marking the feature points with an operator intervening. At this time, the number of feature points to be arranged is P (j = 1, 2,..., P), and the two-dimensional coordinate values of the feature points at the time of placement from the origin in the image coordinate system in FIG. Coordinate values (x _1j , y _1j ), j = 1, 2,..., P are recorded. Next, a time series image following the initial image is read from the database 10 one by one, and feature points arranged in the initial image are converted to a method that focuses on a change in shading between the time series images, and a pixel pattern centered on the feature points. Tracking is performed by using a method of tracking between frames (for example, KLT (Kanade-Lucas-Tomasi) method) or a method considering projection distortion peculiar to an omnidirectional image, and each time series image (i-th image from the initial image). The two-dimensional coordinate values (x _ij , y _ij ) from the origin in the image coordinate system in FIG. 13 are recorded as image coordinate feature points as image coordinate values. Further, when the time series image is continuously read, if the feature point is lost between images due to disappearance or occlusion, the tracking is stopped and the feature point tracking is ended. When the feature point tracking is completed, the number of time-series images read out is F including the initial image (i = 1, 2,..., F). The feature point measurement unit 11 stores the measurement matrix [A] by arranging the amount of change in the temporal image coordinate arrangement of the feature points in each time-series image in the data format of Expression (3).

The measurement matrix conversion unit 12 obtains the image coordinate values (x _ij , y _ij ) of the j-th feature point in the i-th frame from the measurement matrix [A] having the image coordinate values acquired by the feature point measurement unit 11 as elements. Take out. Further, the phase angle ρ _ij and the elevation angle φ _ij are obtained by temporarily converting into relative image coordinate values with the projection center (C _x , C _y ) as the origin, and by the calculations of equations (1) and (2). The phase angle and the elevation angle are subjected to coordinate conversion by calculation according to Equation (4) to obtain uv coordinate values (u _ij , v _ij ). This conversion is performed for all frames and all feature points. As a data format after conversion, the matrix [B] in the element format shown in Expression (5a) is held as an initial value in the measurement matrix input unit 14.

  The obtained u-coordinate value and v-coordinate value are orthogonal to each other in the advancing direction of the camera in the coordinate values obtained by projecting the three-dimensional coordinate values (X, Y, Z) of the external world onto the spherical surface with the viewpoint Ti as the center. This corresponds to a value obtained by dividing the direction component by the height direction component and a value obtained by dividing the moving direction component of the camera by the height direction component. That is, when the three-dimensional coordinate value (X, Y, Z) of the outside world is projected onto a spherical surface having a radius l centered on the viewpoint Ti, (X ′, Y ′, Z ′) = (cos (φ) cos (ρ ), Cos (φ) sin (ρ), sin (φ)) (see FIG. 13). The uv coordinate value is calculated from the unit spherical coordinate value (X ′, Y ′, Z ′) to (u, v) = (X ′ / Z ′, Y ′ / Z ′) = (cot (φ) cos (ρ), cot (φ) sin (ρ)). In the general spherical coordinate system, since the coordinates are defined using the angle φ ′ measured from the Z axis in FIG. 13, (X ′, Y ′, Z ′) = (sin (φ ′) cos (ρ ), Sin (φ ′) sin (ρ), cos (φ ′)).

  On the other hand, in this embodiment, since the coordinates are defined using the elevation angle φ when looking up the three-dimensional coordinate value (X, Y, Z) from the viewpoint in FIG. 13, φ ′ = π / 2. −φ and (sin (π / 2−φ) cos (ρ), sin (π / 2−φ) sin (ρ), cos (π / 2−φ)) = (cos (φ) cos (ρ ), Cos (φ) sin (ρ), sin (φ)).

  FIGS. 12a (1), (2), and (3) show examples of images taken by the omnidirectional camera. In the figure, a (1) is an example of an image taken by an omnidirectional camera and an object is attached with an outline, and a (2) is an example of an image taken by an omnidirectional camera and only the outline of the object. A (3) is an example of an image captured by an omnidirectional camera and does not have a contour line of an object.

  FIGS. 12 b (1), (2), and (3) show examples in which the image of FIG. 12 a (1) is converted into uv coordinate values according to the processing flow disclosed in FIG. 5 and the uv image shown in FIG. 6 is generated. It is a thing. In FIG. 12, b (1) is an example of an image obtained by uv coordinate conversion of an image captured by an omnidirectional camera, and an object is attached with a contour line, and b (2) is an image captured by an omnidirectional camera. B (3) is an example of an image obtained by performing uv coordinate conversion on an image captured by an omnidirectional camera. The outline is not attached.

  As is clear from FIGS. 12 b (1), (2), and (3), in this coordinate conversion, the measurement error is emphasized at a point farther from the v-axis as the feature point (point closer to the bottom of the building) with a smaller elevation angle φ is converted. Is done. That is, when feature points are measured by the same operation, errors that are difficult to understand on the xy plane can be grasped more accurately on the uv plane. Therefore, if the movement of the camera viewpoint and the three-dimensional coordinate value are obtained so that the emphasized error is minimized, highly accurate measurement can be performed.

  Further, in the uv plane, the roll rotation movement can be converted into the coordinate movement in the v direction, the pitch rotation movement can be converted into the coordinate movement in the u direction, and the Z-axis translation movement can be simply converted into scale conversion in the uv coordinate axis system. The rotation, pitch rotation, and Z-axis translation can be easily distinguished. By utilizing this ease of discrimination, in this embodiment, the uv coordinate value is repeatedly deformed using a deformation coefficient, and the camera motion and the three-dimensional coordinate value are continuously restored, so that a more accurate camera can be obtained. It is possible to restore motion and three-dimensional coordinate values.

  The intersection coordinate measurement unit 13 detects a line segment in the image used in the feature point measurement unit 11, and measures coordinates (intersection coordinate) at which a straight line on the line segment intersects in the vicinity of the projection center.

The intersection coordinates will be described with reference to FIG. As shown in FIG. 13, when an omnidirectional camera is photographed in the sky direction or ceiling direction, a vertical line segment, for example, a line segment in the height direction such as a wall indoors, or a line segment in the height direction of the building outdoors is stretched. And have the property of intersecting at intersections as shown in FIG. That is, there is one point having an infinite height, and this point is obtained from the image as an intersection coordinate. Here, the intersection coordinates are obtained by detecting a line segment (vertical line segment) in the height direction of the building in the image. The line segment is detected by an image processing method such as a Hough transform or a Canny operator. A straight line parameter relating to each line segment is obtained by such processing. The straight line parameters are the straight line slope a _i and the intercept b _i , and the intersection coordinate values (D _x , D _y ) are obtained by the calculation shown in the equation (6a). Alternatively, the operator may specify and obtain a point on a straight line.

FIG. 3 shows a pair on a straight line l ₁₂ : p ₁ (x ₁ , y ₁ ) and p ₂ (x ₂ , y ₂ ), and a pair on a straight line l ₃₄ : p ₃ (x ₃ , y ₃ ) and p ₄ (X ₄ , y ₄ ), pair on line l ₅₆ : p ₅ (x ₅ , y ₅ ) and p ₆ (x ₆ , y ₆ ), pair on line l ₇₈ : p ₇ (x ₇ , y ₇ ) And p ₈ (x ₈ , y ₈ ). The intersection coordinate value (D _x , D _y ) is calculated from the coordinate value of each point by the calculation of equation (6b). Furthermore, a value obtained by multiplying the intersection coordinate value (D _x , D _y ) by D / f by the calculation of Expression (6c) is set as the intersection coordinate value. Note that at least two straight lines are sufficient to calculate the intersection coordinates, but more accurate intersection coordinates can be calculated by detecting many vertical straight lines. The intersection coordinates are obtained over all frames, and the intersection coordinate values (ζ _i , η _i ), i = 1, 2,.

Next, the processing outline of the factorization method processing unit 15 will be described. FIG. 2 is a flowchart illustrating the processing steps. When the measurement matrix data is provided from the measurement matrix input unit 14 and the intersection coordinate values are provided from the intersection coordinate measurement unit 13, first, the coefficients ε _ij and δ _ij are initialized as shown in the equation (5b), and a matrix of initial values is obtained. Data shown in the equation (5c) is obtained from [B] and the intersection coordinate value, and this is held again as a matrix [B] (step 101). At this time, the matrix [B] of the processing values held in the measurement matrix input unit 14 is not overwritten and is held until this processing step (FIG. 2) is completed. This matrix [B] becomes decomposition matrix data.

  The matrix [B] is subjected to singular value decomposition (step 102) to be decomposed into three matrices [U], [W], and [V] shown in Expression (7). Here, [U] is a 2F × P size matrix, [W] is a P × P size diagonal matrix, and [V] is a P × P size matrix.

  Further, in noise removal (step 103), as shown in the second term of equation (8), the components of each matrix of rank 4 or higher are deleted. At the time of this deletion, the matrix [U] is taken out, the fourth to Pth columns are deleted from the elements of this matrix, the matrix consisting of the remaining components is held, the matrix [W] is taken out, The fourth to Pth rows and the fourth to Pth columns are deleted in the element, the matrix composed of the remaining components is retained, the matrix [V] is extracted, and the fourth to Pth rows are extracted from the elements of this matrix. Delete up to the eye and hold the matrix composed of the remaining components (Equation (9)). Next, from the matrix obtained by taking the square of the diagonal elements of the matrix [W] from which the 4th to Pth rows and the 4th to Pth columns are deleted, the matrices shown in Expression (10) and Expression (ll) [U ′] and matrix [V ′] are obtained.

  Next, in the transformation matrix calculation (step 104), the held matrix [U ′] is taken out, and the symmetric matrix [C] in the simultaneous conditional expressions shown in Expressions (13) to (15) (Expression (12)). Calculate the coefficients for each element (shown). These coefficient calculations are easily obtained by matrix operation, and this conditional expression is calculated for all frames. Next, each element of the 3 × 3 size matrix [C] is determined using numerical calculation such as the least square method with respect to the simultaneous conditional expressions shown in Expressions (13) to (15) of all frames. . Further, the obtained matrix [C] is subjected to eigenvalue decomposition as shown in Expression (16). Here, a matrix [C ′] of Expression (17) is generated from the square of the eigenvalue matrix and the eigenvalue matrix, and a matrix [Q] having this matrix element as a component is calculated according to Expression (18).

  Subsequently, the matrix [M ′] is calculated from the obtained matrix [Q] and the stored matrix [U ′] by the matrix operation of Expression (19).

Then, the matrix element (m _ix , n _ix ) or (m _iy , n _iy ) of each frame (i-th frame) is extracted from the matrix [M ′], and the yaw rotation θ _i is restored using equation (20). To do. The pitch rotation ψi and the roll rotation ωi are restored from the matrix [M ′] using the equation (21) from the intersection coordinates (ζ _i , η _i ) of each frame (i-th frame). The above is step 106. Further, the matrix element (T _iu , T _iv ) of each frame (i-th frame) is extracted from the matrix [M ′]. From this (T _iu , T _iv ), the XY translational motion (Tx _i , Ty _i ) in the Euclidean space in the i-th frame is calculated using equation (22) (step 105). Further, the error shown in equations (25) and (26) is calculated.

Next, the calculation of equation (27), to restore the translation Tz _i of Z-axis in each frame (i-th frame) (step 108).

On the other hand, the matrix operation shown in Expression (23) is performed from the previously held matrix [V ′] and the matrix [Q] obtained by the conversion matrix calculation to obtain the matrix [S ′]. Then, the transformation shown in the equation (24) is performed on the elements of the matrix [S ′], and the matrix having this as an element is set to [P] (step 107). Each column vector of the matrix [P] is a three-dimensional coordinate value (X _j , Y _j , Z _j ) in the Euclidean space of the j-th feature point.

Here, the error ΔW obtained by the equations (25), (28) and (28a) is calculated (step 109). If this error has converged below a certain value (yes), the process is terminated. On the other hand, if it has not converged (no), the deformation coefficients ε _ij and δ _ij are updated according to the equation (29) (steps 111 and 112). Further, with the update of the deformation coefficients ε _ij and δ _ij , the matrix [B] serving as the matrix decomposition data is updated by the equation (5c). This is repeated until the error ΔW converges below a certain constant value (symbol A shown in FIG. 2).

The deformation coefficients ε _ij and δ _ij are those of the omnidirectional camera (from the viewpoint). This is a coefficient representing the degree of deformation when the translational motion in the Z-axis direction and the rotational motions ωi, ψi around other axes other than the optical axis are represented by deformations of uv coordinate values equivalent to them. In this embodiment, the XY translation components (Txi, Tyi) and the three-dimensional information (Xj, Yj, Zj) of the camera viewpoint obtained in the plane motion and the three-dimensional information restoration step, and the XY axes around the XY axes other than the optical axis. Using the rotation angle (ωi, ψi), a coefficient (ε coefficient; ε _{ij in} equation (29)) for deforming the uV coordinate value is obtained, and each feature point obtained in the plane motion and three-dimensional information restoration step is calculated. A coefficient for converting the uv coordinate value (δ coefficient; equation (29)) using the height information (Zj) and the position (Tzi) of each viewpoint on the Z axis obtained in the step of restoring the Z-axis translational motion. Δ _ij ). Then, the value obtained by multiplying each matrix element of the measurement matrix ([B] in equation (5a)) obtained in the measurement matrix measurement step by the ε coefficient and the measurement matrix obtained in the measurement matrix measurement step Matrix decomposition data ([B] in equation (5c)) that is a new matrix element obtained by combining each coordinate value obtained by subtracting the intersection coordinate value (ζ, η) from each matrix element and the value obtained by multiplying the respective δ coefficients. ) Is generated.

  As described above, according to the restoration device of the present embodiment, from the temporal movement of the feature point of the omnidirectional video, the movement of the omnidirectional camera viewpoint, that is, the rotation around the three axes and the translational movement in the three axial directions, The three-dimensional information constituting the object shape can be restored.

(Embodiment 2)
FIG. 4A is a schematic configuration diagram showing an example of an apparatus for carrying out the invention of claim 2. The restoration device according to the present embodiment includes an image conversion processing unit 21 in the restoration device according to the first embodiment. Here, the point that the image taken out from the time-series image database 10 is used for the image conversion processing unit 21 to generate a deformed image (uv image), and the point at which the intersection coordinates are measured in this uv image is the restoration of the first embodiment. Since this is different from the apparatus, only this point will be described.

  The illustrated restoration device includes a time-series image database 10 in which omnidirectional images are accumulated, an image conversion processing unit 21 that converts each image into a uv image, and a feature point measurement unit that measures coordinate values of feature points in each image. 11, an intersection coordinate measurement unit 13 that obtains an intersection coordinate value near the projection center in each uv image, a measurement matrix input unit 14 that provides a matrix having uv coordinate values as matrix elements as initial values, and input data A factorization method processing unit 15 is provided that restores camera motion and three-dimensional information of the outside world by a factorization method. As in the first embodiment, the time-series image database 10 may be a recording medium such as a hard disk, a RAID device, a CD-ROM, or a remote data resource via a network. Absent.

  FIG. 4B shows a processing configuration for processing in real time, and includes an image input unit 16 instead of the time-series image database 10. Therefore, the restoration apparatus of this embodiment does not necessarily require storage means such as each database unit.

  In the present embodiment, the image conversion processing unit 21 generates a uv image from the image extracted from the time-series image database 10. In order to generate a uv image, the image conversion processing unit 21 designates a uv coordinate value, calculates an xy coordinate value corresponding to this, extracts a pixel value of the coordinate value, and uses it as a uv pixel (see FIG. 6).

FIG. 5 is a flowchart showing this processing step. First, weighting per unit pixel in the u direction and v direction in the uv image is performed. Assume that the image size of the uv image is U × V pixels, and that the elevation angle φ is determined to be in a range of φ _min ≦ φ ≦ 90 degrees (steps 201 and 202). Thereby, the weighting per unit pixel in the u direction and the v direction is determined according to the equation (30).

  Further, the uv coordinate value range of the uv image is set from the center of the image to the origin (0, 0), from −U / 2 to + U / 2 in the u direction, and from −V / 2 to + V / 2 in the v direction. Then, integers m and n representing the coordinate values in the u direction and the v direction (−M ′ ≦ m ≦ M, −N ≦ n ≦ N, M: integer, N: integer, M and N are the same, that is, a uv image. The uv coordinate value is obtained by the calculation of the equation (31) (step 203).

  Next, while changing the integers m and n in the range of −M ≦ m ≦ M and −N ≦ n ≦ N, the phase angle ρ and the elevation angle corresponding to the (u, v) coordinate value are calculated by the equation (32). φ is obtained (steps 204 and 205).

  Next, an image coordinate value is calculated by Expression (33). A value obtained by rounding the image coordinate value to an integer is set to (x, y) (step 206).

  Then, a pixel value corresponding to the image coordinate value (x, y) is extracted (step 207), and the pixel value is embedded in a coordinate value corresponding to the uv coordinate value on the uv image (step 208).

  By performing the above process for all uv coordinate values, a uv image is generated (step 209). Further, this conversion process is sequentially performed on all target time-series images to generate time-series uv images.

The intersection coordinate measurement unit 13 detects a line segment in the uv image and measures a uv coordinate (intersection coordinate) at which a straight line on the line segment intersects in the vicinity of the projection center. FIG. 7 explains the measurement of the intersection coordinates (ζ, η) on the uv image. Similarly to FIG. 3, in the uv image, when the omnidirectional camera is photographed in the sky direction or the ceiling direction, a vertical line segment, for example, a line segment in the height direction of a wall or the like indoors, or a building height outdoors. When the line segment in the direction is extended, the property of intersecting at one point (intersection coordinate) is used. That is, there is one point having an infinite height, and this point is obtained from the uv image as the intersection coordinates (ζ, η). The intersection coordinates are obtained by detecting a line segment (vertical line segment) in the height direction of the building in the uv image. The line segment is detected by an image processing method such as a Hough transform or a Canny operator. By such processing, straight line parameters relating to each line segment are obtained. The straight line parameters are the straight line slope a _i and the intercept b _i , and the intersection coordinate values (ξ, η) are obtained by the calculation shown in the equation (6a). Alternatively, the operator may specify and obtain a point on a straight line.

FIG. 7 shows a pair on the straight line l ₁₂ : p ₁ (x ₁ , y ₁ ) and p ₂ (x ₂ , y ₂ ) and a pair on the straight line l ₃₄ : p ₃ (x ₃ , y ₃ ) and p. ₄ (x ₄ , y ₄ ), pair on line l ₅₆ : p ₅ (x ₅ , y ₅ ) and p ₆ (x ₆ , y ₆ ), pair on line l ₇₈ : p ₇ (x ₇ , y ₇ ) and p ₈ (x ₈ , y ₈ ). From the coordinate value of each point, the intersection coordinate value (ξ, η) is obtained according to the equation (6b). Note that at least two straight lines are sufficient for obtaining the intersection coordinates, but more accurate intersection coordinates can be obtained by detecting many vertical straight lines. The intersection coordinates are obtained over all the frames, and the intersection coordinates (ζ _i , η _i ), i = 1, 2,. Thereafter, from the temporal movement of the feature points of the omnidirectional video, the movement of the omnidirectional camera viewpoint, that is, the rotation around the three axes, the translational movement in the three axis directions, and the object shape are processed by the same processing steps as in the first embodiment. The three-dimensional information to be configured can be restored.

(Embodiment 3)
FIG. 8 a is a basic configuration diagram relating to the invention of claim 3. The restoration device according to the present embodiment includes a restoration processing determination unit 31 in the restoration device according to the first embodiment. Here, when the image coordinate value of the feature point obtained by the feature point measurement unit 11 is converted into the uv coordinate value by the measurement matrix conversion unit 12, the restoration process determination unit 31 performs a process for restoring the camera motion and the three-dimensional information. Since the determination point is different from the restoration device of the first embodiment, only this point will be described. Note that the time-series image database 10 may use a recording medium such as a hard disk, a RAID device, a CD-ROM, or a remote data resource via a network.

  FIG. 8 b shows a schematic configuration of a restoration apparatus when processing is performed in real time, and the restoration apparatus includes a data image input unit 16 instead of the time-series image database 10. Therefore, the restoration apparatus of this embodiment does not necessarily require storage means such as each database unit.

  The measurement matrix conversion unit 12 converts the image coordinate value of the feature point obtained by the feature point measurement unit 11 into a coordinate value and holds the measurement matrix [B]. This measurement matrix [B] is held in the form of equation (5a). The measurement matrix [B] is subjected to a restoration process in the restoration process determination unit 12.

FIG. 9 is a flowchart showing the restoration process. When the measurement matrix [B] is input from the measurement matrix conversion unit 12 (step 301), this matrix is converted into a singular value matrix [W] by singular value decomposition by the calculation of equation (8) (step 302). . The singular value matrix [W] is a diagonal matrix, and the singular values that are the elements thereof are arranged in ascending / descending order and are all positive real numbers. Singular values of the 3 × 3 matrix element W ₃₃ and the 4 × 4 matrix element W ₄₄ are extracted from the singular values. In rank detection (step 303), the determination amount E _W is obtained by the calculation shown in the equation (34a) or the equation (34b).

Singular value W _33, the camera becomes a positive finite value when performing three degrees of freedom or movement, also singular values W44, the camera becomes a positive finite value when performing four degrees of freedom or movement, and Since it has a property that it becomes almost zero when exercise with less than 3 degrees of freedom is performed, the determination amount E _W can be used as a criterion for determining the degree of freedom of motion of the camera.

Then, it is determined whether the allowable value or less than δW is determined amount E _W, or determining the amount of E _W is tolerance δW above (step 304). This allowable value δW is a specific constant value, and the operator can set the value sequentially.

Here, if the determination amount E _W is less than a certain allowable value δw, it is determined that the camera motion has moved in a plane (step 305), and the process proceeds to process A (step 306). On the other hand, if the determination amount E _W is equal to or greater than a certain allowable value δW, it is determined that the camera motion has performed a general motion (other than planar motion) (step 307), and the process proceeds to process B (step 308).

  The process B is the process of the first embodiment, and from the temporal movement of the feature points of the omnidirectional video, the movement of the omnidirectional camera viewpoint, that is, the three-dimensional rotation and translational movement around the three axes, and the object shape This is a process for restoring information. Therefore, hereinafter, the process A will be described.

  FIG. 10 shows a flowchart of the process A (step 306) performed in the factorization processing unit 15.

  Each matrix data subjected to matrix decomposition into the three matrices [U], [W], and [V] of Expression (7) by the singular value decomposition performed in the restoration processing determination unit 15 is input (step 309). Here, [U] is a 2F × P size matrix, [W] is a P × P size diagonal matrix, and [V] is a P × P size matrix.

  First, in noise removal (step 310), the components of each matrix of rank 4 or higher are deleted as shown in the second term of equation (8). At the time of this deletion, the matrix [U] is taken out, the fourth to Pth columns are deleted from the elements of this matrix, the matrix consisting of the remaining components is held, the matrix [W] is taken out, The fourth to Pth rows and the fourth to Pth columns are deleted in the element, the matrix composed of the remaining components is retained, the matrix [V] is extracted, and the fourth to Pth rows are extracted from the elements of this matrix. Delete up to the eye and hold the matrix composed of the remaining components (Equation (9)). Next, from the matrix obtained by taking the square of the diagonal elements of the matrix [W] from which the 4th to Pth rows and the 4th to Pth columns are deleted, the matrix [U shown in Equations (10) and (11) is used. '] And matrix [V'] are obtained.

  Subsequently, in the transformation matrix calculation (step 311), the held matrix [U ′] is taken out, and the symmetric matrix [C] in the simultaneous conditional expressions shown in equations (13) to (15) (equation (12)). Calculate the coefficients for each element (shown). These coefficient calculations are easily obtained by matrix operation, and this conditional expression is calculated for all frames. Next, with respect to the simultaneous conditional expressions shown in the equations (13) to (15) for all frames, each element of the 3 × 3 size matrix [C] is changed by using numerical calculation such as the least square method. decide.

  Further, the obtained matrix [C] is subjected to eigenvalue decomposition as shown in Expression (16). Here, a matrix [C ′] of Expression (17) is generated from the square of the eigenvalue matrix and the eigenvalue matrix, and a matrix [Q] having this matrix element as a component is calculated according to Expression (18).

Subsequently, in the plane motion restoration (step 312), the matrix [M ′] is calculated from the matrix [Q] and the retained matrix [U ′] by the matrix calculation of Expression (19). The matrix element (m _ix , n _ix ) or (m _iy , n _iy ) of each frame (i-th frame) is extracted from the matrix [M ′], and the yaw rotation θ _i is restored by the calculation of equation (20). A matrix element (T _iu , T _iv ) of each frame (i-th frame) is extracted from the matrix [M ′]. From this (T _iu , T _iv ), the XY translational motion (Tx _i , Ty _i ) in the Euclidean space in the i-th frame is calculated by the calculation of Expression (22).

Furthermore, the matrix [S ′] is obtained by performing the matrix operation shown in the equation (23) from the matrix [V ′] held in advance and the matrix [Q] obtained by the conversion matrix calculation. Next, the transformation shown in Expression (24) is performed on the elements of the matrix [S ′], and the matrix having this as an element is set to [P] (step 313). The column vectors whose matrix is [P] are the three-dimensional coordinate values (X _j , Y _j , Z _j ) in the Euclidean space of the j-th feature point.

  As described above, according to the restoration device of the present embodiment, when the restoration process determination unit 31 determines that the camera motion is a plane movement, the time point of the feature point of the omnidirectional video is obtained by executing the process A. From the movement, the movement of the omnidirectional camera viewpoint, that is, the rotation around the optical axis and the XY translational movement, and the three-dimensional information constituting the object shape are restored.

  Further, when it is determined by the restoration processing determination unit 31 that the camera motion is a general motion other than the planar motion, the processing B based on the processing steps shown in FIG. From the temporal movement of the feature point of the azimuth image, the movement of the omnidirectional camera viewpoint, that is, the rotation around the three axes and the translational movement in the three axis directions, and the three-dimensional information constituting the object shape can be restored.

(Embodiment 4)
FIG. 11 a is a schematic configuration diagram of an embodiment for carrying out the invention of claim 4. The restoration apparatus according to the present embodiment includes the restoration processing determination unit 31 in the restoration apparatus according to the second embodiment. Here, after measuring the uv coordinate value of each feature point in the time-series uv image by the feature point measurement unit 11, the coordinate value is used for the restoration processing determination unit 31 to restore the camera motion and the three-dimensional information. Since this is different from the restoration processing apparatus of the second embodiment, only this point will be described. In this embodiment, the time-series image database may be in the form of using a recording medium such as a hard disk, a RAID device, a CD-ROM, or a remote data resource via a network.

  FIG. 11B is a processing configuration diagram in the case of processing in real time, and the restoration device of this embodiment also uses the image input unit 16 instead of the time-series image database 10 as in the restoration devices of the first, second, and third embodiments. I have. Therefore, the restoration apparatus of this embodiment does not necessarily require storage means such as each database unit 10.

The feature point measurement unit 11 obtains uv coordinate values of each feature point and holds a measurement matrix [B] having these as matrix elements. This measurement matrix [B] is held in the form of equation (5a). Next, the measurement matrix [B] is subjected to processing in the restoration processing determination unit 31. The processing flow in the restoration processing determination unit 31 is based on the flowchart shown in FIG. That is, the singular value decomposition is performed by using the measurement matrix [B] for the calculation of Expression (8) to obtain the singular value matrix [W]. The singular value matrix [W] is a diagonal element, and the singular values which are each element are arranged in ascending / descending order and are all positive real numbers. The singular values of the 3 × 3 matrix element W ₃₃ and the 4 × 4 matrix element W ₄₄ are extracted from the singular values. In rank detection, the determination amount E _W is obtained by the calculation shown in Expression (34a) or Expression (34b).

Then, it is determined whether the allowable value or less than δW is determined amount E _W, or determining the amount of E _W is tolerance δW more. This allowable value δW is a specific constant value, and the operator can set the value sequentially. If the determination amount E _W is less than a certain allowable value δW, it is determined that the camera motion is a plane motion, and the process proceeds to process A. On the other hand, when the determination amount E _W is equal to or greater than a certain allowable value δW, it is determined that the camera motion has performed a general motion (other than a planar motion), and the process proceeds to process B. The process B performs the same process as in the second embodiment, and from the temporal movement of the feature points of the omnidirectional video, the movement of the omnidirectional camera viewpoint, that is, the rotation around the three axes and the translational movement in the three axis directions, and the object This is a process for restoring the three-dimensional information constituting the shape. The process A is the same as the process A in the third embodiment, and a description thereof will be omitted.

  In the restoration method described in the above embodiment, it is needless to say that the processing steps shown in FIGS. 1a to 11b can be configured by a computer program and the program can be executed by the computer. For recording a program for causing the computer to execute the process of the processing, such as a FD (floppy disk (registered trademark)), MO, ROM, memory card, CD, It can be recorded on a DVD, a removable disk, etc., and stored or distributed. It is also possible to provide the above program via a network such as the Internet or e-mail.

  Then, the present invention can be implemented by installing the program from these recording media into a computer, or by downloading the program from a network and installing the program into the computer. However, installation on a computer is a computer unit, and when there are multiple computers to be installed due to multiple devices and systems, it is natural that the program is installed for each necessary processing part. is there. In this case, the program may be recorded on a recording medium corresponding to a computer, or downloaded via a network.

The schematic block diagram which showed the example of the form of the apparatus which implements invention of Claim 1. The schematic block diagram which showed the example of the form of the apparatus which implements invention of Claim 1. The flowchart explaining the process process in the factorization method process part 15. FIG. Explanatory drawing explaining the measurement of intersection coordinates. A schematic configuration diagram showing an example of an apparatus for carrying out the invention according to claim 2. A schematic configuration diagram showing an example of an apparatus for carrying out the invention according to claim 2. The flowchart explaining the process process of uv image conversion in the image conversion process part 21. FIG. Explanatory drawing explaining uv image generation. Explanatory drawing explaining the measurement of the intersection coordinate in a uv image. A schematic configuration diagram showing an example of an apparatus for carrying out the invention according to claim 3. A schematic configuration diagram showing an example of an apparatus for carrying out the invention according to claim 3. The flowchart explaining the process process in the restoration process determination part 31. FIG. The flowchart explaining the process A performed in the factorization method process part 15. FIG. FIG. 5 is a schematic configuration diagram showing an example of an apparatus for carrying out the invention according to claim 4; FIG. 5 is a schematic configuration diagram showing an example of an apparatus for carrying out the invention according to claim 4; An example of an image captured by an omnidirectional camera with an outline attached to an object. An example of an image taken by an omnidirectional camera, showing only the outline of the object. An example of an image captured by an omnidirectional camera without a contour line attached to an object. An example of an image obtained by uv coordinate conversion of an image captured by an omnidirectional camera, in which a contour is attached to an object. An example of an image obtained by uv coordinate conversion of an image captured by an omnidirectional camera, showing only the outline of an object. An example of an image obtained by uv coordinate conversion of an image captured by an omnidirectional camera, in which a contour is not attached to an object. The figure which shows the omnidirectional camera, the spatial information to restore | restore, and a camera coordinate.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Time series image database, 11 ... Feature point measurement part, 12 ... Measurement matrix conversion part, 13 ... Intersection coordinate measurement part, 14 ... Measurement matrix input part, 15 ... Factorization method processing part, 16 Image input part 21 ... Image Conversion processing unit 31 ... Restoration processing determination unit

Claims (16)

The movement of the omnidirectional camera viewpoint in time series based on the amount of temporal change in the image coordinate values of the feature points placed in the target image in the chronological omnidirectional image (or wide-field image) acquired from the omnidirectional camera. And a method for restoring the three-dimensional information constituting the object shape of the outside world,
In the feature point coordinate system set for the acquired time-series omnidirectional image (or wide-field image), the image coordinate value indicating the temporal change amount of the feature point in each image and the line segment near the projection center in each image A step (measurement matrix measurement step) of measuring an image coordinate value (intersection coordinate value) of an intersection formed by
Different coordinate values (uv coordinate values) are obtained from the phase angle and elevation angle calculated from the image coordinate values of each feature point obtained in this measurement matrix measurement step, and the uv coordinate values and intersection coordinate values of these feature points are obtained. Is used to generate matrix data (matrix decomposition data) for restoration processing, singular value decomposition of this matrix decomposition data, and noise removal to obtain matrix data representing motion information and matrix data representing three-dimensional information From the motion information component, a transformation matrix that satisfies the conditions set to define the motion is obtained, and this transformation matrix is applied to the matrix data that is motion information to rotate around the optical axis of the omnidirectional camera viewpoint. Restoring the motion and translational motion on the plane perpendicular to it (XY plane) (this motion with 3 degrees of freedom is called plane motion), and using the rotation angle around the optical axis and the intersection coordinate value, Restore rotational movement around the axis And, further, a step (restoration step of planar motion and three-dimensional information) to restore the three-dimensional information that constitutes the object shape by applying an inverse matrix of the transformation matrix matrix data representing the three-dimensional information,
The re-projection coordinate value in the uv coordinate system for each feature point is obtained from the plane interlock and the three-dimensional information obtained in the plane motion and the three-dimensional information restoration step, and the reprojected coordinate value and matrix elements constituting the matrix decomposition data Using this error and the Z coordinate value (component of 3D information) of the feature point obtained in the plane motion and 3D information restoration step, the translation of the camera viewpoint in the Z-axis direction is calculated. A step of restoring motion (Z-axis translational motion restoration step);
It is determined whether or not the error obtained by the Z-axis translational motion recovery snubber has converged to a certain value or less. If not, the planar motion obtained by the planar motion and the three-dimensional information restoration step and 3 Determining the deformation motion from the dimensional information and the rotational motion around the axis other than the optical axis, combining this deformation coefficient with the uv coordinate value of each feature point to determine a new uv coordinate value, and using this as a matrix element Generating decomposition data, returning to the step of restoring the plane motion and the three-dimensional information, and repeating the steps until the error converges to a certain value or less to restore the omnidirectional camera motion and the three-dimensional information. To restore omnidirectional camera motion and 3D information. The movement of the omnidirectional camera viewpoint in time series based on the amount of temporal change in the image coordinate values of the feature points placed in the target image in the chronological omnidirectional image (or wide-field image) acquired from the omnidirectional camera. And a method for restoring the three-dimensional information constituting the object shape of the outside world, and for the acquired time-series omnidirectional images (or wide-field images), depending on the phase angle and elevation angle obtained from each image coordinate value A time-series image (uv image) is generated so that each image coordinate value is converted into a uv coordinate value and each pixel is associated with the uv coordinate value. In the feature point coordinate system set for this uv image, a feature in each image is generated. A step of measuring a uv coordinate value indicating a temporal change amount of a point and a uv coordinate value (intersection coordinate value) of an intersection formed by a line segment in the vicinity of the projection center in each uv image (uv measurement matrix measurement step) When,
Generate matrix data (matrix decomposition data) for restoration processing using the uv coordinate value of each feature point obtained in the uv measurement matrix measurement step and the intersection coordinate value, and singular value decomposition of the matrix decomposition data, The matrix data representing the motion information and the matrix data representing the three-dimensional information are obtained by performing noise removal, and a transformation matrix satisfying the conditions set for defining the motion is obtained from the motion information components, and the motion information and This transformation matrix is applied to the matrix data to restore the rotational motion around the optical axis of the omnidirectional camera viewpoint and the translational motion on this plane (XY plane) (this motion with 3 degrees of freedom is called planar motion). Then, using the rotation angle around the optical axis and the coordinate value of the intersection point, the rotational motion around the other axis other than the optical axis is restored, and the inverse matrix of this transformation matrix is applied to the matrix data representing the three-dimensional information. Configure object shape And step (restoration step of planar motion and three-dimensional information) to restore the three-dimensional information that,
A reprojection coordinate value in the uv coordinate system for each feature point is obtained from the plane motion and 3D information obtained in the plane motion and 3D information restoration step, and the reprojected coordinate value and matrix elements constituting matrix decomposition data are obtained. Using this error and the Z coordinate value (component of 3D information) of the feature point obtained in the plane motion and 3D information restoration step, the translational motion of the camera viewpoint in the Z-axis direction is calculated. Restoring (Z-axis translational motion restoring step),
It is determined whether or not the error obtained in the Z-axis translational motion restoration step has converged to a certain value or less. If the error has not converged, the planar motion obtained in the planar motion and three-dimensional information restoration step and 3 Deformation coefficient is obtained from dimensional information and rotational motion around an axis other than the optical axis, and this deformation coefficient is combined with the uv coordinate value of each feature point to obtain a new uv coordinate value, and matrix decomposition using this as a matrix element Generating data, returning to the plane motion and 3D information restoration step, and repeating the process until the error converges to a certain value or less to restore the omnidirectional camera motion and 3D information. Omni-directional camera motion and 3D information restoration method. In the measurement matrix measurement step, a determination value representing a degree of freedom of movement is calculated from a singular value component obtained by singular value decomposition of the measurement matrix,
When the determination value is less than a certain value, the camera motion is regarded as a plane motion, and in the step of restoring the omnidirectional camera motion and the three-dimensional information, the rotation around the optical axis and the plane perpendicular to the optical axis are performed. Perform 3 degrees of freedom of plane motion consisting of the above motion and processing to restore 3D information,
If the judgment value is greater than a certain value, the camera motion is regarded as a general motion other than a plane motion, and the omnidirectional camera motion and the three-dimensional information are restored, and the free motion consisting of rotation and translational motion about the camera The method of restoring an omnidirectional camera motion and three-dimensional information according to claim 1, further comprising a step of performing a process of restoring the motion of degree 6 and the three-dimensional information of the outside world. In the uv measurement matrix measurement step, a determination value representing motion is calculated from a singular value component obtained by singular value decomposition of the measurement matrix,
When the determination value is less than a certain value, the camera motion is considered to be a plane motion, and in the step of restoring the omnidirectional camera motion and the three-dimensional information, the rotation around the optical axis and the direction perpendicular to the optical axis are performed. Perform 3 degrees of freedom of plane motion consisting of motion on a plane and processing to restore 3D information,
If the judgment value is greater than a certain value, the camera motion is regarded as a general motion other than a plane motion, and the omnidirectional camera motion and the three-dimensional information are restored, and the free motion consisting of rotation and translational motion about the camera The method of restoring an omnidirectional camera motion and three-dimensional information according to claim 2, further comprising a step of performing a process of restoring the motion of degree 6 and the three-dimensional information of the outside world. The movement of the omnidirectional camera viewpoint in time series based on the amount of temporal change in the image coordinate values of the feature points placed in the target image in the chronological omnidirectional image (or wide-field image) acquired from the omnidirectional camera. And a device for restoring the three-dimensional information constituting the object shape of the outside world,
A feature point measurement unit that measures coordinate values of feature points in each image;
A measurement matrix conversion unit that converts the image coordinate values obtained by the feature point measurement unit into a uv image;
An intersection coordinate measurement unit for obtaining an intersection coordinate value near the projection center in each image;
a measurement matrix input unit that gives a matrix having uv coordinate values as matrix elements as an initial value;
An omnidirectional camera motion comprising a factorization processing unit that restores the motion of the camera and the three-dimensional information of the outside world from the data supplied from the intersection coordinate measurement unit and the measurement matrix input unit by a factorization method And 3D information restoration device. The movement of the omnidirectional camera viewpoint in time series based on the amount of temporal change in the image coordinate values of the feature points placed in the target image in the chronological omnidirectional image (or wide-field image) acquired from the omnidirectional camera. And a device for restoring the three-dimensional information constituting the object shape of the outside world,
An image conversion processing unit for converting each image into a uv image;
A feature point measurement unit that measures coordinate values of feature points in each image;
An intersection coordinate measurement unit for obtaining an intersection coordinate value near the projection center in each uv image;
a measurement matrix input unit that gives a matrix having uv coordinate values as matrix elements as an initial value;
An omnidirectional camera motion comprising a factorization processing unit that restores the motion of the camera and the three-dimensional information of the outside world from the data supplied from the intersection coordinate measurement unit and the measurement matrix input unit by a factorization method And 3D information restoration device. A restoration processing determination unit that calculates a determination value representing a degree of freedom of movement from a singular value component obtained by performing singular value decomposition on the uv image supplied from the measurement matrix conversion unit;
When the determination value is less than a certain value, it is considered that the camera motion is a plane motion, and the factorization processing unit has a degree of freedom 3 consisting of a rotation around the optical axis and a motion on a plane perpendicular to the optical axis. To restore the plane motion and 3D information of
If the determination value is equal to or greater than a certain value, it is assumed that the camera motion is a general motion other than a plane motion, and the factorization processing unit performs a motion of 6 degrees of freedom consisting of rotation and translational motion related to the camera and 3 of the external world. 6. The apparatus for restoring omnidirectional camera motion and three-dimensional information according to claim 5, wherein a process for restoring the dimensional information is performed. A restoration process determination unit that calculates a determination value representing motion from a singular value component obtained by performing singular value decomposition on the uv image supplied from the feature point measurement unit;
When the judgment value is less than a certain value, it is considered that the camera motion is a plane motion, and the factorization processing unit has a degree of freedom comprising a rotation around the optical axis and a motion on a plane perpendicular to the optical axis. 3 to restore the plane motion and 3D information,
If the determination value is equal to or greater than a certain value, it is assumed that the camera motion is a general motion other than a plane motion, and the factorization processing unit performs a motion of 6 degrees of freedom consisting of rotation and translational motion related to the camera and 3 of the external world. 7. The apparatus for restoring omnidirectional camera motion and three-dimensional information according to claim 6, wherein processing for restoring the dimensional information is performed. The movement of the omnidirectional camera viewpoint in time series based on the amount of temporal change in the image coordinate values of the feature points placed in the target image in the chronological omnidirectional image (or wide-field image) acquired from the omnidirectional camera. And a program for executing, by a computer, a method for restoring the three-dimensional information constituting the object shape of the outside world,
In a feature point coordinate system set for a time-series omnidirectional image (or wide-field image), an image coordinate value indicating a temporal change amount of a feature point in each image and a line segment near the projection center in each image are formed. A step of measuring the image coordinate value (intersection coordinate value) of the intersection (measurement matrix measurement step);
Different coordinate values (uv coordinate values) are obtained from the phase angle and elevation angle calculated from the image coordinate values of each feature point obtained in this measurement matrix measurement step, and the uv coordinate values and intersection coordinate values of these feature points are obtained. Is used to generate matrix data (matrix decomposition data) for restoration processing, singular value decomposition of this matrix decomposition data, and noise removal to obtain matrix data representing motion information and matrix data representing three-dimensional information From the motion information component, a transformation matrix that satisfies the conditions set to define the motion is obtained, and this transformation matrix is applied to the matrix data that is motion information to rotate around the optical axis of the omnidirectional camera viewpoint. Restoring the motion and translational motion on the plane perpendicular to it (XY plane) (this motion with 3 degrees of freedom is called plane motion), and using the rotation angle around the optical axis and the intersection coordinate value, Restore rotational movement around the axis And, further, a step (restoration step of planar motion and three-dimensional information) to restore the three-dimensional information that constitutes the object shape by applying an inverse matrix of the transformation matrix matrix data representing the three-dimensional information,
The re-projection coordinate value in the uv coordinate system for each feature point is obtained from the plane interlock and the three-dimensional information obtained in the plane motion and the three-dimensional information restoration step, and the reprojected coordinate value and matrix elements constituting the matrix decomposition data Using this error and the Z coordinate value (component of 3D information) of the feature point obtained in the plane motion and 3D information restoration step, the translation of the camera viewpoint in the Z-axis direction is calculated. A step of restoring motion (Z-axis translational motion restoration step);
It is determined whether the error obtained by the Z-axis translational motion recovery snubber has converged to a certain value or less. If not, the planar motion and the planar motion obtained in the three-dimensional information recovery step A deformation motion is obtained from the three-dimensional information and a rotational motion around an axis other than the optical axis, and this deformation coefficient is combined with the uv coordinate value of each feature point to obtain a new uv coordinate value, which is used as a matrix element. Generating matrix-decomposed data, returning to the plane motion and three-dimensional information restoration step, and repeating the process until the error converges to a certain value or less to perform processing for restoring the omnidirectional camera motion and the three-dimensional information. A program for executing an omnidirectional camera motion and a method for restoring three-dimensional information. The movement of the omnidirectional camera viewpoint in time series based on the amount of temporal change in the image coordinate values of the feature points placed in the target image in the chronological omnidirectional image (or wide-field image) acquired from the omnidirectional camera. And a program for executing, by a computer, a method for restoring the three-dimensional information constituting the object shape of the outside world,
With respect to the acquired time-series omnidirectional image (or wide-field image), each image coordinate value is converted into a uv coordinate value by a phase angle and an elevation angle obtained from each image coordinate value, and each pixel is converted to its uv coordinate value. A time-series image (uv image) is generated so as to be associated, and in the feature point coordinate system set for this uv image, the uv coordinate value indicating the temporal change amount of the feature point in each image and the vicinity of the projection center in each uv image Measuring a uv coordinate value (intersection coordinate value) of an intersection formed by a line segment at (uv measurement matrix measurement step);
Generate matrix data (matrix decomposition data) for restoration processing using the uv coordinate value of each feature point obtained in the uv measurement matrix measurement step and the intersection coordinate value, and singular value decomposition of the matrix decomposition data, The matrix data representing the motion information and the matrix data representing the three-dimensional information are obtained by performing noise removal, and a transformation matrix satisfying the conditions set for defining the motion is obtained from the motion information components, and the motion information and This transformation matrix is applied to the matrix data to restore the rotational motion around the optical axis of the omnidirectional camera viewpoint and the translational motion on this plane (XY plane) (this motion with 3 degrees of freedom is called planar motion). Then, using the rotation angle around the optical axis and the coordinate value of the intersection point, the rotational motion around the other axis other than the optical axis is restored, and the inverse matrix of this transformation matrix is applied to the matrix data representing the three-dimensional information. Configure object shape And step (restoration step of planar motion and three-dimensional information) to restore the three-dimensional information that,
A reprojection coordinate value in the uv coordinate system for each feature point is obtained from the plane motion and 3D information obtained in the plane motion and 3D information restoration step, and the reprojected coordinate value and matrix elements constituting matrix decomposition data are obtained. Using this error and the Z coordinate value (component of 3D information) of the feature point obtained in the plane motion and 3D information restoration step, the translational motion of the camera viewpoint in the Z-axis direction is calculated. Restoring (Z-axis translational motion restoring step),
It is determined whether or not the error obtained in the Z-axis translational motion restoration step has converged to a certain value or less. If the error has not converged, the planar motion obtained in the planar motion and three-dimensional information restoration step and 3 Deformation coefficient is obtained from dimensional information and rotational motion around an axis other than the optical axis, and this deformation coefficient is combined with the uv coordinate value of each feature point to obtain a new uv coordinate value, and matrix decomposition using this as a matrix element Generating data, returning to the plane motion and 3D information restoration step, and repeating the process until the error converges to a certain value or less to restore the omnidirectional camera motion and 3D information. A program that executes omnidirectional camera motion and 3D information restoration method. In the measurement matrix measurement step, a determination value representing a degree of freedom of movement is calculated from a singular value component obtained by singular value decomposition of the measurement matrix,
When the determination value is less than a certain value, the camera motion is regarded as a plane motion, and in the step of restoring the omnidirectional camera motion and the three-dimensional information, the rotation around the optical axis and the plane perpendicular to the optical axis are performed. Perform 3 degrees of freedom of plane motion consisting of the above motion and processing to restore 3D information,
If the judgment value is greater than a certain value, the camera motion is regarded as a general motion other than a plane motion, and the omnidirectional camera motion and the three-dimensional information are restored, and the free motion consisting of rotation and translational motion about the camera 10. The program for executing an omnidirectional camera motion and three-dimensional information restoration method according to claim 9, further comprising a step of performing a process of restoring the motion of degree 6 and the three-dimensional information of the outside world. In the uv measurement matrix measurement step, a determination value representing motion is calculated from a singular value component obtained by singular value decomposition of the measurement matrix,
When the determination value is less than a certain value, the camera motion is considered to be a plane motion, and in the step of restoring the omnidirectional camera motion and the three-dimensional information, the rotation around the optical axis and the direction perpendicular to the optical axis are performed. Perform 3 degrees of freedom of plane motion consisting of motion on a plane and processing to restore 3D information,
If the judgment value is greater than a certain value, the camera motion is regarded as a general motion other than a plane motion, and the omnidirectional camera motion and the three-dimensional information are restored, and the free motion consisting of rotation and translational motion about the camera 11. The program for executing an omnidirectional camera motion and three-dimensional information restoration method according to claim 10, further comprising a step of performing processing for restoring the motion of degree 6 and the three-dimensional information of the outside world. The movement of the omnidirectional camera viewpoint in time series based on the amount of temporal change in the image coordinate values of the feature points placed in the target image in the chronological omnidirectional image (or wide-field image) acquired from the omnidirectional camera. And a computer-readable recording medium recording a program for executing a method for restoring the three-dimensional information constituting the object shape of the outside world by a computer,
In a feature point coordinate system set for a time-series omnidirectional image (or wide-field image), an image coordinate value indicating a temporal change amount of a feature point in each image and a line segment near the projection center in each image are formed. A step of measuring the image coordinate value (intersection coordinate value) of the intersection (measurement matrix measurement step);
Different coordinate values (uv coordinate values) are obtained from the phase angle and elevation angle calculated from the image coordinate values of each feature point obtained in this measurement matrix measurement step, and the uv coordinate values and intersection coordinate values of these feature points are obtained. Is used to generate matrix data (matrix decomposition data) for restoration processing, singular value decomposition of this matrix decomposition data, and noise removal to obtain matrix data representing motion information and matrix data representing three-dimensional information From the motion information component, a transformation matrix that satisfies the conditions set to define the motion is obtained, and this transformation matrix is applied to the matrix data that is motion information to rotate around the optical axis of the omnidirectional camera viewpoint. Restoring the motion and translational motion on the plane perpendicular to it (XY plane) (this motion with 3 degrees of freedom is called plane motion), and using the rotation angle around the optical axis and the intersection coordinate value, Restore rotational movement around the axis And, further, a step (restoration step of planar motion and three-dimensional information) to restore the three-dimensional information that constitutes the object shape by applying an inverse matrix of the transformation matrix matrix data representing the three-dimensional information,
The re-projection coordinate value in the uv coordinate system for each feature point is obtained from the plane interlock and the three-dimensional information obtained in the plane motion and the three-dimensional information restoration step, and the reprojected coordinate value and matrix elements constituting the matrix decomposition data Using this error and the Z coordinate value (component of 3D information) of the feature point obtained in the plane motion and 3D information restoration step, the translation of the camera viewpoint in the Z-axis direction is calculated. A step of restoring motion (Z-axis translational motion restoration step);
It is determined whether or not the error obtained by the Z-axis translational motion recovery snubber has converged to a certain value or less. If not, the planar motion obtained by the planar motion and the three-dimensional information restoration step and 3 Determining the deformation motion from the dimensional information and the rotational motion around the axis other than the optical axis, combining this deformation coefficient with the uv coordinate value of each feature point to determine a new uv coordinate value, and using this as a matrix element Generating decomposition data, returning to the step of restoring the plane motion and the three-dimensional information, and repeating the steps until the error converges to a certain value or less to restore the omnidirectional camera motion and the three-dimensional information. A computer-readable recording medium having recorded thereon a computer program for executing an omnidirectional camera motion and a three-dimensional information restoration method. The movement of the omnidirectional camera viewpoint in time series based on the amount of temporal change in the image coordinate values of the feature points placed in the target image in the chronological omnidirectional image (or wide-field image) acquired from the omnidirectional camera. And a computer-readable recording medium recording a program for executing a method for restoring the three-dimensional information constituting the object shape of the outside world by a computer,
With respect to the acquired time-series omnidirectional image (or wide-field image), each image coordinate value is converted into a uv coordinate value by a phase angle and an elevation angle obtained from each image coordinate value, and each pixel is converted to its uv coordinate value. A time-series image (uv image) is generated so as to be associated, and in the feature point coordinate system set for this uv image, the uv coordinate value indicating the temporal change amount of the feature point in each image and the vicinity of the projection center in each uv image Measuring a uv coordinate value (intersection coordinate value) of an intersection formed by a line segment at (uv measurement matrix measurement step);
Generate matrix data (matrix decomposition data) for restoration processing using the uv coordinate value of each feature point obtained in the uv measurement matrix measurement step and the intersection coordinate value, and singular value decomposition of the matrix decomposition data, The matrix data representing the motion information and the matrix data representing the three-dimensional information are obtained by performing noise removal, and a transformation matrix satisfying the conditions set for defining the motion is obtained from the motion information components, and the motion information and This transformation matrix is applied to the matrix data to restore the rotational motion around the optical axis of the omnidirectional camera viewpoint and the translational motion on this plane (XY plane) (this motion with 3 degrees of freedom is called planar motion). Then, using the rotation angle around the optical axis and the coordinate value of the intersection point, the rotational motion around the other axis other than the optical axis is restored, and the inverse matrix of this transformation matrix is applied to the matrix data representing the three-dimensional information. Configure object shape And step (restoration step of planar motion and three-dimensional information) to restore the three-dimensional information that,
A reprojection coordinate value in the uv coordinate system for each feature point is obtained from the plane motion and 3D information obtained in the plane motion and 3D information restoration step, and the reprojected coordinate value and matrix elements constituting matrix decomposition data are obtained. Using this error and the Z coordinate value (component of 3D information) of the feature point obtained in the plane motion and 3D information restoration step, the translational motion of the camera viewpoint in the Z-axis direction is calculated. Restoring (Z-axis translational motion restoring step),
It is determined whether or not the error obtained in the Z-axis translational motion restoration step has converged to a certain value or less. If the error has not converged, the planar motion obtained in the planar motion and 3D information restoration step and 3 Deformation coefficient is obtained from dimensional information and rotational motion around an axis other than the optical axis, and this deformation coefficient is combined with the uv coordinate value of each feature point to obtain a new uv coordinate value, and matrix decomposition using this as a matrix element Generating data, returning to the plane motion and 3D information restoration step, and repeating the process until the error converges to a certain value or less to restore the omnidirectional camera motion and 3D information. A computer-readable recording medium storing a program for executing an omnidirectional camera motion and a three-dimensional information restoration method by a computer. In the measurement matrix measurement step, a determination value representing a degree of freedom of movement is calculated from a singular value component obtained by singular value decomposition of the measurement matrix,
When the determination value is less than a certain value, the camera motion is regarded as a plane motion, and in the step of restoring the omnidirectional camera motion and the three-dimensional information, the rotation around the optical axis and the plane perpendicular to the optical axis are performed. Perform 3 degrees of freedom of plane motion consisting of the above motion and processing to restore 3D information,
If the judgment value is greater than a certain value, the camera motion is regarded as a general motion other than a plane motion, and the omnidirectional camera motion and the three-dimensional information are restored, and the free motion consisting of rotation and translational motion about the camera 14. A computer having recorded thereon a program for executing an omnidirectional camera motion and three-dimensional information restoration method by a computer according to claim 13, further comprising a step of restoring the motion of degree 6 and the three-dimensional information of the outside world. A readable recording medium. In the uv measurement matrix measurement step, a determination value representing motion is calculated from a singular value component obtained by singular value decomposition of the measurement matrix,
When the determination value is less than a certain value, the camera motion is considered to be a plane motion, and in the step of restoring the omnidirectional camera motion and the three-dimensional information, the rotation around the optical axis and the direction perpendicular to the optical axis are performed. Perform 3 degrees of freedom of plane motion consisting of motion on a plane and processing to restore 3D information,
If the judgment value is greater than a certain value, the camera motion is regarded as a general motion other than a plane motion, and the omnidirectional camera motion and the three-dimensional information are restored, and the free motion consisting of rotation and translational motion about the camera 15. A computer recording a program for executing the omnidirectional camera motion and the three-dimensional information restoration method by the computer according to claim 14, further comprising a step of restoring the motion of degree 6 and the three-dimensional information of the outside world. A readable recording medium.

Images