JP2005063013A - Method and device for restoring view point motions of full azimuth camera and three-dimensional information, and program and recording medium with the same recorded - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To highly precisely acquire and restore the shape of an object concerning an object from time series images acquired from a full azimuth camera. <P>SOLUTION: This device is configured to restore full azimuth camera view point motions in time series and three-dimensional information configuring the shape of an object in an external field from the secular change amounts of image coordinate values concerning featured points arranged in a target image in full azimuth images(or wide-angle view images) in time series acquired from the full azimuth camera. This device is provided with a featured point measuring part 11 to measure the coordinate values of featured points in each image, a measurement matrix converting part 12 to convert the image coordinate values acquired by the featured point measuring part 11 into a uv image, a measurement matrix inputting part 13 to apply a matrix with the uv coordinate values of the uv image as matrix elements as an initial value and a factorization processing part 14 to restore the motions of the camera and the three-dimensional information of an external field by factorization from data supplied from the measurement matrix inputting part 13. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention is used for vehicle-mounted images or indoor images acquired using a camera equipped with a fish-eye lens (hereinafter referred to as an omnidirectional camera), general time-series images such as sea images from a ship, aerial photography, etc. The three-dimensional shape of the outside world, that is, the subject (object), moving from the time-series image acquired by the omnidirectional camera to the omnidirectional camera viewpoint related to the roll, pitch, and yaw rotation, the distribution motion, and the time-series video. ) To restore the three-dimensional information constituting the external shape.

  In the field of computer vision, there are three-dimensional analysis methods using stereo measurement and epipolar plane analysis as methods for measuring or acquiring the shape of an object from time-series image data. According to this method, it is possible to restore the three-dimensional position information related to the spatial shape or the spatial structure and the motion related to the camera viewpoint from a plurality of time-series images in which the object is photographed.

  However, it is difficult to seamlessly acquire images in time-series images taken while moving the camera using moving means etc. due to the shooting environment and subtle movements of the camera. In some cases, it is difficult to accurately restore camera motion and object shape.

  Furthermore, the camera motion and object shape restoration in the Euclidean space can be performed using a factorization method (for example, Non-Patent Document 1 and Non-Patent Document 2) from image coordinate values that can be expressed in an orthogonal coordinate system with a flat image surface. Although it is possible, when a camera with a finite field angle is used, it is difficult to restore the delicate rotational motion of the camera, and the translation and rotational motion may not be accurately discriminated.

On the other hand, in order to avoid a finite field angle, imaging with an omnidirectional camera is conceivable as means for acquiring a landscape of 360 degrees at a time. An omnidirectional camera is often used with its camera optical axis set vertically upward, and is suitable for posture estimation of an omnidirectional camera viewpoint and space measurement related to the outside world (for example, Non-Patent Document 3). In addition, by using an omnidirectional camera, it is possible to restore a minute rotational motion, which is excellent in that both translational motion and rotational motion can be accurately restored.
C. tomasi and T.M. Kanade; “Shape and Motion from Image Stream Under Orthography: A Factorization Method”, International Journal of Computer Vision, Vol. 9, no. 2, 1992. C. J. et al. Poelman and T.W. Kanade; "A Paraperspective Factor Method for Shape and Motion Recovery", IEEE trans. Pattern Anal. and Mach. Intel, Vol. 19, no. 3, pp. 206-218, 1997. Miyakawa, Wakabayashi, Arikawa; “Motion and shape restoration from omnidirectional images by omnidirectional projection factorization”, Pattern Recognition and Media Understanding (PRMU) Study Group, June, 2002

  Acquire spatial information of objects in the outside world using a measuring method that applies the principle of stereo vision in a time-series image acquired in a shooting environment that visualizes a 360-degree landscape at once, such as an omnidirectional camera using a moving means. However, in moving shooting in various shooting environments, since the camera moves minutely, a seamless time-series image cannot be easily acquired. For this reason, the influence of random noise is large, and it is difficult to restore the camera movement and the shape of the object simultaneously and with high accuracy at all times.

  For example, when the method of Non-Patent Document 3 is applied, as shown in FIG. 10, roll rotation, pitch rotation, and translational motion in the Z-axis direction are applied in addition to the optical axis rotation (yaw rotation) of the omnidirectional camera. Therefore, it was difficult to restore accurate motion and three-dimensional shape. In other words, as shown in FIG. 10, the omnidirectional camera is often used in a posture in which the optical axis is installed vertically upward, and therefore, a plane motion with a degree of freedom 3 consisting of yaw rotation and XY translational motion is restored. The main focus was placed.

  However, in the real environment, when roll rotation, pitch rotation, and Z-axis translational motion are included in addition to such planar motion, simple calculations using an omnidirectional camera cannot be used, and the camera motion and three-dimensional shape are restored. The calculation of the above becomes complicated, and the minute camera posture change related to the omnidirectional camera as described above is mixed with noise, and there is a problem that it cannot be accurately and robustly restored.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide an omnidirectional camera viewpoint motion capable of acquiring and restoring an object shape related to an object with high accuracy from a time-series image acquired from an omnidirectional camera. An object of the present invention is to provide a three-dimensional information restoration method, apparatus and program thereof, and a recording medium recording the same.

  In order to solve the above-mentioned problems, a viewpoint motion and three-dimensional information restoration method, apparatus and program thereof, and a recording medium on which the omnidirectional camera of the present invention is recorded are feature points in an omnidirectional image acquired by the omnidirectional camera. From the temporal change of the coordinates of the three-degree-of-freedom motion (planar motion) of the three-dimensional shape of the target object and the rotational motion around the optical axis (of the viewpoint) of the omnidirectional camera and the translational motion on the XY plane by the factorization method Furthermore, after obtaining the translational motion of the omnidirectional camera in the Z-axis direction (of the viewpoint) by calculation, iteratively calculates until these errors are below a predetermined value, and as a result, the three-dimensional The information and the four-degree-of-freedom movement (of the viewpoint) of the omnidirectional camera are restored.

That is, the invention according to claim 1 is based on the temporal change amount of the image coordinate value related to the feature point arranged in the target image in the time-series omnidirectional image (or wide-field image) acquired from the omnidirectional camera. , The movement of the omnidirectional camera viewpoint in time series, and the method of restoring the three-dimensional information constituting the object shape of the outside world,
In the feature point coordinate system set in the acquired time-series omnidirectional image (or wide-field image), a step of measuring an image coordinate value indicating a temporal change amount of the feature point in each image (measurement matrix measurement step);
Another coordinate value (uv coordinate value) is obtained from the phase angle and elevation angle calculated from the image coordinate value of each feature point obtained in the measurement matrix measurement step, and the restoration process is performed from the uv coordinate value of each feature point. Generate matrix data (matrix decomposition data), perform singular value decomposition on the matrix decomposition data and perform noise removal to obtain matrix data representing motion information and matrix data representing three-dimensional information. In the component of the motion information, A transformation matrix that satisfies the conditions set to define the motion is obtained, and this transformation matrix is applied to the matrix data that becomes the motion information, and the rotational motion around the optical axis of the omnidirectional camera viewpoint and the plane perpendicular to the axis ( 3rd order that reconstructs translational motion on the XY plane (this motion with 3 degrees of freedom is called planar motion), and further forms an object shape by applying an inverse matrix of this transformation matrix to matrix data representing three-dimensional information. And step (restoration step of planar motion and three-dimensional information) to restore the information,
The reprojected coordinate value in the uv coordinate system for each feature point is obtained from the planar motion and 3D information obtained in the plane motion and 3D information restoration step, and the reprojected coordinate value and matrix elements constituting the matrix decomposition data Using this error and the Z coordinate value (component of 3D information) of the feature point obtained in the plane motion and 3D information restoration step, the translation of the camera viewpoint in the Z-axis direction is calculated. A step of restoring motion (Z-axis translational motion restoration step);
It is determined whether the error obtained in the Z-axis translational motion restoration step has converged to a certain value or less. If not, the planar motion obtained in the planar motion and three-dimensional information restoration step. Then, the deformation coefficient is obtained from the three-dimensional information, and the deformation coefficient is combined with the uv coordinate value of each feature point to obtain a new uv coordinate value, and matrix decomposition data having this as a matrix element is generated to generate the plane motion and 3 Returning to the step of restoring the dimensional information, the method includes repeating the steps until the error converges to a certain value or less to restore the omnidirectional camera motion and the three-dimensional information.

According to the second aspect of the present invention, in the time-series omnidirectional image (or wide-field image) acquired from the omnidirectional camera, the temporal change amount of the image coordinate value regarding the feature point arranged in the target image is A method of restoring the three-dimensional information constituting the movement of the omnidirectional camera viewpoint in the sequence and the object shape of the outside world,
With respect to the acquired time-series omnidirectional image (or wide-field image), each image coordinate value is converted into a uv coordinate value by a phase angle and an elevation angle obtained from each image coordinate value, and each pixel is converted to its uv coordinate value. A time series image (uv image) is generated so as to be associated, and in the feature point coordinate system set in the uv image, a uv coordinate value indicating a temporal change amount of the feature point in each image is measured (uv measurement matrix measurement) Step) and
Generate matrix data (matrix decomposition data) for restoration processing from the uv coordinate values of each feature point obtained in the uv measurement matrix measurement step, perform singular value decomposition on the matrix decomposition data, perform noise removal, and obtain motion information. The matrix data representing the three-dimensional information and the matrix data representing the three-dimensional information are obtained, and the transformation matrix satisfying the conditions set for defining the motion is obtained in the motion information component. The rotational motion around the optical axis of the omnidirectional camera viewpoint and the translational motion on the plane perpendicular to the axis (XY plane) (this three-degree-of-freedom motion is referred to as plane motion) are restored, and three-dimensional A step of restoring the three-dimensional information constituting the object shape by applying an inverse matrix of this transformation matrix to matrix data representing information (a step of restoring plane motion and three-dimensional information);
The reprojected coordinate value in the uv coordinate system for each feature point is obtained from the planar motion and 3D information obtained in the plane motion and 3D information restoration step, and the reprojected coordinate value and matrix elements constituting the matrix decomposition data Using this error and the Z coordinate value (component of 3D information) of the feature point obtained in the plane motion and 3D information restoration step, the translation of the camera viewpoint in the Z-axis direction is calculated. A step of restoring motion (Z-axis translational motion restoration step);
It is judged whether the error obtained by this Z-axis translational motion recovery snubber has converged to a certain value or less. Then, the deformation coefficient is obtained from the three-dimensional information, and the deformation coefficient is combined with the uv coordinate value of each feature point to obtain a new uv coordinate value, and matrix decomposition data having this as a matrix element is generated to generate the plane motion and 3 Returning to the step of restoring the dimensional information, the method includes repeating the steps until the error converges to a certain value or less to restore the omnidirectional camera motion and the three-dimensional information.

According to a third aspect of the present invention, a singular value component obtained by singular value decomposition of the measurement matrix in the measurement matrix measurement step in the omnidirectional camera viewpoint motion and three-dimensional information restoration method according to the first aspect is described below. A decision value representing the degree of freedom of
If the determination value is less than a certain value, the camera motion is regarded as a plane motion, and in the step of restoring the omnidirectional camera motion and the three-dimensional information, the degree of freedom comprising rotation about the optical axis and motion on a certain plane 3 to restore the plane motion and 3D information,
If the determination value is equal to or greater than a certain value, the camera motion is considered to be a motion with 4 degrees of freedom, and the omnidirectional camera motion and the step of restoring the three-dimensional information comprise rotation and translational motions related to the camera. It restores motion with 4 degrees of freedom and 3D information of the outside world.

According to a fourth aspect of the present invention, in the uv measurement matrix measurement step in the omnidirectional camera viewpoint movement and three-dimensional information restoration method according to the second aspect, in the singular value component obtained by singular value decomposition of the measurement matrix, Calculate a judgment value representing exercise,
If the determination value is less than a certain value, the camera motion is regarded as a plane motion, and in the step of restoring the omnidirectional camera motion and the three-dimensional information, the degree of freedom comprising rotation about the optical axis and motion on a certain plane 3 to restore the plane motion and 3D information,
The restoration method according to claim 2, wherein if the determination value is equal to or greater than a certain value, the camera movement is regarded as a movement with 4 degrees of freedom, and the omnidirectional camera movement and the three-dimensional information are restored. The movement of the degree of freedom 4 consisting of rotation and translational motion about the camera and the three-dimensional information of the outside world are restored.

According to the fifth aspect of the present invention, in the time-series omnidirectional image (or wide-field image) acquired from the omnidirectional camera, the temporal change amount of the image coordinate value regarding the feature point arranged in the target image is A device that restores the movement of the omnidirectional camera viewpoint in the series and the three-dimensional information constituting the object shape of the outside world,
A feature point measurement unit that measures coordinate values of feature points in each image;
A measurement matrix conversion unit that converts the image coordinate values obtained by the feature point measurement unit into a uv image;
A measurement matrix input unit for giving a matrix having the uv coordinate value of the uv image as a matrix element as an initial value;
A factorization processing unit that restores the motion of the camera and the three-dimensional information of the outside world from the data supplied from the intersection coordinate measurement unit and the measurement matrix input unit by a factorization method;

According to the sixth aspect of the present invention, in the time-series omnidirectional image (or wide-field image) acquired from the omnidirectional camera, the temporal change amount of the image coordinate value related to the feature point arranged in the target image is A device that restores the movement of the omnidirectional camera viewpoint in the series and the three-dimensional information constituting the object shape of the outside world,
An image conversion processing unit for converting each image into a uv image;
A feature point measurement unit that measures coordinate values of feature points in each image;
A measurement matrix input unit for giving a matrix having the uv coordinate value of the uv image as a matrix element as an initial value;
A factorization processing unit that restores the motion of the camera and the three-dimensional information of the outside world from the data supplied from the measurement matrix input unit by a factorization method;

According to a seventh aspect of the present invention, in the omnidirectional camera motion and three-dimensional information restoration device according to the fifth aspect, the motion is performed from the singular value component obtained by singular value decomposition of the uv image supplied from the measurement matrix converter. A restoration processing determination unit that calculates a determination value representing the degree of freedom of
If the judgment value is less than a certain value, it is considered that the camera motion is a plane motion,
The factorization processing unit performs a process of restoring three-dimensional information and three-dimensional information with three degrees of freedom consisting of rotation around the optical axis and movement on a plane perpendicular to the optical axis,
If the determination value is equal to or greater than a certain value, the camera motion is regarded as having a motion of 4 degrees of freedom, and the factorization processing unit performs a motion of 4 degrees of freedom consisting of rotation and translational motion related to the camera and the 3D of the external world. Processing to restore information is performed.

The invention according to claim 8 is the omnidirectional camera motion and three-dimensional information restoration device according to claim 6, wherein the determination value represents motion from a singular value component obtained by singular value decomposition of the uv image from the feature point measurement unit. A restoration processing determination unit for calculating
When the judgment value is less than a certain value, it is considered that the camera motion is a plane motion, and the factorization processing unit has a degree of freedom comprising a rotation around the optical axis and a motion on a plane perpendicular to the optical axis. 3 to restore the plane motion and 3D information,
If the determination value is equal to or greater than a certain value, the camera motion is regarded as having a motion of 4 degrees of freedom, and the factorization processing unit performs a motion of 4 degrees of freedom consisting of rotation and translational motion related to the camera and the 3D of the external world. Processing to restore information is performed.

  In the above invention, the wide-field image is a technical term synonymous with the omnidirectional image (hereinafter, the same applies to embodiments of the invention described later).

  The uv coordinate value is a value obtained by dividing a three-dimensional coordinate value of the external world onto a spherical surface centered on the viewpoint, and a value obtained by dividing the orthogonal direction component of the moving direction of the camera by the height direction component, and the moving direction component of the camera Is obtained by dividing the height component by the height direction component, and obtained as the u coordinate and the v coordinate, respectively.

  The deformation coefficient represents the degree of deformation when the translational motion in the Z-axis direction (of the viewpoint) of the omnidirectional camera and the rotational motion around other axes other than the optical axis are expressed by deformation of the uv coordinate value equivalent to them. It is a coefficient.

  The viewpoint movement and three-dimensional information restoration method and apparatus of the omnidirectional camera of the present invention can be realized by a computer and a program, and the program can be recorded on a computer-readable recording medium or provided through a network. It is. Examples of the recording medium include FD (floppy disk (registered trademark)), MO, ROM, memory card, CD, DVD, and removable disk.

  According to the omnidirectional camera viewpoint motion and three-dimensional information restoration method, apparatus and program thereof, and the recording medium on which the omnidirectional camera of the present invention is recorded, all time-series images acquired using the omnidirectional camera (using moving means) The object shape related to the object can be obtained and restored with high accuracy from the vehicle-mounted image, sea image, aerial image, indoor image, etc. In addition, high-accuracy three-dimensional stereoscopic viewing similar to conventional surveying techniques is possible.

  In particular, when applied to an in-vehicle camera, the camera vibrates with the traveling of the car. However, even in such a situation, the present invention is robust against noise and has a minute camera posture fluctuation (rotation around the optical axis: yaw rotation). In addition, it is possible to accurately measure translational motion with accuracy for interpolating a remote sensor such as GPS. In addition, since most of the operations used in the present invention are composed of linear operations, they can be easily implemented in a computer language.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(Embodiment 1)
1a and 1b are schematic configuration diagrams showing an embodiment of an apparatus for carrying out the invention of claim 1. FIG. In the present embodiment, a fish-eye camera (hereinafter referred to as a camera) using a fish-eye lens optically designed by equidistant projection (projection proportional to the incident angle from the optical axis) may be abbreviated as a camera. (Referred to as an omnidirectional camera). The present invention can also be applied to a time-series image acquired by a camera (wide-field camera) using a fisheye lens designed by another projection (solid angle projection, equal solid angle projection).

Referring to FIG. 10, the point P (X _j , Y _j , Z _j ) in the space to be restored in this embodiment, the camera motion, that is, the yaw rotation (θ _i ) and the translational motion T _i (Tx _i , Ty _i , Tz _i ) will be described.

The figure shows the positional relationship between the camera and the object (subject). For convenience of explanation, the axis along which the camera travels is the Y axis. At this time, the camera optical axis of the camera is the Z-axis direction. At this time, the camera optical axis of the camera is the Z-axis direction. In FIG. 10, the position of the projection center C (viewpoint position T _i ) is considered as the center of motion, and the translational motion in the i-th frame is represented by T _i (Tx _i , Ty _i , Tz _i ). Further, rotation around the camera optical axis (Z axis shown in FIG. 10) is assumed to be yaw rotation. On the other hand, a point P (X _j , Y _j , Z _j ) in space is projected onto an image coordinate value (x _ij , y _ij ) on the image plane in the image circle. Expression (1) is an optical projection called equidistant projection, and the phase angle ρ _{ij in} FIG. 10 can be obtained from the image coordinate values (x _ij , y _ij ) by the calculation of Expression (2). In Expression (1), f is the focal length of the fisheye lens, and D is a constant determined by D = πf / (2R). Here, R is the radius of the image circle shown in FIG. In FIG. 10, the image coordinate value of the projection center C on the image plane is (C _x , C _y ).

  The restoration apparatus shown in FIG. 1a includes a time-series database 10 in which omnidirectional images are accumulated, a feature point measurement unit 11 that measures coordinate values of feature points in each image, and image coordinate values obtained by the measurement unit. A measurement matrix conversion unit 12 for converting to a uv image, a measurement matrix input unit 13 for giving a matrix having uv coordinate values as matrix elements as initial values, and three-dimensional information of camera motion and the external world from these input data by a factorization method And a factorization processing unit 14 for restoring. The time-series image database 10 may use a recording medium such as a hard disk, a RAID device, a CD-ROM, or a remote data resource via a network.

  FIG. 1b is a schematic configuration diagram showing an example of a restoration device in the case of processing in real time. The restoration apparatus includes a data image input unit 15 instead of the time-series image database 10. Therefore, the restoration apparatus of this embodiment does not necessarily require storage means such as each database unit.

  Hereinafter, each component of the present embodiment will be described.

The feature point measurement unit 11 takes out an omnidirectional image capturing a landscape from the time-series image database 10 and sets a feature point on the omnidirectional screen. For this feature point setting, pixels with a large gradient value due to a one-dimensional or two-dimensional change in shading are detected (edge detection, etc.), and feature points are automatically generated by image processing techniques such as Canny operator and Hough transform. Let Further, the feature points may be arranged on the initial image by appropriately marking the feature points with an operator intervening. At this time, the number of feature points to be arranged is P (j = 1, 2,..., P), and the two-dimensional coordinate values of the feature points at the time of placement are determined from the origin in the image coordinate system in FIG. Coordinate values (x _1j , y _1j ), j = 1, 2,..., P are recorded. Next, a time series image following the initial image is read from the database 10 one by one, and feature points arranged in the initial image are converted to a method that focuses on a change in shading between the time series images, and a pixel pattern centered on the feature points. A tracking method between each frame (for example, KLT (Kanade-Lucas-Tomasi) method) or a method that takes into account projection distortion peculiar to an omnidirectional image is used for tracking and tracking. The two-dimensional coordinate values (x _ij , y _ij ) from the origin in the image coordinate system in FIG. 10 are recorded as image coordinate values as feature points of the (i-th image). When the time series image is continuously read, if the feature point disappears between images due to disappearance or occlusion, the tracking is stopped and the feature point tracking is ended. When the feature point tracking is completed, the number of time-series images read out is F including the initial image (i = 1, 2,..., F). The feature point measurement unit 11 stores the measurement matrix [A] by arranging the amount of change in the temporal image coordinate arrangement of the feature points in each time-series image in the data format of Expression (3).

The measurement matrix conversion unit 12 obtains the image coordinate values (x _ij , y _ij ) of the j-th feature point in the i-th frame from the measurement matrix [A] having the image coordinate values acquired by the feature point measurement unit 11 as elements. Take out. Further, the phase angle ρ _ij and the elevation angle φ _ij are obtained by temporarily converting into relative image coordinate values with the projection center (C _x , C _y ) as the origin, and by the calculations of equations (1) and (2). The phase angle and the elevation angle are subjected to coordinate conversion by calculation according to Equation (4) to obtain uv coordinate values (u _ij , v _ij ). This conversion is performed for all frames and all feature points. As a converted data format, the matrix [B] in the element format shown in Expression (5a) is held as an initial value in the measurement matrix input unit 13.

  The u-coordinate value and the v-coordinate value are obtained by projecting the three-dimensional coordinate values (X, Y, Z) of the external world onto a spherical surface with the viewpoint Ti as the center, and the orthogonal direction component of the moving direction of the camera is high. It corresponds to a value obtained by dividing by the direction component and a value obtained by dividing the moving direction component of the camera by the height direction component. That is, when the three-dimensional coordinate value (X, Y, Z) of the outside world is projected onto a spherical surface having a radius l centered on the viewpoint Ti, (X ′, Y ′, Z ′) = (cos (φ) cos (ρ ), Cos (φ) sin (ρ), sin (φ)) (see FIG. 10). The uv coordinate value is calculated from the unit spherical coordinate value (X ′, Y ′, Z ′) to (u, v) = (X ′ / Z ′, Y ′ / Z ′) = (cot (φ) cos (ρ), cot (φ) sin (ρ)). In the general spherical coordinate system, since the coordinates are defined using the angle φ ′ measured from the Z axis in FIG. 10, (X ′, Y ′, Z ′) = (sin (φ ′) cos (ρ ), Sin (φ ′) sin (ρ), cos (φ ′)).

  On the other hand, in this embodiment, since the coordinates are defined using the elevation angle φ when looking up the three-dimensional coordinate value (X, Y, Z) from the viewpoint in FIG. 10, φ ′ = π / 2. −φ and (sin (π / 2−φ) cos (ρ), sin (π / 2−φ) sin (ρ), cos (π / 2−φ)) = (cos (φ) cos (ρ ), Cos (φ) sin (ρ), sin (φ)).

  FIGS. 15a (1), (2) and (3) show examples of images taken by the omnidirectional camera. In the figure, a (1) is an example of an image taken by an omnidirectional camera and an object is attached with an outline, and a (2) is an example of an image taken by an omnidirectional camera and only the outline of the object. A (3) is an example of an image captured by an omnidirectional camera and does not have a contour line of an object.

  FIGS. 15b (1), (2), and (3) show examples in which the image of FIG. 15a (1) is converted into uv coordinate values according to the processing flow disclosed in FIG. 5, and the uv image shown in FIG. 4 is generated. It is a thing. In FIG. 15, b (1) is an example of an image obtained by uv coordinate conversion of an image captured by an omnidirectional camera, and an object is attached with an outline, and b (2) is an image captured by an omnidirectional camera. B (3) is an example of an image obtained by performing uv coordinate conversion on an image captured by an omnidirectional camera. The outline is not attached.

  As is apparent from FIGS. 15 b (1), (2), and (3), in this coordinate conversion, the measurement error is emphasized to a point that is further away from the v-axis as the feature point (point closer to the bottom of the building) with a smaller elevation angle φ is converted. Is done. That is, when feature points are measured by the same operation, errors that are difficult to understand on the xy plane can be grasped more accurately on the uv plane. Therefore, if the movement of the camera viewpoint and the three-dimensional coordinate value are obtained so that the emphasized error is minimized, highly accurate measurement can be performed.

  Further, in the uv plane, the roll rotation movement can be converted into the coordinate movement in the v direction, the pitch rotation movement can be converted into the coordinate movement in the u direction, and the Z-axis translation movement can be simply converted into scale conversion in the uv coordinate axis system. The rotation, pitch rotation, and Z-axis translation can be easily distinguished. By utilizing this ease of discrimination, the present invention repeatedly deforms the uv coordinate value using the deformation coefficient, and continues to reconstruct the camera motion and the three-dimensional coordinate value repeatedly, so that the camera motion with higher accuracy can be obtained. It is possible to restore the three-dimensional coordinate values.

Next, an outline of processing of the factorization method processing unit 14 will be described. FIG. 2 is a flowchart showing the processing steps. First, when measurement matrix data is provided from the measurement matrix input unit 14, the coefficient ε _ij is initialized as shown in the equation (5b), and the data shown in the equation (6) from the matrix [B] of the processing values and the intersection coordinate values. Is again stored as a matrix [B] (step 101). At this time, the matrix [B] of the processing values held in the measurement matrix input unit is not overwritten and is held until the processing of FIG. 2 ends. This matrix [B] becomes decomposition matrix data.

  The matrix [B] is subjected to singular value decomposition (step 102) to be decomposed into three matrices [U], [W], and [V] shown in Expression (7). Here, [U] is a 2F × P size matrix, [W] is a P × P size diagonal matrix, and [V] is a P × P size matrix.

  Further, in noise removal (step 103), as shown in the second term of equation (8), the components of each matrix of rank 4 or higher are deleted. At the time of this deletion, the matrix [U] is taken out, the fourth to Pth columns are deleted from the elements of this matrix, the matrix consisting of the remaining components is held, the matrix [W] is taken out, The fourth to Pth rows and the fourth to Pth columns are deleted in the element, the matrix composed of the remaining components is retained, the matrix [V] is extracted, and the fourth to Pth rows are extracted from the elements of this matrix. Delete up to the eye and hold the matrix composed of the remaining components (Equation (9)). Next, from the matrix obtained by taking the square of the diagonal elements of the matrix [W] from which the 4th to Pth rows and the 4th to Pth columns are deleted, the matrices shown in Expression (10) and Expression (ll) [U ′] and matrix [V ′] are obtained.

  Next, in the transformation matrix calculation (step 104), the held matrix [U ′] is taken out, and the symmetric matrix [C] in the simultaneous conditional expressions shown in Expressions (13) to (15) (Expression (12)). Calculate the coefficients for each element (shown). These coefficient calculations are easily obtained by matrix operation, and this conditional expression is calculated for all frames. Next, each element of the 3 × 3 size matrix [C] is determined using numerical calculation such as the least square method with respect to the simultaneous conditional expressions shown in Expressions (13) to (15) of all frames. . Further, the obtained matrix [C] is subjected to eigenvalue decomposition as shown in Expression (16). Here, a matrix [C ′] of Expression (17) is generated from the square of the eigenvalue matrix and the eigenvalue matrix, and a matrix [Q] having this matrix element as a component is calculated according to Expression (18).

Next, the matrix [M ′] is calculated from the obtained matrix [Q] and the stored matrix [U ′] by the matrix operation of Expression (19). The matrix element (m _ix , n _ix ) or (m _iy , n _iy ) of each frame (i-th frame) is extracted from the matrix [M ′], and the yaw rotation θ _i is restored using equation (20).

Further, the matrix element (T _iu , T _iv ) of each frame (i-th frame) is extracted from the matrix [M ′]. From this (T _iu , T _iv ), the XY translational motion (Tx _i , Ty _i ) in the Euclidean space in the i-th frame is calculated using equation (21) (step 105). Further, the error shown in the equations (24) and (25) is calculated.

Next, the calculation of Expression (26) is performed to restore the translational motion Tz _i of the Z axis in each frame (i-th frame) (step 106).

Subsequently, from the matrix [V ′] held in advance and the matrix [Q] obtained by the conversion matrix calculation, the matrix calculation shown in Expression (22) is performed to obtain the matrix [S ′]. Then, the transformation shown in Expression (23) is performed on the elements of the matrix [S ′], and the matrix having these elements as [P] (step 107). The column vectors whose matrix is [P] are the three-dimensional coordinate values (X _j , Y _j , Z _j ) in the Euclidean space of the j-th feature point.

Here, the error ΔW obtained by the equations (27) and (27a) is calculated (step 108). Here, a determination is made as to whether or not this error has converged below a certain value (step 109). If the error has converged to a certain value, the process is terminated. On the other hand, if not converged, the deformation coefficient δ _ij is updated by the calculation of Expression (28) (step 110). Further, with this coefficient update, the matrix [B], which is the matrix decomposition data, is updated by Expression (6). The above process is repeated until the error ΔW converges to a certain value or less.

The deformation coefficients ε _ij and δ _ij are those of the omnidirectional camera (from the viewpoint). This is a coefficient representing the degree of deformation when the translational motion in the Z-axis direction and the rotational motions ωi, ψi around other axes other than the optical axis are represented by deformations of uv coordinate values equivalent to them. In this embodiment, the XY translation components (Txi, Tyi) and the three-dimensional information (Xj, Yj, Zj) of the camera viewpoint obtained in the plane motion and the three-dimensional information restoration step, and the XY axes around the XY axes other than the optical axis. Using the rotation angle (ωi, ψi), a coefficient (ε coefficient; ε _{ij in} equation (29)) for deforming the uV coordinate value is obtained, and each feature point obtained in the plane motion and three-dimensional information restoration step is calculated. A coefficient for converting the uv coordinate value (δ coefficient; equation (29)) using the height information (Zj) and the position (Tzi) of each viewpoint on the Z axis obtained in the step of restoring the Z-axis translational motion. Δ _ij ). Then, the value obtained by multiplying each matrix element of the measurement matrix ([B] in equation (5a)) obtained in the measurement matrix measurement step by the ε coefficient and the measurement matrix obtained in the measurement matrix measurement step Matrix decomposition data ([B] in equation (5c)) that is a new matrix element obtained by combining each coordinate value obtained by subtracting the intersection coordinate value (ζ, η) from each matrix element and the value obtained by multiplying the respective δ coefficients. ) Is generated.

  As described above, according to the restoration device of the present embodiment, from the temporal movement of the feature point of the omnidirectional video, the movement of the omnidirectional camera viewpoint, that is, the rotation around the optical axis and the translational movement in the triaxial direction, The three-dimensional information constituting the object shape can be restored.

(Embodiment 2)
FIG. 3A is a schematic configuration diagram showing an example of an apparatus for carrying out the invention of claim 2. The restoration apparatus according to the present embodiment includes the image conversion processing unit 21 in the restoration apparatus according to the first embodiment. Here, since the image taken out from the time-series image database 10 is used for the image conversion processing unit 21 to generate a deformed image (uv image), it differs from the restoration apparatus of the first embodiment, and only this point will be described.

  The restoration apparatus according to the present embodiment includes a time-series image database 10 in which omnidirectional images are accumulated, an image conversion processing unit 21 that converts each image into a uv image, and a feature point measurement that measures coordinate values of feature points in each image. Unit 11, measurement matrix input unit 13 that gives a matrix having uv coordinate values as matrix elements as initial values, and a factorization method processing unit that restores camera motion and three-dimensional information of the external world from these input data by factorization 14. Note that the time-series image database 14 may be in the form of using a recording medium such as a hard disk, a RAID device, a CD-ROM, or a remote data resource via a network.

  FIG. 3B is a processing configuration diagram in the case of processing in real time. The restoration apparatus includes an image input unit 15 instead of the time-series image database 10. Therefore, the restoration apparatus of this embodiment does not necessarily require storage means such as each database unit.

  In the present embodiment, the image conversion processing unit 21 generates a uv image from the image extracted from the time-series image database 10. In order to generate a uv image, the image conversion processing unit 21 designates a uv coordinate value, calculates an xy coordinate value corresponding to the uv coordinate value, extracts a pixel value of the coordinate value, and sets it as a uv pixel (see FIG. 5).

FIG. 4 is a flowchart showing this processing step. First, weighting per unit pixel in the u direction and v direction in the uv image is performed. Assume that the image size of the uv image is U × V pixels, and that the elevation angle φ is determined to be in a range of φ _min ≦ φ ≦ 90 degrees (steps 201 and 202). Thereby, the weighting per unit pixel in the u direction and the v direction is determined according to the equation (29).

  Further, the uv coordinate value range of the uv image is set from the center of the image to the origin (0, 0), from −U / 2 to + U / 2 in the u direction, and from −V / 2 to + V / 2 in the v direction. Integer m, n (−M ′ ≦ m ≦ M, −N ≦ n ≦ N, M: integer, N: integer, M and N are the same, that is, the uv image is square, representing each coordinate value in the u direction and v direction The uv coordinate value is obtained by the calculation of equation (30) (step 203).

While changing the integers m and n, the phase angle ρ and the elevation angle φ corresponding to the (u _ij , v _ij ) coordinate value are obtained by the calculation of the equation (31) (steps 204 and 205).

Next, an image coordinate value is calculated by the calculation of Expression (32). A value obtained by rounding the image coordinate values into an integer is set to (x _ij , y _ij ) (step 206).

A pixel value corresponding to the image coordinate value (x _ij , y _ij ) is extracted (step 207), and the pixel value is embedded in the coordinate value corresponding to the uv coordinate value on the uv image (step 208). By performing this operation for all uv coordinate values, a uv image is generated (step 209). Further, all the time series images of interest are sequentially used to generate a time series uv image using this conversion.

  Thereafter, by the same processing steps as in the first embodiment, from the temporal movement of the feature point of the omnidirectional video, the movement of the omnidirectional camera viewpoint, that is, the rotation around the optical axis and the translational movement in the three axis directions, and the object Three-dimensional information constituting the shape can be restored.

(Embodiment 3)
FIG. 6 a is a basic configuration diagram relating to the invention of claim 3. The restoration device according to the present embodiment includes a restoration processing determination unit 31 in the restoration device according to the first embodiment. Here, when the image coordinate value of the feature point obtained by the feature point measurement unit 11 is converted into the uv coordinate value by the measurement matrix conversion unit 12, the restoration process determination unit 31 performs a process for restoring the camera motion and the three-dimensional information. Since the determination point is different from the restoration device of the first embodiment, only this point will be described. Note that the time-series image database 10 may use a recording medium such as a hard disk, a RAID device, a CD-ROM, or a remote data resource via a network.

  FIG. 6 b shows a schematic configuration of a restoration device when processing is performed in real time, and the restoration device includes an image input unit 16 instead of the time-series image database 10. Therefore, the restoration apparatus of this embodiment does not necessarily require storage means such as each database unit.

  The measurement matrix conversion unit 12 converts the image coordinate value of the feature point obtained by the feature point measurement unit 11 into the uv coordinate value and holds the measurement matrix [B]. This measurement matrix [B] is held in the form of equation (5a). Next, processing in the restoration process determination unit 31 is performed.

FIG. 7 is a flowchart showing the process in the restoration process determination unit 31. When the measurement matrix [B] is input from the measurement matrix conversion unit 12 (step 301), this matrix is converted into a singular value matrix [W] by singular value decomposition by the calculation of equation (8) (step 302). . The singular value matrix [W] is a diagonal matrix, and the singular values that are the elements thereof are arranged in ascending / descending order and are all positive real numbers. Singular values of the 3 × 3 matrix element W ₃₃ and the 4 × 4 matrix element W ₄₄ are extracted from the singular values. In rank detection (step 303), the determination amount E _W is obtained by the calculation shown in the equation (33a) or the equation (33b).

The singular value W ₃₃ is a positive finite value when the camera performs a motion with 3 degrees of freedom or more, and the singular value W ₄₄ is a positive finite value when the camera performs a motion with 4 or more degrees of freedom. In addition, since it has a property that it becomes almost zero when exercise with less than 3 degrees of freedom is performed, the determination amount E _W can be used as a criterion for determining the degree of freedom of motion of the camera.

Then, it is determined whether the allowable value or less than δW is determined amount E _W, or determining the amount of E _W is tolerance δW above (step 304). This allowable value δW is a specific constant value, and the operator can set the value sequentially.

Here, when the determination amount E _W is less than a certain allowable value δw, it is determined that the camera motion has moved in a plane (step 305), and the process proceeds to process A (step 306). On the other hand, if the determination amount E _W is equal to or greater than a certain allowable value δW, it is determined that the camera motion has performed a motion with 4 degrees of freedom (step 307), and the process proceeds to process B (step 308).

  The process B performs the process of the first embodiment, and constitutes the movement of the omnidirectional camera viewpoint, that is, the rotation around the optical axis, the XYZ translational movement, and the object shape from the temporal movement of the feature points of the omnidirectional video. This process restores the dimension information. Therefore, hereinafter, the process A will be described.

  FIG. 8 shows a flowchart of the process A performed in the factorization processing unit 14.

  Each matrix data subjected to matrix decomposition into the three matrices [U], [W], and [V] of Expression (7) by the singular value decomposition performed in the restoration processing determination unit 15 is input (step 309). Here, [U] is a 2F × P size matrix, [W] is a P × P size diagonal matrix, and [V] is a P × P size matrix. First, in noise removal (step 310), the components of each matrix of rank 4 or higher are deleted as shown in the second term of equation (8). At the time of this deletion, the matrix [U] is taken out, the fourth to Pth columns are deleted from the elements of this matrix, the matrix consisting of the remaining components is held, the matrix [W] is taken out, The fourth to Pth rows and the fourth to Pth columns are deleted in the element, the matrix composed of the remaining components is retained, the matrix [V] is extracted, and the fourth to Pth rows are extracted from the elements of this matrix. Delete up to the eye and hold the matrix composed of the remaining components (Equation (9)). Next, from the matrix obtained by taking the square of the diagonal elements of the matrix [W] from which the 4th to Pth rows and the 4th to Pth columns are deleted, the matrix [U shown in Equations (10) and (11) is used. '] And matrix [V'] are obtained.

  Subsequently, in the transformation matrix calculation (step 311), the held matrix [U ′] is taken out, and the symmetric matrix [C] in the simultaneous conditional expressions shown in equations (13) to (15) (equation (12)). Calculate the coefficients for each element (shown). These coefficient calculations are easily obtained by matrix operation, and this conditional expression is calculated for all frames. Next, with respect to the simultaneous conditional expressions shown in the equations (13) to (15) for all frames, each element of the 3 × 3 size matrix [C] is changed by using numerical calculation such as the least square method. decide.

  Further, the obtained matrix [C] is subjected to eigenvalue decomposition as shown in Expression (16). Here, a matrix [C ′] of Expression (17) is generated from the square of the eigenvalue matrix and the eigenvalue matrix, and a matrix [Q] having this matrix element as a component is calculated according to Expression (18).

Subsequently, in the plane motion restoration (step 312), the matrix [M ′] is calculated from the matrix [Q] and the retained matrix [U ′] by the matrix calculation of Expression (19). The matrix element (m _ix , n _ix ) or (m _iy , n _iy ) of each frame (i-th frame) is extracted from the matrix [M ′], and the yaw rotation θ _i is restored by the calculation of equation (20). A matrix element (T _iu , T _iv ) of each frame (i-th frame) is extracted from the matrix [M ′]. From this (T _iu , T _iv ), the XY translational motion (Tx _i , Ty _i ) in the Euclidean space in the i-th frame is calculated by the calculation of equation (21).

Further, in the three-dimensional information restoration (step 313), the matrix operation shown in the equation (22) is performed from the matrix [v ′] held in advance and the matrix [Q] obtained by the conversion matrix calculation, The matrix [S ′] is obtained. Next, the transformation shown in Expression (23) is performed on the elements of the matrix [S ′], and the matrix having this as an element is defined as [P]. The column vector of the matrix [P] is a three-dimensional coordinate value (X _j , Y _j , Z _j ) in the Euclidean space of the jth feature point.

  As described above, according to the restoration device of the present embodiment, when the restoration process determination unit 31 determines that the camera motion is a plane movement, the time point of the feature point of the omnidirectional video is obtained by executing the process A. From the movement, the movement of the omnidirectional camera viewpoint, that is, the rotation around the optical axis and the XY translational movement, and the three-dimensional information constituting the object shape are restored.

  Further, when it is determined by the restoration process determination unit 31 that the camera movement is a movement with 4 degrees of freedom, as in the first embodiment, by performing the process B based on the process steps shown in FIG. From the temporal movement of the feature points of the image, the motion of the omnidirectional camera viewpoint, that is, the rotation around the optical axis and the translational motion in the three-axis direction, and the three-dimensional information constituting the object shape can be restored.

(Embodiment 4)
FIG. 9 a is a schematic configuration diagram of an embodiment for carrying out the invention of claim 4. The restoration apparatus according to the present embodiment includes the restoration processing determination unit 31 in the restoration apparatus according to the second embodiment. Here, when the uv coordinate value of each feature point is measured by the feature point measurement unit 11 in the time series uv image, the coordinate value is used for the restoration processing determination unit 31 to restore the camera motion and the three-dimensional information. Since this is different from the restoration processing apparatus of the second embodiment, only this point will be described. In this embodiment, the time-series image database may be in the form of using a recording medium such as a hard disk, a RAID device, a CD-ROM, or a remote data resource via a network.

  FIG. 9B is a processing configuration diagram in the case of processing in real time, and the restoration apparatus of this embodiment also uses the image input unit 16 instead of the time-series image database 10 as in the restoration apparatuses of the first, second, and third embodiments. I have. Therefore, the restoration apparatus of this embodiment does not necessarily require storage means such as each database unit 10.

The feature point measurement unit 11 obtains uv coordinate values of each feature point and holds a measurement matrix [B] having these as matrix elements. This measurement matrix [B] is held in the form of equation (5a). Next, the measurement matrix [B] is subjected to processing in the restoration processing determination unit 31. The processing flow in the restoration processing determination unit 31 is based on the flowchart shown in FIG. That is, the singular value decomposition is performed by using the measurement matrix [B] for the calculation of Expression (7) to obtain the singular value matrix [W]. The singular value matrix [W] is a diagonal element, and the singular values which are each element are arranged in ascending / descending order and are all positive real numbers. The singular values of the 3 × 3 matrix element W ₃₃ and the 4 × 4 matrix element W ₄₄ are extracted from the singular values. In rank detection, the determination amount E _W is obtained by the calculation shown in Expression (33a) or Expression (33b).

Then, it is determined whether the allowable value or less than δW is determined amount E _W, or determining the amount of E _W is tolerance δW more. The allowable value δW is a specific constant value, and the operator can set the value sequentially. If the determination amount E _W is less than a certain allowable value δW, it is determined that the camera motion is a plane motion, and the process proceeds to process A. On the other hand, when the determination amount Ew is equal to or greater than a certain allowable value δW, it is determined that the camera motion has performed a motion with 4 degrees of freedom, and the process proceeds to process B. The process B performs the same process as in the second embodiment, and from the temporal movement of the feature point of the omnidirectional video, the movement of the omnidirectional camera viewpoint, that is, the rotation around the optical axis and the translational movement in the three axis directions, This is a process for restoring the three-dimensional information constituting the shape. The process A is the same as the process A in the third embodiment, and a description thereof will be omitted.

  Next, various usage modes of the restoration apparatus of the present invention will be described.

  FIG. 11 shows a form in the case of shooting using the omnidirectional camera 1 provided with the restoration device of the present invention when there is an up-down direction such as the stairs 2. In this case, since there is a motion in the Z-axis direction, the restoration device of the present invention can restore the Z-axis motion and the three-dimensional information of the outside world at the same time.

  FIG. 12 shows a form in which the omnidirectional camera 1 provided with the restoration device of the present invention is hung and photographed. The figure on the left shows a form in which the omnidirectional camera 1 hung from the roof or ceiling of the building 5 by the horizontal maintaining unit 4 is moved up and down by a hanging means 3 such as a wire and photographed. The right figure is a form in the case of photographing while hanging the omnidirectional camera 1 by the horizontal maintaining part 4 in the hollow part of the hollow cylindrical member 6 such as piping. In any case, there is a movement in the Z-axis direction, and according to the restoration device of the present invention, the restoration of the Z-axis movement and the three-dimensional information of the outside world can be restored at the same time.

  FIG. 13 shows a form in which the omnidirectional camera 1 equipped with the restoring device of the present invention installed on the ceiling portion 70 of the room 7 is moved in the horizontal direction or in the vertical direction for shooting. In this case, according to the restoration device of the present invention, the room 7 can be three-dimensionalized, and the three-dimensional information on the objects 71 and 72 including the person 71 can be restored only by moving in the Z-axis direction. .

  FIG. 14 shows a case where the omnidirectional camera 1 equipped with the restoration device of the present invention is attached to the vehicle-mounted 8. When the motion and the three-dimensional information are restored using the omnidirectional camera 1 that accompanies the movement of the vehicle, it is difficult to restore it robustly because the omnidirectional camera vibrates minutely. According to the omnidirectional camera 1 equipped with the restoration device of the present invention, it is possible to robustly restore the three-dimensional information of the outside world even when the vehicle is driven or stopped. Even if it is not attached to the vehicle 8, the same effect can be obtained by attaching the omnidirectional camera 1 equipped with the restoring device of the present invention to a tripod and moving the camera in the vertical direction.

  In the restoration method described in the above embodiment, it is needless to say that the processing steps shown in FIGS. 1a to 9b can be configured by a computer program and the program can be executed by the computer. For recording a program for causing the computer to execute the process of the processing, such as a FD (floppy disk (registered trademark)), MO, ROM, memory card, CD, It can be recorded on a DVD, a removable disk, etc., and stored or distributed. It is also possible to provide the above program via a network such as the Internet or e-mail.

  Then, the present invention can be implemented by installing the program from these recording media into a computer, or by downloading the program from a network and installing the program into the computer. However, installation on a computer is a computer unit, and when there are multiple computers to be installed due to multiple devices and systems, it is natural that the program is installed for each necessary processing part. is there. In this case, the program may be recorded on a recording medium corresponding to a computer, or downloaded via a network.

The schematic block diagram which showed the example of the form of the apparatus which implements invention of Claim 1. The schematic block diagram which showed the example of the form of the apparatus which implements invention of Claim 1. The flowchart explaining the process process in the factorization method process part 15. FIG. A schematic configuration diagram showing an example of an apparatus for carrying out the invention according to claim 2. A schematic configuration diagram showing an example of an apparatus for carrying out the invention according to claim 2. The flowchart explaining the process process of uv image conversion in the image conversion process part 21. FIG. Explanatory drawing explaining uv image generation. A schematic configuration diagram showing an example of an apparatus for carrying out the invention according to claim 3. A schematic configuration diagram showing an example of an apparatus for carrying out the invention according to claim 3. The flowchart explaining the process process in the restoration process determination part 31. FIG. The flowchart explaining the process A performed in the factorization method process part 15. FIG. FIG. 5 is a schematic configuration diagram showing an example of an apparatus for carrying out the invention according to claim 4; FIG. 5 is a schematic configuration diagram showing an example of an apparatus for carrying out the invention according to claim 4; The figure which shows the omnidirectional camera, the spatial information to restore | restore, and a camera coordinate. The utilization form of the omnidirectional camera 1 provided with the decompression | restoration apparatus of this invention. The utilization form of the omnidirectional camera 1 provided with the decompression | restoration apparatus of this invention. The utilization form of the omnidirectional camera 1 provided with the decompression | restoration apparatus of this invention. The utilization form of the omnidirectional camera 1 provided with the decompression | restoration apparatus of this invention. An example of an image captured by an omnidirectional camera with an outline attached to an object. An example of an image taken by an omnidirectional camera, showing only the outline of the object. An example of an image captured by an omnidirectional camera without a contour line attached to an object. An example of an image obtained by uv coordinate conversion of an image captured by an omnidirectional camera, in which a contour is attached to an object. An example of an image obtained by uv coordinate conversion of an image captured by an omnidirectional camera, showing only the outline of an object. An example of an image obtained by uv coordinate conversion of an image captured by an omnidirectional camera, in which a contour is not attached to an object.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Time series image database, 11 ... Feature point measurement part, 12 ... Measurement matrix conversion part, 13 ... Intersection coordinate measurement part, 14 ... Measurement matrix input part, 15 ... Molecular decomposition method processing part, 16 Image input part 21 ... Image Conversion processing unit 31 ... Restoration processing determination unit

Claims (16)

The movement of the omnidirectional camera viewpoint in time series based on the amount of temporal change in the image coordinate values of the feature points placed in the target image in the chronological omnidirectional image (or wide-field image) acquired from the omnidirectional camera. And a method for restoring the three-dimensional information constituting the object shape of the outside world,
In the feature point coordinate system set in the acquired time-series omnidirectional image (or wide-field image), a step of measuring an image coordinate value indicating a temporal change amount of the feature point in each image (measurement matrix measurement step);
Another coordinate value (uv coordinate value) is obtained from the phase angle and elevation angle calculated from the image coordinate value of each feature point obtained in the measurement matrix measurement step, and the restoration process is performed from the uv coordinate value of each feature point. Generate matrix data (matrix decomposition data), perform singular value decomposition on the matrix decomposition data and perform noise removal to obtain matrix data representing motion information and matrix data representing three-dimensional information. In the component of the motion information, A transformation matrix that satisfies the conditions set to define the motion is obtained, and this transformation matrix is applied to the matrix data that becomes the motion information, and the rotational motion around the optical axis of the omnidirectional camera viewpoint and the plane perpendicular to the axis ( 3rd order that reconstructs translational motion on the XY plane (this motion with 3 degrees of freedom is called planar motion), and further forms an object shape by applying an inverse matrix of this transformation matrix to matrix data representing three-dimensional information. And step (restoration step of planar motion and three-dimensional information) to restore the information,
The reprojected coordinate value in the uv coordinate system for each feature point is obtained from the planar motion and 3D information obtained in the plane motion and 3D information restoration step, and the reprojected coordinate value and matrix elements constituting the matrix decomposition data Using this error and the Z coordinate value (component of 3D information) of the feature point obtained in the plane motion and 3D information restoration step, the translation of the camera viewpoint in the Z-axis direction is calculated. A step of restoring motion (Z-axis translational motion restoration step);
It is determined whether the error obtained in the Z-axis translational motion restoration step has converged to a certain value or less. If not, the planar motion obtained in the planar motion and three-dimensional information restoration step. Then, the deformation coefficient is obtained from the three-dimensional information, and the deformation coefficient is combined with the uv coordinate value of each feature point to obtain a new uv coordinate value, and matrix decomposition data having this as a matrix element is generated to generate the plane motion and 3 Returning to the dimensional information restoration step, iterating until the error converges to a certain value or less, and omnidirectional camera motion and three-dimensional information restoration step. How to restore. The movement of the omnidirectional camera viewpoint in time series based on the amount of temporal change in the image coordinate values of the feature points placed in the target image in the chronological omnidirectional image (or wide-field image) acquired from the omnidirectional camera. , As well as a method for restoring the three-dimensional information constituting the object shape of the outside world,
With respect to the acquired time-series omnidirectional image (or wide-field image), each image coordinate value is converted into a uv coordinate value by a phase angle and an elevation angle obtained from each image coordinate value, and each pixel is converted to its uv coordinate value. A time series image (uv image) is generated so as to be associated, and in the feature point coordinate system set in the uv image, a uv coordinate value indicating a temporal change amount of the feature point in each image is measured (uv measurement matrix measurement) Step) and
Generate matrix data (matrix decomposition data) for restoration processing from the uv coordinate values of each feature point obtained in the uv measurement matrix measurement step, perform singular value decomposition on the matrix decomposition data, perform noise removal, and obtain motion information. The matrix data representing the three-dimensional information and the matrix data representing the three-dimensional information are obtained, and the transformation matrix satisfying the conditions set for defining the motion is obtained in the motion information component. The rotational motion around the optical axis of the omnidirectional camera viewpoint and the translational motion on the plane perpendicular to the axis (XY plane) (this three-degree-of-freedom motion is referred to as plane motion) are restored, and three-dimensional A step of restoring the three-dimensional information constituting the object shape by applying an inverse matrix of this transformation matrix to matrix data representing information (a step of restoring plane motion and three-dimensional information);
The reprojected coordinate value in the uv coordinate system for each feature point is obtained from the planar motion and 3D information obtained in the plane motion and 3D information restoration step, and the reprojected coordinate value and matrix elements constituting the matrix decomposition data Using this error and the Z coordinate value (component of 3D information) of the feature point obtained in the plane motion and 3D information restoration step, the translation of the camera viewpoint in the Z-axis direction is calculated. A step of restoring motion (Z-axis translational motion restoration step);
It is judged whether the error obtained by this Z-axis translational motion recovery snubber has converged to a certain value or less. Then, the deformation coefficient is obtained from the three-dimensional information, and the deformation coefficient is combined with the uv coordinate value of each feature point to obtain a new uv coordinate value, and matrix decomposition data having this as a matrix element is generated to generate the plane motion and 3 Returning to the dimensional information restoration step, iterating until the error converges to a certain value or less, and omnidirectional camera motion and three-dimensional information restoration step. How to restore. In the measurement matrix measurement step, in the singular value component obtained by singular value decomposition of the measurement matrix, a determination value representing the degree of freedom of movement is calculated from the component,
If the determination value is less than a certain value, the camera motion is regarded as a plane motion, and in the step of restoring the omnidirectional camera motion and the three-dimensional information, the degree of freedom comprising rotation about the optical axis and motion on a certain plane 3 to restore the plane motion and 3D information,
If the determination value is equal to or greater than a certain value, the camera motion is considered to be a motion with 4 degrees of freedom, and the omnidirectional camera motion and the step of restoring the three-dimensional information comprise rotation and translational motions related to the camera. 2. The method of restoring an omnidirectional camera viewpoint motion and three-dimensional information according to claim 1, wherein the four-dimensional motion and the external three-dimensional information are restored. In the uv measurement matrix measurement step, in a singular value component obtained by singular value decomposition of the measurement matrix, a determination value representing motion is calculated from the component,
If the determination value is less than a certain value, the camera motion is regarded as a plane motion, and in the step of restoring the omnidirectional camera motion and the three-dimensional information, the degree of freedom comprising rotation about the optical axis and motion on a certain plane 3 to restore the plane motion and 3D information,
The restoration method according to claim 2, wherein if the determination value is equal to or greater than a certain value, the camera movement is regarded as a movement with 4 degrees of freedom, and the omnidirectional camera movement and the three-dimensional information are restored. The method of restoring an omnidirectional camera viewpoint motion and three-dimensional information according to claim 2, wherein the four-degree-of-freedom motion consisting of rotation and translational motion about the camera and three-dimensional information of the outside world are restored. The movement of the omnidirectional camera viewpoint in time series based on the amount of temporal change in the image coordinate values of the feature points placed in the target image in the chronological omnidirectional image (or wide-field image) acquired from the omnidirectional camera. And a device for restoring the three-dimensional information constituting the object shape of the outside world,
A feature point measurement unit that measures coordinate values of feature points in each image;
A measurement matrix conversion unit that converts the image coordinate values obtained by the feature point measurement unit into a uv image;
A measurement matrix input unit for giving a matrix having the uv coordinate value of the uv image as a matrix element as an initial value;
An omnidirectional camera motion and three-dimensional information characterized by comprising a factorization method processing unit for restoring the three-dimensional information of the camera motion and the outside world by factorization from the data supplied from the measurement matrix input unit Restore device. The movement of the omnidirectional camera viewpoint in time series based on the amount of temporal change in the image coordinate values of the feature points placed in the target image in the chronological omnidirectional image (or wide-field image) acquired from the omnidirectional camera. And a device for restoring the three-dimensional information constituting the object shape of the outside world,
An image conversion processing unit for converting each image into a uv image;
A feature point measurement unit that measures coordinate values of feature points in each image;
A measurement matrix input unit for giving a matrix having the uv coordinate value of the uv image as a matrix element as an initial value;
An omnidirectional camera motion and three-dimensional information restoration comprising a factorization processing unit that restores the camera motion and the external three-dimensional information by factorization from the data supplied from the measurement matrix input unit apparatus. A restoration processing determination unit that calculates a determination value representing a degree of freedom of movement from a singular value component obtained by performing singular value decomposition on the uv image supplied from the measurement matrix conversion unit;
If the judgment value is less than a certain value, it is considered that the camera motion is a plane motion,
The factorization processing unit performs a process of restoring three-dimensional information and three-dimensional information with three degrees of freedom consisting of rotation around the optical axis and movement on a plane perpendicular to the optical axis,
If the determination value is equal to or greater than a certain value, the camera motion is regarded as having a motion of 4 degrees of freedom, and the factorization processing unit performs a motion of 4 degrees of freedom consisting of rotation and translational motion related to the camera and the 3D of the external world. 6. The apparatus for restoring omnidirectional camera motion and three-dimensional information according to claim 5, wherein processing for restoring information is performed. A restoration process determination unit that calculates a determination value representing motion from a singular value component obtained by performing singular value decomposition of the uv image from the feature point measurement unit;
When the judgment value is less than a certain value, it is considered that the camera motion is a plane motion, and the factorization processing unit has a degree of freedom comprising a rotation around the optical axis and a motion on a plane perpendicular to the optical axis. 3 to restore the plane motion and 3D information,
If the determination value is equal to or greater than a certain value, the camera motion is regarded as having a motion of 4 degrees of freedom, and the factorization processing unit performs a motion of 4 degrees of freedom consisting of rotation and translational motion related to the camera and the 3D of the external world. 7. The apparatus for restoring omnidirectional camera motion and three-dimensional information according to claim 6, wherein processing for restoring information is performed. The movement of the omnidirectional camera viewpoint in time series based on the amount of temporal change in the image coordinate values of the feature points placed in the target image in the chronological omnidirectional image (or wide-field image) acquired from the omnidirectional camera. And a program for executing, by a computer, a method for restoring the three-dimensional information constituting the object shape of the outside world,
In the feature point coordinate system set in the acquired time-series omnidirectional image (or wide-field image), a step of measuring an image coordinate value indicating a temporal change amount of the feature point in each image (measurement matrix measurement step);
Another coordinate value (uv coordinate value) is obtained from the phase angle and elevation angle calculated from the image coordinate value of each feature point obtained in the measurement matrix measurement step, and the restoration process is performed from the uv coordinate value of each feature point. Generate matrix data (matrix decomposition data), perform singular value decomposition on the matrix decomposition data and perform noise removal to obtain matrix data representing motion information and matrix data representing three-dimensional information. In the component of the motion information, A transformation matrix that satisfies the conditions set to define the motion is obtained, and this transformation matrix is applied to the matrix data that becomes the motion information, and the rotational motion around the optical axis of the omnidirectional camera viewpoint and the plane perpendicular to the axis ( 3rd order that reconstructs translational motion on the XY plane (this motion with 3 degrees of freedom is called planar motion), and further forms an object shape by applying an inverse matrix of this transformation matrix to matrix data representing three-dimensional information. And step (restoration step of planar motion and three-dimensional information) to restore the information,
The reprojected coordinate value in the uv coordinate system for each feature point is obtained from the planar motion and 3D information obtained in the plane motion and 3D information restoration step, and the reprojected coordinate value and matrix elements constituting the matrix decomposition data Using this error and the Z coordinate value (component of 3D information) of the feature point obtained in the plane motion and 3D information restoration step, the translation of the camera viewpoint in the Z-axis direction is calculated. A step of restoring motion (Z-axis translational motion restoration step);
It is determined whether the error obtained in the Z-axis translational motion restoration step has converged to a certain value or less. If not, the planar motion obtained in the planar motion and three-dimensional information restoration step. Then, the deformation coefficient is obtained from the three-dimensional information, and the deformation coefficient is combined with the uv coordinate value of each feature point to obtain a new uv coordinate value, and matrix decomposition data having this as a matrix element is generated to generate the plane motion and 3 Returning to the dimensional information restoration step, iterating until the error converges to a certain value or less, and omnidirectional camera motion and three-dimensional information restoration step. Program that performs the restore method. The movement of the omnidirectional camera viewpoint in time series based on the amount of temporal change in the image coordinate values of the feature points placed in the target image in the chronological omnidirectional image (or wide-field image) acquired from the omnidirectional camera. , As well as a program for executing, by a computer, a method for restoring the three-dimensional information constituting the object shape of the outside world,
With respect to the acquired time-series omnidirectional image (or wide-field image), each image coordinate value is converted into a uv coordinate value by a phase angle and an elevation angle obtained from each image coordinate value, and each pixel is converted to its uv coordinate value. A time series image (uv image) is generated so as to be associated, and in the feature point coordinate system set in the uv image, a uv coordinate value indicating a temporal change amount of the feature point in each image is measured (uv measurement matrix measurement) Step) and
Generate matrix data (matrix decomposition data) for restoration processing from the uv coordinate values of each feature point obtained in the uv measurement matrix measurement step, perform singular value decomposition on the matrix decomposition data, perform noise removal, and obtain motion information. The matrix data representing the three-dimensional information and the matrix data representing the three-dimensional information are obtained, and the transformation matrix satisfying the conditions set for defining the motion is obtained in the motion information component. The rotational motion around the optical axis of the omnidirectional camera viewpoint and the translational motion on the plane perpendicular to the axis (XY plane) (this three-degree-of-freedom motion is referred to as plane motion) are restored, and three-dimensional A step of restoring the three-dimensional information constituting the object shape by applying an inverse matrix of this transformation matrix to matrix data representing information (a step of restoring plane motion and three-dimensional information);
The reprojected coordinate value in the uv coordinate system for each feature point is obtained from the planar motion and 3D information obtained in the plane motion and 3D information restoration step, and the reprojected coordinate value and matrix elements constituting the matrix decomposition data Using this error and the Z coordinate value (component of 3D information) of the feature point obtained in the plane motion and 3D information restoration step, the translation of the camera viewpoint in the Z-axis direction is calculated. A step of restoring motion (Z-axis translational motion restoration step);
It is judged whether the error obtained by this Z-axis translational motion recovery snubber has converged to a certain value or less. Then, the deformation coefficient is obtained from the three-dimensional information, and the deformation coefficient is combined with the uv coordinate value of each feature point to obtain a new uv coordinate value, and matrix decomposition data having this as a matrix element is generated to generate the plane motion and 3 Returning to the dimensional information restoration step, iterating until the error converges to a certain value or less, and omnidirectional camera motion and three-dimensional information restoration step. Program that performs the restore method. In the measurement matrix measurement step, in the singular value component obtained by singular value decomposition of the measurement matrix, a determination value representing the degree of freedom of movement is calculated from the component,
If the determination value is less than a certain value, the camera motion is regarded as a plane motion, and in the step of restoring the omnidirectional camera motion and the three-dimensional information, the degree of freedom comprising rotation about the optical axis and motion on a certain plane 3 to restore the plane motion and 3D information,
If the determination value is equal to or greater than a certain value, the camera motion is considered to be a motion with 4 degrees of freedom, and the omnidirectional camera motion and the step of restoring the three-dimensional information comprise rotation and translational motions related to the camera. 10. The program for executing an omnidirectional camera viewpoint motion and three-dimensional information restoration method according to claim 9, wherein the four-degree-of-freedom motion and the external three-dimensional information are restored. In the uv measurement matrix measurement step, in a singular value component obtained by singular value decomposition of the measurement matrix, a determination value representing motion is calculated from the component,
If the determination value is less than a certain value, the camera motion is regarded as a plane motion, and in the step of restoring the omnidirectional camera motion and the three-dimensional information, the degree of freedom comprising rotation about the optical axis and motion on a certain plane 3 to restore the plane motion and 3D information,
The restoration method according to claim 2, wherein if the determination value is equal to or greater than a certain value, the camera movement is regarded as a movement with 4 degrees of freedom, and the omnidirectional camera movement and the three-dimensional information are restored. 11. The program for executing an omnidirectional camera viewpoint motion and three-dimensional information restoration method according to claim 10, wherein the four-degree-of-freedom motion consisting of rotation and translation about the camera and three-dimensional information of the outside world are restored. The movement of the omnidirectional camera viewpoint in time series based on the amount of temporal change in the image coordinate values of the feature points placed in the target image in the chronological omnidirectional image (or wide-field image) acquired from the omnidirectional camera. And a computer-readable recording medium recording a program for executing a method for restoring the three-dimensional information constituting the object shape of the outside world by a computer,
In the feature point coordinate system set in the acquired time-series omnidirectional image (or wide-field image), a step of measuring an image coordinate value indicating a temporal change amount of the feature point in each image (measurement matrix measurement step);
Another coordinate value (uv coordinate value) is obtained from the phase angle and elevation angle calculated from the image coordinate value of each feature point obtained in the measurement matrix measurement step, and the restoration process is performed from the uv coordinate value of each feature point. Generate matrix data (matrix decomposition data), perform singular value decomposition on the matrix decomposition data and perform noise removal to obtain matrix data representing motion information and matrix data representing three-dimensional information. In the component of the motion information, A transformation matrix that satisfies the conditions set to define the motion is obtained, and this transformation matrix is applied to the matrix data that becomes the motion information, and the rotational motion around the optical axis of the omnidirectional camera viewpoint and the plane perpendicular to the axis ( 3rd order that reconstructs translational motion on the XY plane (this motion with 3 degrees of freedom is called planar motion), and further forms an object shape by applying an inverse matrix of this transformation matrix to matrix data representing three-dimensional information. And step (restoration step of planar motion and three-dimensional information) to restore the information,
The reprojected coordinate value in the uv coordinate system for each feature point is obtained from the planar motion and 3D information obtained in the plane motion and 3D information restoration step, and the reprojected coordinate value and matrix elements constituting the matrix decomposition data Using this error and the Z coordinate value (component of 3D information) of the feature point obtained in the plane motion and 3D information restoration step, the translation of the camera viewpoint in the Z-axis direction is calculated. A step of restoring motion (Z-axis translational motion restoration step);
It is determined whether the error obtained in the Z-axis translational motion restoration step has converged to a certain value or less. If not, the planar motion obtained in the planar motion and three-dimensional information restoration step. Then, the deformation coefficient is obtained from the three-dimensional information, and the deformation coefficient is combined with the uv coordinate value of each feature point to obtain a new uv coordinate value, and matrix decomposition data having this as a matrix element is generated to generate the plane motion and 3 Returning to the dimensional information restoration step, iterating until the error converges to a certain value or less, and omnidirectional camera motion and three-dimensional information restoration step. A recording medium on which a program for executing the restoration method is recorded. The movement of the omnidirectional camera viewpoint in time series based on the amount of temporal change in the image coordinate values of the feature points placed in the target image in the chronological omnidirectional image (or wide-field image) acquired from the omnidirectional camera. And a computer-readable recording medium storing a program for executing a method for restoring three-dimensional information constituting an object shape of the outside world by a computer,
With respect to the acquired time-series omnidirectional image (or wide-field image), each image coordinate value is converted into a uv coordinate value by a phase angle and an elevation angle obtained from each image coordinate value, and each pixel is converted to its uv coordinate value. A time series image (uv image) is generated so as to be associated, and in the feature point coordinate system set in the uv image, a uv coordinate value indicating a temporal change amount of the feature point in each image is measured (uv measurement matrix measurement) Step) and
Generate matrix data (matrix decomposition data) for restoration processing from the uv coordinate values of each feature point obtained in the uv measurement matrix measurement step, perform singular value decomposition on the matrix decomposition data, perform noise removal, and obtain motion information. The matrix data representing the three-dimensional information and the matrix data representing the three-dimensional information are obtained, and the transformation matrix satisfying the conditions set for defining the motion is obtained in the motion information component. The rotational motion around the optical axis of the omnidirectional camera viewpoint and the translational motion on the plane perpendicular to the axis (XY plane) (this three-degree-of-freedom motion is referred to as plane motion) are restored, and three-dimensional A step of restoring the three-dimensional information constituting the object shape by applying an inverse matrix of this transformation matrix to matrix data representing information (a step of restoring plane motion and three-dimensional information);
The reprojected coordinate value in the uv coordinate system for each feature point is obtained from the planar motion and 3D information obtained in the plane motion and 3D information restoration step, and the reprojected coordinate value and matrix elements constituting the matrix decomposition data Using this error and the Z coordinate value (component of 3D information) of the feature point obtained in the plane motion and 3D information restoration step, the translation of the camera viewpoint in the Z-axis direction is calculated. A step of restoring motion (Z-axis translational motion restoration step);
It is judged whether the error obtained by this Z-axis translational motion recovery snubber has converged to a certain value or less. Then, the deformation coefficient is obtained from the three-dimensional information, and the deformation coefficient is combined with the uv coordinate value of each feature point to obtain a new uv coordinate value, and matrix decomposition data having this as a matrix element is generated to generate the plane motion and 3 Returning to the dimensional information restoration step, iterating until the error converges to a certain value or less, and omnidirectional camera motion and three-dimensional information restoration step. Recording medium on which a program for executing the restoration method is recorded by a computer. In the measurement matrix measurement step, in the singular value component obtained by singular value decomposition of the measurement matrix, a determination value representing the degree of freedom of movement is calculated from the component,
If the determination value is less than a certain value, the camera motion is regarded as a plane motion, and in the step of restoring the omnidirectional camera motion and the three-dimensional information, the degree of freedom comprising rotation about the optical axis and motion on a certain plane 3 to restore the plane motion and 3D information,
If the determination value is equal to or greater than a certain value, the camera motion is considered to be a motion with 4 degrees of freedom, and the omnidirectional camera motion and the step of restoring the three-dimensional information comprise rotation and translational motions related to the camera. 14. A recording medium storing a program for executing the omnidirectional camera viewpoint movement and the three-dimensional information restoring method by a computer according to claim 13, wherein the movement has four degrees of freedom and the three-dimensional information of the outside world is restored. In the uv measurement matrix measurement step, in a singular value component obtained by singular value decomposition of the measurement matrix, a determination value representing motion is calculated from the component,
If the determination value is less than a certain value, the camera motion is regarded as a plane motion, and in the step of restoring the omnidirectional camera motion and the three-dimensional information, the degree of freedom comprising rotation about the optical axis and motion on a certain plane 3 to restore the plane motion and 3D information,
The restoration method according to claim 2, wherein if the determination value is equal to or greater than a certain value, the camera movement is regarded as a movement with 4 degrees of freedom, and the omnidirectional camera movement and the three-dimensional information are restored. 15. The computer-executable computer program for restoring an omnidirectional camera viewpoint motion and three-dimensional information according to claim 14, wherein the four-degree-of-freedom motion consisting of rotation and translational motion and three-dimensional information of the outside world are restored. A recording medium on which is recorded.

Images