JP2006252275A - Restoration system of camera motion and object shape - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To highly accurately and efficiently restore a camera motion and the three-dimensional shape of an object by using only an image sequence photographed while moving a camera. <P>SOLUTION: In initial estimation, a degeneration reference value is calculated (S13) for a feature point sequence (S12) appearing in common from an image 1 to an image k, k is defined as the appropriate number of images to be used for the initial estimation in the case that the degeneration reference value exceeds a threshold (Yes in S14), and the camera motion and the object shape are obtained. Also, in order to further improve accuracy, a partial set g<SB>1</SB>to be the solution candidate of the camera motion and the object shape and its restoration reference value are obtained (S15) by using robust estimation for the feature point sequence (S12), the degeneration reference value is calculated for the partial set g<SB>i</SB>whose restoration reference value exceeds a threshold (S16), k is defined as the appropriate number of the images to be used for the initial estimation in the case that the degeneration reference value exceeds the threshold (Yes in S17), and the partial set g<SB>i</SB>of the best score is adopted as a feature point sequence for obtaining the camera motion and the object shape. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a system for simultaneously restoring camera motion and object shape from a sequence of images taken while moving a camera.

Methods for restoring the three-dimensional shape of an object from a camera image sequence include a method based on epipolar geometry and a factorization method. In both methods, a feature point is associated between images, and the three-dimensional motion (translation and rotation) of the camera and the three-dimensional shape of the object are estimated from the position of the feature point on the image. Here, the three-dimensional shape of the object is represented by a set of feature points restored in a three-dimensional space. For example, as proposed in Non-Patent Document 1, feature points are extracted and matched, and features corresponding to corner points and intersections of image edges are extracted and matched well with the neighborhood of feature points in the current image. This is realized by searching the area to be performed within a predetermined range from the next image.
In the method based on epipolar geometry, the basic equations that hold between the position and orientation of the camera, the position of the image feature point, and the position of the three-dimensional feature point are arranged between the two images for eight or more feature points. It is common to solve the equation to obtain a basic matrix and restore the camera motion and the object shape based on the basic matrix (Non-patent Document 2, pages 262 to 265).
In the factorization method, a linear model such as weak perspective projection is assumed as the camera model, the measurement matrix in which image feature points are arranged is decomposed into a camera motion matrix and an object shape matrix by singular value decomposition, and the orthogonal constraint of the coordinate axes is set. It is common to perform Euclidean reconstruction using this method (Non-patent Document 3).

However, the above-described prior art method has a problem that the restoration result becomes unstable when the translational component of the camera motion is small with respect to the distance to the target object, both in the method based on epipolar geometry and in the factorization method. . For example, if the translation component is 0, degeneration occurs and no solution can be obtained in principle. In an actual image, there is no complete degeneration due to the influence of an image feature point extraction error or the like, and a restored image in which an object shape is crushed on a plane is obtained. When the translational component is not 0 but is very small, the same phenomenon occurs to a different extent. In this specification, this case is also referred to as degeneration.
FIG. 5 shows an example of degeneration. FIG. 5A is an example of an image in an image sequence used for restoration. The target object shown in (a) is a magazine that intersects at a right angle, and the camera moves in the horizontal direction while looking at the magazine in front and takes an image sequence. White points on the image are extracted feature points. FIGS. 5B and 5C show restoration results displayed from the viewpoint of the target object (magazine) viewed from the side, and FIG. 5B shows the restoration results when the translational component of the camera motion is small. In the figure, the arrow indicated by 510 is the direction of the line of sight of the camera, and the point indicated by 520 is the object shape restoration result. FIG. 5C is a diagram showing a restoration result when the translational component is large. The arrow indicated by 530 is the direction of the camera line of sight, and the point indicated by 540 is the restoration result of the object shape. Originally, the restoration points should be arranged on a plane intersecting at right angles as shown in FIG. 5C, but in FIG. 5B, since the translational components are small, all feature points are arranged on the same plane. Has been restored.

In order to cope with such degeneration, it is necessary to capture an image sequence so that the translational component of the camera motion is sufficiently large, or to select an image group in which the translational component is large from the image sequence. In a system in which humans intervene, it suffices for humans to select images visually, but it takes time to select an appropriate number of images. In addition, when shape restoration is performed in a system in which a camera is mounted on a mobile robot, a mechanism for automatically selecting an appropriate image is necessary for the system.
One method for dealing with this problem is to measure the translation component of the camera motion and the distance to the target object using another sensor. For example, in Patent Document 1, the translational component of a camera is measured with an acceleration sensor or the like, and the result of measuring the distance to an object with a distance sensor is used to perform highly accurate shape restoration with little influence of noise. Is going. However, this method uses a sensor other than a camera, such as an acceleration sensor or a distance sensor, so that there is a problem that the apparatus becomes large. There is also a problem that the applicable distance range is limited to the specifications of the distance sensor.
On the other hand, as a method of detecting degeneration only from an image, a method for evaluating whether a set of image feature points conforms to epipolar geometry or planar homography, and determines that degeneration has occurred when following planar homography Is proposed in Non-Patent Document 4. Further, Non-Patent Document 5 proposes a method for selecting an image interval that can provide a sufficient amount of translation using this criterion.
However, this method requires calculation of planar homography in addition to epipolar geometry. In particular, robust estimation is performed for both epipolar geometry and planar homography to cope with image feature point errors. There is a problem that a lot of processing time is required.

Japanese Patent Laid-Open No. 11-063949 “3-D Shape Restoration Apparatus and Method” J. Shi and C. Tomasi: “Good Features to Track,” Proceedings of International Conference on Computer Vision and Pattern Recognition, pp. 593-600, 1994. R. Hartley and A. Zisserman: “Multiple View Geometry in Computer Vision,” Cambridge University Press, 2000. C. Tomasi and T. Kanade: “Shape and Motion from Image Streams under Orthography: A Factorization Approach,” International Journal of Computer Vision, Vol.9, No.2, pp.137-154, 1992. P. H. S. Torr, A. W. Fitzgibbon, and A. Zisserman: “The Problem of Degeneracy in Structure and Motion Recovery from Uncalibrated Image Sequences,” International Journal of Computer Vision, Vol. 32, No. 1, pp. 27-44, 1999. D. Nister: “Reconstruction from Uncalibrated Sequences with a Hierarchy of Trifocal Tensors,” Proceedings of European Conference on Computer Vision, pp. 649-663, 2000.

  An object of the present invention is to provide an image sequence from a camera when simultaneously reconstructing a camera motion and a three-dimensional shape of an object from an image sequence taken while moving a camera (hereinafter, the image input device is generically called a camera). It is to construct a restoration system that can efficiently realize high-precision camera motion and object shape restoration using

In order to achieve the above object, the present invention provides a camera motion / object shape restoration system for restoring a camera motion and a three-dimensional shape of an object from an image sequence taken while moving the camera. Is extracted and tracked by the feature point extracting / tracking means for storing the matched feature points in the feature point storing means, and an initial value k is given. The feature points up to the image are read from the feature point storage means, a basic equation matrix is generated from the feature points, a singular value of the basic equation matrix is obtained, a degenerate reference value is calculated from the singular value, and the degeneration is performed. When the reference value does not exceed the predetermined threshold, the calculation of the degeneration reference value is repeated while increasing k, and when the degeneration reference value exceeds the threshold, the feature points from the first image to the kth image First camera from Initial estimation means for obtaining the posture and the three-dimensional position of the feature point of the object and storing them in the camera movement storage means and the three-dimensional shape storage means, respectively, and the camera posture and the three-dimensional shape storage stored in the camera movement storage means And a camera motion / object shape reading means for reading out the object shape stored in the means.
The camera motion / object shape restoration system further includes a camera that obtains the camera posture of the current image in order from the (k + 1) th image to the last image after the processing of the initial estimation unit, and stores it in the camera motion storage unit There may be provided sequential estimation means for alternately and repeatedly performing posture estimation and object shape restoration obtained by obtaining the three-dimensional position of the feature point and fusing with the three-dimensional shape storage means.
The initial estimation means obtains a subset of feature points used for camera motion and object shape restoration by robust estimation from the feature points from the first image to the k-th image, and a basic equation matrix from the subset of the feature points Generating a singular value of the basic equation matrix and calculating a degeneration reference value from the singular value. If the degeneration reference value does not exceed a predetermined threshold, the calculation of the degeneration reference value is repeated while increasing k, When the degeneration reference value exceeds the threshold value, the initial posture of the camera and the three-dimensional position of the object feature point are obtained from the subset of the feature points and stored in the camera motion storage unit and the three-dimensional shape storage unit, respectively. The initial estimation means may obtain a restoration reference value for each subset when a plurality of subsets of the feature points are obtained, and calculate from the restoration reference value and the degenerate reference value. The initial posture of the camera and the three-dimensional position of the feature point of the object may be obtained from the subset with the best score to be calculated and stored in the camera motion storage means and the three-dimensional shape storage means, respectively.
The initial estimation means obtains a degeneration reference value from the feature points from the first image to the k-th image before obtaining the subset of the feature points, and if the degeneration reference value does not exceed a predetermined threshold value The calculation of the degeneration reference value may be repeated while increasing k, and the feature point subset may be obtained when the degeneration reference value exceeds the threshold.
The object shape restoration in the successive estimation means is performed by performing three-dimensional restoration on the feature point of the current image and the corresponding feature point of the image at each past time point stored in the feature point storage means. A covariance matrix may be obtained, and a three-dimensional restoration result in which the value of the covariance matrix is minimum and less than a predetermined threshold may be merged with the three-dimensional shape storage means as the three-dimensional position of the feature point.
The camera motion / object shape restoration program for configuring the computer system with the functions of the camera motion / object shape restoration system described above and a recording medium recording the program are also the present invention.

According to the camera motion and object shape restoration system of the present invention, translation of camera motion is performed using only an image sequence from the camera without requiring a human instruction or a sensor other than the camera as in the prior art. It is possible to automatically select the number of images that can ensure a sufficiently large component, and to efficiently realize highly accurate camera movement and three-dimensional shape reconstruction of the object.
Such a restoration system of the present invention automatically restores camera motion and object shape from an image sequence obtained by mounting a camera on a mobile robot or a moving mechanism such as an automobile or airplane, This is very effective when a three-dimensional object shape is read and used. Further, by targeting the entire environment as a three-dimensional object, it is also useful when restoring a three-dimensional map of the environment and reading out and using the environment map and camera motion.

Hereinafter, an example of an embodiment of the camera motion and object shape restoration system of the present invention will be described in detail with reference to the drawings. It should be noted that here, the entire image input apparatus is referred to as “camera”.
First, (1) the outline of the processing of the camera motion and object shape restoration system of this embodiment will be described, and then (2) the baseline length selection method (for restoration) which is a new method used in the restoration system of this embodiment. A method for determining an appropriate number of images will be described in detail. Finally, (3) the results of experiments using the restoration system of this embodiment are shown.

(1. Overview of processing of camera motion and object shape restoration system)
First, an outline of processing of the camera motion and object shape restoration system of the present embodiment will be described. As an example, here, the restoration system of the present invention is constructed in a computer such as a personal computer, an image sequence photographed by a camera is input to the computer, and the camera motion and the three-dimensional object are automatically obtained from the input image sequence. Description will be made assuming that the object shape is restored. For example, the restoration system of the present invention can be constructed inside a mobile robot equipped with a camera as described above, and the stored three-dimensional object shape and camera movement (robot movement) can be read and used.
In this embodiment, without using an internal sensor such as odometry, the camera posture is estimated from a sequence of images taken with a monocular camera with a standard lens, and at the same time, a three-dimensional object shape is restored. A three-dimensional object shape is a set of points obtained by restoring feature points extracted from an image in three dimensions. Assume that the camera posture has six degrees of freedom in alignment with the position in a three-dimensional space. When the present invention is used for a mobile robot equipped with a camera, the camera posture and the robot posture are regarded as the same. Camera motion is used in the sense of time series of camera posture, but camera posture and camera motion may be used without distinction.

The flow of processing of the camera motion and object shape restoration system of this embodiment is shown in FIG. Regarding the system overview described here, the basic flow is in line with the system of Yamazaki and Tono (K. Yamazaki, M. Tomono, T. Tsubouchi, and S. Yuta: “3-D Object Modeling by a Camera Equipped on a Mobile Robot, ”Proceedings of the IEEE International Conference on Robotics and Automation, 2004.)) In this embodiment, using the baseline length selection method, which is a new method of this embodiment, In the initial estimation process (S150 in FIG. 1), an appropriate number m of images that can restore the camera posture and the three-dimensional object shape without degeneration is determined. The baseline length selection method will be described in the next section (2. Baseline length selection method). In addition, in the object shape restoration process in the successive estimation process, an appropriate number of images that minimizes the restoration error is obtained.
The internal parameters of the camera are assumed to be known. The time t used here is a virtual time, which is a discrete amount corresponding to the number of images.

(1) Feature point extraction / tracking process (S140)
As shown in FIG. 1, in the restoration system of this embodiment, first, a captured image sequence 110 is input to a computer or the like in which the restoration system of this embodiment is constructed, and feature points are extracted and tracked (between images). (Association of feature points) is performed (S140).
Shape restoration is performed based on the principle of stereo vision based on the features associated with each other. In the present embodiment, the KLT tracker, which is a conventional technique, is used to extract and associate the feature points. However, other methods may be used. The KLT tracker extracts features corresponding to the corner points and intersections of the image edge, and searches the next image for a region that closely matches the vicinity of the feature point in the current image within a predetermined range, thereby Tracking is performed (refer to Non-Patent Document 1 for details).
The KLT tracker can deal with feature points fairly well, but if there is a similar feature nearby, a correspondence error occurs. In addition, the position error of the corresponding feature point may become large in a portion where deformation due to perspective projection is strong or in the vicinity of occlusion. These problems are addressed using RANSAC of the prior art as will be described later.
For example, as shown in FIG. 4, the feature points obtained here are stored in a storage area such as an array using a number that uniquely identifies the feature point and a number that indicates which image is obtained as an index. To do. The stored feature points are used in each process of an initial estimation process (S150) and a sequential estimation process (S160 to S170) described later.

(2) Initial estimation process (S150)
Next, initial estimation is performed from the feature points obtained in S140 (S150).
The feature points of the object shape restored three-dimensionally by the restoration system of the present embodiment are merged into the storage area 130 to form a three-dimensional object shape. However, at the start of the reconstruction of the three-dimensional object shape (t = 1 to m), the restoration system of the present embodiment has the camera posture and 3 from the input image sequence without any three-dimensional object shape. The dimensional object shape must be estimated simultaneously. Therefore, using the factorization method is prior art, the initial attitude r ₁ from the feature point camera obtained in _S140, ..., r _m and the object of the initial shape (a set of three-dimensional feature point of an object shape) at the same time (See Non-Patent Document 3 for details). Since the factorization method has an error due to approximation to weak perspective projection, an accurate solution is obtained using nonlinear minimization. Instead of the factorization method, a technique based on epipolar geometry, which is also a conventional technique, may be used.
In the factorization method, a matrix W in which the coordinates of feature points are arranged is decomposed into a product W = MS of a matrix M representing a camera motion and a matrix S representing an object shape by using singular value decomposition. Although this decomposition is not unique, M is uniquely determined using a metric constraint on the orthogonality of the coordinate axes of the camera pose.

In the factorization method, perspective projection is linearly approximated to parallel projection or weak perspective projection. For this reason, there is an advantage that it can be solved very stably, but an error of linear approximation is included in the obtained restoration result. Therefore, the linear approximation error is removed by nonlinear minimization, and the camera motion and object shape under the perspective projection are obtained. This is done based on equation (1).

Here, (u _{t, i} , v _{t, i} ) are the coordinates of the i-th feature point in the image at time t, and R _t and T _t are the rotation matrix and translation amount of the camera coordinate system at time t, respectively. , P _i = (X _i , Y _i , Z _i ) ^T is the three-dimensional position of the i-th feature point. P _{t, i} is the three-dimensional position of the i-th feature point viewed from the camera coordinate system at time t. M is the number of images, and n is the number of feature points. This equation represents the error that backprojected onto an image based on the feature point P _i of the three-dimensional space to the camera posture. This nonlinear minimization problem is solved using an iterative method with a solution obtained by a factorization method as an initial value.
In this process, the scale of the target object cannot be determined, so it is necessary to give an actual scale in some form. For example, a method in which a person explicitly teaches a scale, a robot recognizes a known object and automatically sets the scale, or uses another sensor such as a distance sensor can be considered.

In order to deal with miscorresponding feature points, robust estimation is performed using RANSAC, which is a conventional technology (for details, see “M. Fischler and R. Bolles:“ Random Sample Consensus: a Paradigm for Model Fitting with Application to Image Analysis and Automated Cartography ”, Communications ACM, 24: 381-395, 1981.). First, randomly selects n ₁ one point from the feature point, after obtaining the camera position by factorization and nonlinear minimization described above, the three-dimensional _one feature point remaining n-n based on the camera posture Restore. If the feature points are miscorresponding, the three-dimensional reconstruction has a large error. Therefore, the restored three-dimensional point is back-projected on the image, and an error whose error from the corresponding image feature point is equal to or greater than a predetermined threshold is deleted as an incorrect correspondence. By repeating this, the camera posture that maximizes the number of correct feature points is adopted as the solution.
The initial posture of the camera obtained here is stored in the storage area 120, and the initial shape (three-dimensional feature point) of the object is stored in the storage area 130.

(3) Sequential estimation (S160 to S170)
After the initial estimation (S150), the camera motion and the object shape are restored by performing sequential estimation (S160 to S170) for alternately estimating the camera posture and the three-dimensional feature point.
(I) Camera posture estimation (S160)
The camera posture at time t is calculated so that the three-dimensional feature point (stored in the storage area 130) of the object shape obtained up to time t-1 matches the feature point of the image at time t. First, feature points are tracked between images at times t-1 and t. The feature points obtained here include feature points existing before time t−1, but there are also feature points newly extracted in place of feature points that have disappeared due to occlusion or the like. Since the feature points already registered in the storage area 130 are before the time t−1, only these feature points are used for estimating the camera posture. Assuming that the three-dimensional feature points registered in the storage area 130 up to time t−1 are P _i = (X _i , Y _i , Z _i ) ^T , camera postures R _t and T _t that minimize the following expression are obtained. .
This represents an error of back projecting the three-dimensional point of the object shape onto the image at time t. Since this is also a nonlinear minimization problem, it is solved using an iterative method with the camera posture at t−1 (stored in the storage area 120) as an initial value. Again, RANSAC is used to deal with mis-corresponding feature points.
The camera posture obtained here is stored in the storage area 120.

(Ii) Restoration of object shape (S170)
For each feature point, the three-dimensional position of the feature point is obtained from the camera posture before time t-1 and the camera posture at time t by the principle of triangulation.
Time t ₁ (stored in the storage area 120) and t ₂ of the camera orientation and determine the position of the three-dimensional point from the image. When the camera posture of _{t 2} as seen from the time _{t 1} camera R, is T, both time characteristic points _q t1, _i, the following relationship holds between the _{q t2, i.} s ₁ and s ₂ are parameters representing the depth.
In most cases, this equation cannot be solved exactly due to errors in feature point extraction and camera posture estimation, so a solution that minimizes the following square error is obtained.
The three-dimensional restored feature points are merged with the three-dimensional feature points already stored in the storage area 130. In this way, the three-dimensional object shape is restored. A method of fusing the three-dimensional restored feature points into a three-dimensional object shape will be described in detail later.

(2. Baseline length selection method)
As described in the background art, in the conventional camera motion and object shape restoration, if the translational component of the camera motion is very small with respect to the distance to the target object, the restoration result becomes unstable due to degeneration. There was a problem of becoming.
In the present embodiment, the above problem is solved by proposing a method for detecting a degeneration related to a translation component only from an input image sequence and obtaining an appropriate number of images that can secure a sufficient translation component in the initial estimation process. To solve.
Hereinafter, the baseline length selection method proposed in the restoration system of the present embodiment will be described in detail.
The baseline length selection method proposed in this embodiment is a process for determining the number of images (baseline length) used in the initial estimation process (S150 in FIG. 1). Specifically, for example, the baseline length selection proposed here is used. Each step of the process of the method can be realized by incorporating it in the initial estimation process (S150).

In order to address the above problem, in this embodiment, by examining the singular values of the basic equation matrix composed of a sequence of image feature points, the camera motion and the object shape can be restored without causing degeneration from the feature point sequence. Find out.
Degeneration checks are carried out by two of the first image I ₁ and the k-th image I _k of the image sequence. If the feature points on the image I _k corresponding to the feature points q _{1, i} on the image I ₁ are q _{k, i} and the basic matrix between the two images is F _{1, k} , the following basic equation holds. (Refer nonpatent literature 2).

Expanding the above equation, each image feature point q _{1, i} = (u _1i , v _1i , 1) ^T , q _{k, i} = (u _ki , v _ki , 1) ^T , ( Summarizing with respect to 1 ≦ i ≦ n), the following equation is obtained.
Here, f is a vector in which elements of the basic matrix F _{1, k} are arranged in a line. In this specification, this matrix A is called a basic equation matrix.

According to pages 279 to 280 of Non-Patent Document 2, when degeneration occurs due to minute translation, the rank of the matrix A becomes 6. Therefore, if the rank of the matrix A is examined, it can be determined whether or not degeneration has occurred. However, in practice, even if the translation component is not completely zero, degeneration in a broad sense as described above occurs. Further, the rank of the matrix A does not become exactly 6 due to the influence of noise.
Therefore, in the present invention, the matrix A is subjected to singular value decomposition, the singular values are arranged in descending order, and the seventh singular value (seventh singular value) and the eighth singular value (eighth singular value) are close to zero. Determining degeneration by how. For example, assuming that the singular values are s ₇ and s ₈ , D = √s ₇ s ₈ , which is the geometric mean, is used as a degeneration measure. This is called the degeneration standard. Depending on the size of the degeneration criterion D, not only the presence or absence of degeneration but also the degree of degeneration can be known. Thereby, it can be determined whether or not high-precision restoration can be performed without degeneration depending on whether or not D exceeds an appropriate threshold. That is, while increasing the number k of images, k is determined for which the degeneration reference value exceeds the threshold, and k at that time is adopted as an appropriate number of images that can ensure a sufficient translational component (baseline length).
If there is a mistracking of image feature points, a large noise will be included in the basic equation matrix, so that the degeneration reference value may become large even though degeneration actually occurs. Therefore, after removing the erroneous tracking of the image feature points by robust estimation, the degeneration reference value is calculated.
As described above, the present invention is characterized in that an appropriate number of images is obtained in consideration of the degree of degeneration based on the degeneration reference value, and that a highly reliable degeneration reference value is calculated after removing noise. .

FIG. 2 is a flowchart of processing when the baseline length selection method described above is processed in the restoration system of the present embodiment. As described above, this can be realized by incorporating the steps of the baseline length selection method described here into the initial estimation process (S150) shown in FIG.
First, in step S11, giving the initial value _{k 0} of the number of images k. Next, in step S12, a feature point series F that appears in common from image 1 to image k is extracted. The feature points are obtained and stored by the feature point extraction / tracking process (S140 in FIG. 1).
Next, in step S13, a degeneration reference value D is calculated for the entire feature point series F. In step S14, it is determined whether or not the degeneration reference value of F exceeds a predetermined threshold value. If it exceeds the threshold value, the process proceeds to step S15. If it does not exceed the threshold value, k is increased in step S19 and the process returns to step S12. . When degeneration occurs in the entire feature point series F, degeneration is expected to occur regardless of which subset of F is selected. In this case, it is useless to proceed to step S15 and subsequent steps, and in order to improve the efficiency of the process, the entire F is judged to be degenerated in steps S13 to S14.

Note that k in the case where the degeneration reference value of F exceeds the threshold value in step S14 can be adopted as an appropriate number of images without performing the processing after step 15. (In this case, the process ends if the answer is Yes in step S14).
Alternatively, the process may proceed from step 12 to step S15 without performing steps 13 and 14 (without performing degeneracy determination for the entire feature point series F before performing robust estimation).

In step S15, camera motion and object shape are determined using robust estimation such as RANSAC. This robust estimation is performed in order to realize high-precision restoration by removing the mistracking of the feature points and to improve the reliability of the degeneration reference value in step S16. Then, the mistracked feature points are removed from F to obtain a subset of the feature point series. In this case, a subset of the feature point sequence give good recovery of precision because it may be obtained plurality, it registers a family of subsets {g _i} as candidate solutions. Then, in order to one subset of a proper solution selected from the solution candidate, for each subset g _i, computes the restored reference value representing a goodness of restoration. When only one subset is obtained, the subset at the optimum number of images k may be adopted as a solution as it is.
Next, in step S16, to calculate the degeneration reference value for each _{g i.} Then, in step S17, it is judged if there is a g _i to degenerate reference value exceeds the threshold value, the process proceeds to step S18 if there is g _i that exceeds the threshold value, increasing k in step S19 if there is g _i exceeds the threshold value The process returns to step S12. In step S18, k is adopted as an appropriate number of images for which a sufficient translation component (baseline length) can be secured, and g _i that gives the best score calculated from the degeneration reference value and the restoration reference value as the feature point series is used. select.
Hereinafter, each step of the processing flow of FIG. 2 will be described in detail.

In step S11, the number k of images to be extracted from the image sequence is defined, and the first image and the kth image are extracted from the image sequence. The initial value k ₀ of k may give an appropriate value. However, k ₀ is 2 or more in a manner that is based on epipolar geometry, when reconstitution with factorization method is k ₀ must be 3 or more.
In step S12, a set of feature points that appear in common from image 1 to image k is obtained. The feature points of each image may be traced between images by extracting corner points of image edges and complex points of textures by the method described in the conventional technique and the feature point extraction / tracking process (S140) described above. . As shown in FIG. 4, the obtained feature points are stored in a storage area such as an array using a number that uniquely identifies the feature point and a number that indicates which image is obtained from the index.

In step S13, a degeneration reference value is calculated for the feature point series F according to the procedure shown in FIG. In FIG. 3, first, in step S21, the basic equation matrix A is generated from the feature point series according to the equation (7). At this time, in order to reduce the influence of noise and to standardize the range of singular values, the coordinate values of feature points are normalized in accordance with a method often used in restoration based on epipolar geometry. That is, the centroid of the entire image feature points q _{1, i} , q _{k, i} is calculated, and the range of the u coordinate value and the v coordinate value of q _{1, i} , q _{k, i} is −√2 with the centroid as the origin. Normalize each q _{1, i} , q _{k, i} so that it falls within the range of ~ √2 (for details, see R. Hartley: “In defense of the eight-point algorithm,” IEEE Transaction on Pattern Analysis and Machine Intelligence, Vol. 19, No. 6, pp. 580-593, 1997.).
Next, in step S22, the matrix A is subjected to singular value decomposition. Sort singular values in descending order. In step S23, a degeneration criterion is calculated from the seventh singular value and the eighth singular value. As the degeneration criterion, for example, the geometric average may be used as described above, or another criterion such as an arithmetic average or a square average may be used.
Step S14 is a step of determining the degeneration reference value by a threshold value, which is given empirically.

In step S15, robust estimation is performed using, for example, RANSAC in order to deal with miscorresponding feature points. First, n ₁ feature points are randomly selected from the feature point series F, and after obtaining the camera posture for restoring the n ₁ points using the above-described epipolar geometry-based method or factorization method, to restore the remaining n-n ₁ pieces of feature points 3D based on the camera posture. If the feature points are miscorresponding, the three-dimensional reconstruction has a large error. In view of this, an error in which the position error between the position where the restored three-dimensional point is back-projected on the original image and the corresponding image feature point is equal to or greater than a predetermined threshold is deleted as a false tracking. Let g _i be a subset of the feature point series obtained by removing the mistracked part from the feature point series F. Then, the number of feature points included in g _i is set as a restoration reference value that represents the goodness of restoration of g _i . This indicates that a camera posture having many feature points with a small backprojection error is better. The above process is repeated, and g _i whose restoration reference value exceeds the threshold is set as a solution candidate. Details of RANSAC are described in the literature of M. Fischler and R. Bolles introduced above. In addition to RANSAC, other robust estimation methods such as LMeds may be used.

In step S16, a degeneration reference value is calculated for the subset g _i of the feature point series by the procedure shown in FIG. 3 as in step S13.
Similar to step S14, step S17 is a step of determining the degeneration reference value by a threshold value, which is given empirically.
At step S18, the k at that time and sufficient translation component can be secured appropriate number of images, also of the candidate g _i, those score is best, chosen as a feature point sequence used to restore. As the score here, for example, the restoration reference value or the degeneration reference value may be used as it is. Alternatively, the score may be defined by an average of the restoration reference value and the degeneration reference value.
In step S19, an increment dk is added to the current number k of images, and the first image and the k + dk-th image are extracted from the image sequence. The value of dk is typically 1, but dk may be larger than 1 for an image with a small interval such as a moving image.

Thus the process, automatically determine the number of images k degeneration reference value exceeds the threshold value, moreover, it is possible to determine the subset g _i of the feature point sequence restoring the reference value is increased. Then, using the selected number of images and a subset of the feature point series, the camera motion and the object shape are restored by the method described in the initial estimation process (S150 in FIG. 1), and the initial camera posture is stored. The data is stored in the area 120, and the 3D point of the object shape (the initial shape of the 3D object) is stored in the storage area 130.
In general, the degeneration reference value increases as the translational component of the camera motion is longer and as the number of feature points is larger. However, as the translation component increases, the feature points that can be extracted in common between images decrease, and the degenerative reference value decreases from a certain point in time. Further, depending on how the camera moves and the shape of the target object, the degeneration reference value may decrease without exceeding the threshold value, no matter how many images are increased. In order to deal with this, an upper limit value for the number of images is set in advance, and if the degeneration reference value does not exceed the threshold even if the number of images exceeds the upper limit value, processing for changing to another image sequence and retrying is added. May be. Alternatively, when the degenerative reference value is reduced, it may be changed to another image sequence and retried.

(Reconstruction method of 3D points under micro translation)
In the baseline length selection method described above, it is possible to improve the camera posture and the reconstruction of the three-dimensional feature points in the initial estimation process by selecting the number of images that satisfy the criteria best for the set of feature points. .
However, another problem when the translational component is close to 0 is that the reconstruction accuracy of the three-dimensional point is extremely deteriorated. Since the three-dimensional restoration of feature points is performed based on the principle of triangulation, if the baseline length is short, the restoration accuracy is deteriorated. If the accuracy of the three-dimensional position of the feature point is poor, an error is propagated to the next camera posture estimation, and the error is enlarged. This problem appears in both initial and sequential estimation. In order to deal with this, here, a method is proposed in which the three-dimensional feature points are not restored until the baseline length is sufficiently long so that the restoration accuracy is within an allowable range in the successive estimation process. Note that the three-dimensional feature points restored in the initial estimation process can also be the target of the process described here.
Specifically, it can be realized by incorporating the process described here into the object shape restoration process (S170 in FIG. 1) in the sequential estimation process.

Here, a three-dimensional restoration covariance matrix V [P] = E [ΔPΔP ^T ] of feature points is used for evaluation of restoration accuracy. This is calculated as follows:
However, _{s 1} is a depth parameter of the above formula _{(3), V [s 1} ] = E [(Δs 1) 2], a _{V [q, s 1] =} E [ΔqΔs 1]. For these derivations, see “K. Kanatani:“ Statistical Optimization for Geometric Computation: Theory and Practice, ”Elsevier, Amsterdam, 1996.”.

In general, since the same feature point can be tracked between a plurality of images, a plurality of restoration results can be obtained. They are fused using a covariance matrix. If the three-dimensional position of the feature point registered as a three-dimensional object shape in the storage area 130 by time t−1 is P ^t−1 and its covariance matrix is V [P ^t−1 ], it is obtained at time t. The three-dimensional position stored in the storage area 130 is updated by fusing the restored result P _t and the covariance matrix V [P _t ].
In the restoration of the three-dimensional point, the accuracy of the restoration result varies depending on which image pair is selected and the restoration calculation is performed. If the error distribution of the feature point position q on the image is the same for all images, the restoration accuracy depends on the camera posture. Intuitively, the longer the baseline length, the better the accuracy. Therefore, an image pair having as small an error as possible is selected using Expression (8). The procedure is shown below.

(1) A feature point corresponding to an image feature point q _{t, i} at time t is obtained from a past image, and the set is defined as Q _i .
(2) _{Q i} characteristic points _{q tk} at each time _{t k} in the _{for i, q t,} in combination with _i performs three-dimensional reconstruction by sequential methods of estimating described above, expression from the result ( 8) is used to calculate V [P _{t, i} ].
(3) If there is q _{tk, i} having a minimum determinant value of the covariance matrix V [P _{t, i} ] and less than a predetermined threshold, three-dimensional reconstruction is performed using that and q _{t, i.} The result obtained is adopted as a solution and fused to the three-dimensional object shape stored in the storage area 130 based on the equation (9).
In this way, the restoration accuracy can be improved in the object shape restoration process (S170 in FIG. 1) in the successive estimation process of the present embodiment.

(3. Experiment using restoration system of this embodiment)
Finally, the results of experiments using the camera motion and object shape restoration system of the present embodiment described above will be shown.
As a condition of the experiment, a human had a digital camera and took a still image in continuous shooting mode while walking. The original still image is 640 × 480, but this was used after being reduced to 320 × 240. An offline restoration experiment was performed from the captured images.

(1) Selection of optimum baseline length (number of images) FIG. 5 shows an experimental example in which the optimum number of images (baseline length) for restoration is automatically selected by the baseline length selection method of this embodiment. FIG. 5A shows an image example in the used image sequence and feature points extracted in each image. The number of feature points tracked is 50. In this example, the camera was moved slowly to reduce the translational component. (B) is a readout of a restoration example in which initial estimation was performed using the first three images. The arrow 510 indicates the direction of camera line of sight, and the point indicated by 520 is the restoration result. In fact, it can be seen that the part that should be perpendicular is gradual and has a shape close to a plane. The degeneration reference value D at this time was 0.02. (C) is a readout of the restoration result obtained by automatically obtaining an appropriate baseline length by the method of the present embodiment. The arrow 530 is the direction of camera line of sight, and the point indicated by 540 is the restoration result. It can be seen that the right-angled part has been successfully restored. The number of required images was 24, and the degeneration reference value D was 0.22.
FIG. 6 shows the state of error dispersion in the above-described three-dimensional restoration. The target object is the same as in FIG. In FIG. 6, the standard deviation in the line-of-sight direction is represented by the length of the line segment. The restoration point is the midpoint of each line segment. (A) is a case where the baseline length is 10 images. The arrow 610 is the direction of the camera line of sight, and the line segment indicated by 620 is the restoration result (line segment indicating the standard deviation). (B) is a case where the baseline length is 20 images. An arrow 630 is the direction of the camera line of sight, and a line segment indicated by 640 is a restoration result (a line segment indicating a standard deviation). It can be seen that the longer the baseline length, the smaller the error. It can also be seen that the point farther from the camera has a larger error. Since the actual scale is indefinite, the standard deviation values are 0.055 to 0.07 in (a) and 0.017 to 0.02 in (b) when expressed as a ratio to the distance from the camera.

(2) Example of Restoring Camera Posture and Object Shape (i) Stairs FIG. 7 shows an example of restoring the staircase at the entrance of the university building from 90 images. The baseline length in the initial estimation was 3 images. (A)-(c) is an example of the input image, (d) is the restoration result seen from the side. Reference numeral 710 denotes a restored camera posture, and reference numeral 720 denotes a three-dimensional point of the restored object. Further, (e) is a restoration result viewed from the front, where 730 is a restored camera posture and 740 is a three-dimensional point of the restored object. It can be seen that the camera movement in the vertical direction and the stairs are restored. This suggests that the robot can be used for moving on rough terrain.
(Ii) Building FIG. 8 shows an example in which a university building is restored from 120 images. The baseline length in the initial estimation was 9 images. (A)-(c) is an example of the input image, (d) is the restoration result seen from the side. Reference numeral 810 denotes a restored camera posture, and reference numeral 820 denotes a three-dimensional point of the restored object. Further, (e) is a restoration result viewed from the front, where 830 is the restored camera posture and 840 is the three-dimensional point of the restored object. As described above, in this method in which the baseline length can be adjusted, a distant object can be restored. The state of going down the stairs during shooting (850) has also been restored.
(Iii) Indoor FIG. 9 shows an example of restoration from an image obtained by photographing a desk in the laboratory. The baseline length in the initial estimation was 4 images. (A) is an example of input images (eight images), and (b) is a restoration result viewed from the side. 910 is the restored camera posture, and 920 is the three-dimensional point of the restored object. Further, (c) is a restoration result viewed from the front, 930 is the restored camera posture, and 940 is the restored three-dimensional point of the object. The camera movement is irregular, but it is well restored.

It is a figure which shows the whole processing flow of the decompression | restoration system of this embodiment. It is a flowchart which shows an example of the procedure of the camera motion of this embodiment, and an object shape restoration process. It is a flowchart which shows an example of the calculation procedure of a degeneration reference | standard. It is a figure which shows an example of a structure of an image feature point. It is a figure which shows the experimental result using the decompression | restoration system of this embodiment. It is a figure which shows the experimental result using the decompression | restoration system of this embodiment. It is a figure which shows the experimental result using the decompression | restoration system of this embodiment. It is a figure which shows the experimental result using the decompression | restoration system of this embodiment. It is a figure which shows the experimental result using the decompression | restoration system of this embodiment.

Claims (8)

A camera motion / object shape restoration system that restores the camera motion and the three-dimensional shape of an object from an image sequence taken while moving the camera,
A feature point extracting / tracking unit that extracts feature points from the input image, associates them between images, and stores the associated feature points in a feature point storage unit;
An initial value k is given, feature points from the first input image to the k-th image are read from the feature point storage means, a basic equation matrix is generated from the feature points, and a singular value of the basic equation matrix is obtained. When the degeneration reference value is calculated from the singular value and the degeneration reference value does not exceed the predetermined threshold, the calculation of the degeneration reference value is repeated while increasing k. When the degeneration reference value exceeds the threshold, Initial estimation means for obtaining the initial posture of the camera and the three-dimensional position of the feature point of the object from the feature points from the first image to the k-th image and storing them in the camera motion storage means and the three-dimensional shape storage means, respectively;
Camera motion / object shape restoration means comprising camera motion / object shape reading means for reading out the camera posture stored in the camera motion storage means and the object shape stored in the three-dimensional shape storage means system. The camera motion / object shape restoration system according to claim 1,
Further, after the processing of the initial estimation means, the camera orientation estimation for obtaining the camera orientation of the current image in order from the (k + 1) th image to the last image and storing it in the camera motion storage means, and the three-dimensional position of the feature point What is claimed is: 1. A camera motion / object shape restoration system comprising: successive estimation means for alternately and repeatedly obtaining and reconstructing an object shape to be fused with the three-dimensional shape storage means. In the camera movement / object shape restoration system according to claim 1 or 2,
The initial estimation means obtains a subset of feature points used for camera motion and object shape restoration by robust estimation from the feature points from the first image to the k-th image, and a basic equation matrix from the subset of the feature points Generating a singular value of the basic equation matrix and calculating a degeneration reference value from the singular value. If the degeneration reference value does not exceed a predetermined threshold, the calculation of the degeneration reference value is repeated while increasing k, When the degeneration reference value exceeds the threshold value, the initial posture of the camera and the three-dimensional position of the object feature point are obtained from the subset of the feature points and stored in the camera motion storage unit and the three-dimensional shape storage unit, respectively. Camera motion / object shape restoration system. The camera motion / object shape restoration system according to claim 3,
When a plurality of subsets of the feature points are obtained, the initial estimation means obtains a restoration reference value for each subset, and a score calculated from the restoration reference value and the degeneration reference value is the best. A camera motion / object shape restoration system, wherein an initial posture of a camera and a three-dimensional position of a feature point of an object are obtained from the subset and stored in camera motion storage means and three-dimensional shape storage means, respectively. In the camera movement / object shape restoration system according to claim 3 or 4,
The initial estimation means obtains a degeneration reference value from the feature points from the first image to the k-th image before obtaining the subset of the feature points, and if the degeneration reference value does not exceed a predetermined threshold value A camera motion / object shape restoration system characterized by repeatedly calculating a degeneration reference value while increasing k, and obtaining a subset of the feature points when the degeneration reference value exceeds the threshold. In the camera movement / object shape restoration system according to any one of claims 2 to 5,
The object shape restoration in the successive estimation means is performed by performing three-dimensional restoration on the feature point of the current image and the corresponding feature point of the image at each past time point stored in the feature point storage means. Camera motion characterized by obtaining a covariance matrix and fusing the three-dimensional shape storage means with a three-dimensional restoration result having a minimum covariance matrix value less than a predetermined threshold as a three-dimensional position of the feature point -Object shape restoration system. A camera motion / object shape restoration program for causing a computer system to configure the functions of the camera motion / object shape restoration system according to claim 1. A recording medium on which a camera motion / object shape restoration program for causing a computer system to configure the function of the camera motion / object shape restoration system according to claim 1 is recorded.