KR100922273B1 - Mechanism for reconstructing a 3D model using key-frames selected from image sequences - Google Patents

Abstract

The present invention provides a method for selecting an important image from an image sequence using a camera reprojection error and reconstructing a 3D model using only the selected important image. According to an embodiment of the present invention, first, by selecting an image having a minimum reprojection error in a predetermined number of image sequences as an important image, the error of the reconstructed three-dimensional model is minimized. The projection matrix of each of the selected important images is estimated through the automatic camera calibration process. We then obtain the base matrix from the estimated projection matrix by using algebraic derivation, remove the incorrect correspondence points by applying the epipolar constraint using the base matrix, and then reconstruct the accurate 3D model using only the remaining correspondence points. According to the exemplary embodiment of the present invention, an algorithm for selecting an important image in a short time compared to other existing algorithms can be performed, and the error of the reconstructed 3D data is the smallest. The basic matrix derived from the projection matrix of the important image takes much shorter time than other algorithms, but the mean error is similar to the conventional method.

Description

Mechanism for reconstructing a 3D model using key-frames selected from image sequences}

The present invention relates to reconstruction of an omnidirectional three-dimensional model, and more particularly, to a procedure for selecting an optimal important image from an image sequence and efficiently reconstructing the three-dimensional model using the selected important image.

Recently, applications using realistic 3D models such as graphic rendering, digital libraries, and virtual environments have attracted attention. As a result, interest in a mechanism for reconstructing a geometric three-dimensional model using image sequences acquired from one or several cameras is also increasing. In particular, as 3D graphics technology has increased significantly on a desktop computer or a web browser, many studies have been conducted to generate realistic 3D models without using expensive CAD (Aided Design) devices. Coming.

As one of techniques for generating a three-dimensional model, a synthetic image technique for reconstructing a three-dimensional model using acquired photographs has also been steadily developed. Three-dimensional modeling techniques (programs) developed to date include Maya, Softimage, and 3D Studio MAX. These three-dimensional modeling techniques make it possible to reconstruct a certain level of a three-dimensional model without the aid of expensive programs, but when the complexity of the three-dimensional model increases, the reality or quality of the reconstructed three-dimensional model is somewhat deteriorated. have.

A recent trend in the technique of reconstructing a three-dimensional model is to reconstruct the three-dimensional model by obtaining accurate information about the object from one or more pictures (image sequences) of the real object. This new technology, called Image Base Modeling (IBM), is a modeling technique that typically uses real photographic images. The biggest advantage of this technique is that we can use the texture information about the object from the real world to create a more realistic three-dimensional model. The 3D model reconstruction technology for such objects or scenes can be used in augmented reality systems or virtual reality systems, such as restoration of cultural assets, internet shopping malls, and special effects of movies.

One of the essential steps for reconstructing a three-dimensional model using images (photographs) from one or more cameras is camera calibration. One of the camera calibration procedures is a self-calibration method, which uses a self-image of a real model acquired by a camera, without any prior knowledge about the acquired scene or constraints on the real model. This method is a vision-based method that uses only the acquired images without using any equipment other than the camera. After camera calibration, the projection matrix of each image using only the image sequence (for example, single-axis rotation images). Reconstruct the 3D model by estimating the (Projection Matrix) and the corresponding point. On the other hand, there is also a method using a camera calibration equipment, three-dimensional position sensor and pattern information, etc. This method requires expensive equipment for camera calibration, there is a disadvantage that requires a limited acquisition environment.

In image-based modeling, it is not efficient to perform camera calibration and three-dimensional model reconstruction using all acquired image sequences. This is because, in the case of using all of the image sequences, the processing time is long because the calculation amount is large, and the noise included in each image may interfere with the accurate reconstruction of the three-dimensional model. Therefore, as one method for solving this problem, a method of selecting a key image (Key-Frame) among image sequences and reconstructing a 3D model using only the selected important image has been proposed. According to this, the process of selecting an important image is an essential process for reconstructing a 3D model from a plurality of comparative image sequences or for implementing augmented reality.

For the selection of important images, the following algorithm has been proposed. For example, S. Gibson et al. Describe the fundamental matrix and projection plane homography ("Accurate Camera Calibration for Off-line, Video-Based Augmented Reality" (IEEE ACM ISMAR, pp. 37-46, 2002). We proposed an algorithm that selects medium based on error value by calculating 2D Projective Transform Matrix. However, the method proposed by Gibson et al. Has a disadvantage in that a large amount of computation is required to extract the exact correspondence point. In addition, since the basic matrix is sensitive to noise, it takes a long time to perform the calculation repeatedly to estimate the correct value by this method. In addition, M. Pollefeys et al. [3D Models from Extended Uncalibrated Video Sequences: Addressing Key-frame Selection and Projective Drift] (3DDIM, June 2005) adds and selects another second medium image to important images for camera automatic correction. According to this, geometrical information of three important images is used for automatic correction. However, this method not only increases the number of selected important images to three, but also increases the amount of calculation accordingly. In addition, D. Neister proposed an algorithm to select an efficient image using the sharpness of the image in "Frame Decimation for Structure and Motion" (In Proc, SMILE, Dublin, Ireland, July 2000). However, due to the intrinsic nature of sharpness, this algorithm has the disadvantage that many images are unnecessarily selected as important images when the image quality is good, resulting in unnecessary computation.

On the other hand, in the reconstruction algorithm of the three-dimensional model, the basic matrix estimation is the only method of calculating the epipolar geometry information. The basic matrix represents the algebraic relationship between successive images. To find the basic matrix, we need to know the points that correspond to each other in at least two images. Many methods for obtaining a basic matrix using corresponding points between the images have been studied in the past. There are two main ways to get the basic matrix. The first method is linear methods (7-points, 8-points, linear rank-2 constraints, etc.). These methods can achieve good performance if the corresponding point problems can be solved linearly. However, since the corresponding point problems are generally not solved linearly, the first method described above is not suitable for general applications.

The second method is nonlinear methods such as M-Estimators (M-Estimators), least median of square (LMedS), random sample consensus (RANSAC), etc., and is more robust than the first method. Among these, the M-evaluation method can reduce the influence of outliers due to large noise by applying a weight function to the error of the matching point. The RANSAC method proposed by Torrr and Murray is a simple and successful robust estimation method. According to the RANSAC method, in order to estimate the basic matrix, a minimum set of matching points is randomly extracted from all matching points to obtain an initial value, and the error is calculated for the remaining matching points. If the calculated error is higher than the threshold, the corresponding point is determined as an outlier. However, if it is lower than the threshold, the corresponding point is determined as an inlier. The above process is repeated to obtain the inlier set, and then the case is used to select the case with the largest number of coincidence points and recalculate the base matrix using the set. The LMedS, MLESAC, and MAPSAC methods are robust to noise and effectively eliminate outliers based on the RANSAC method. However, the most important constraint of these methods is that due to randomly chosen matches, different inlier sets are selected each time the result is performed, and thus the result obtained is also affected by the selected inlier set.

The problem to be solved by the present invention is to select the important image to minimize the error between the real object and the re-projection image possible when reconstructing the three-dimensional model, and also to use the selected important image efficiently and accurately three-dimensional model It is to provide a reconstruction procedure of a three-dimensional model that can be reconstructed.

The reconstruction method of the three-dimensional model newly proposed in the present invention is largely divided into three processes.

The first process is the selection process of important images. In the process of selecting an important image according to an embodiment of the present invention, an image having a small reprojection error is selected as an important image from an image sequence by using re-projection errors of projective reconstruction. That is, an important image is selected from an image having a minimum reprojection error among a predetermined number of images continuously input. Selecting an important image in this way is not only a small amount of calculation, but also a fast processing speed, it is possible to restore the optimal three-dimensional model by reducing the reconstruction error as much as possible when reconstructing the three-dimensional model.

In the second process, the camera projection matrix is obtained using only the selected important images, and then the basic matrix is used. More specifically, first, pre-auto calibration is performed to calculate the projective reconstruction and metric homography on the selected important images. The camera projection matrix is estimated through the full auto calibration between the selected important images, and then the basic matrix is obtained through the algebraic derivation using the estimated projection matrix.

In the final process, the inaccurate corresponding points are removed by using the basic matrix, and then the 3D model is reconstructed by extracting the dense corresponding points having the minimum error from the edge of the image from which the inaccurate corresponding points are finally removed. Eliminating inaccurate correspondence can improve the accuracy and precision of the reconstructed three-dimensional model, as well as reduce the computational complexity for the reconstruction model.

A method for reconstructing a 3D model from an image sequence according to an embodiment of the present invention for achieving the above problem is that a portion of the image sequence is k (k is an integer greater than or equal to 1) and the k-th important image A plurality of comparison images subsequent to the image, and using the reprojection error of each of the comparison images together with the ratio of the corresponding points of each of the comparison images to the feature point for the k-th important image among the comparison images. selecting a (k + 1) th important image, performing a pre-camera calibration procedure using the important images selected from all of the image sequences, and obtaining a projection matrix of each of the important images; Obtaining a base matrix using the projection matrix, and using an epipolar constraint using the base matrix Removing the noise from each of the edges is important group image, and a step of reconstructing a 3D model using the noise is removed edge.

According to an aspect of the embodiment, the important image k is 1 may be the first image of the video sequence. And a part of the image sequence includes images of the image sequence from the kth important image to a comparison image in which an image having a ratio smaller than a first threshold first occurs, and further includes the (k + 1) th important image. May be a comparison image of which a reprojection error is minimal among a portion of the image sequence. In this case, the (k + 1) th important image may be selected from comparison images in which the ratio is less than or equal to a second threshold, and the second threshold may be smaller than the first threshold. The reprojection error (R.E.) can be obtained using the following equation (E-1).

(E-1)

(E-1)

Here, P ⁱ denotes a projection projection matrix, X _j denotes an arbitrary point in three-dimensional space, and x ⁱ _j denotes a point in the projection coordinate system corresponding to X _j .

According to another aspect of the embodiment, it is possible to calculate the base matrix through algebraic derivation from the projection matrix.

According to an embodiment of the present invention, since the reprojection error is used in the process of selecting an important image, the calculation speed is small and the processing speed can be improved, and since the image having the smallest reprojection error is selected as the important image, accurate 3D Reconstruction of the model is possible. In addition, since the basic matrix is estimated from the camera projection matrix through algebraic derivation, the processing speed can be further improved while maintaining the average error level compared with the conventional method.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart schematically illustrating a procedure of selecting an important image from an image sequence, which is an input image, and reconstructing a 3D model using the selected important image, according to an embodiment of the present invention. to be. Referring to FIG. 1, a reconstruction procedure of a 3D model according to an embodiment of the present invention may include a key frame selection process (S100), a full auto calibration process (S200), and a 3D model. 3D Dense Reconstruction (S300). Details of each process illustrated in FIG. 1 briefly illustrate embodiments of the present invention, and a detailed description of each process will be described later with reference to other accompanying drawings.

According to an embodiment of the present invention, there is no limitation in the method of obtaining the image sequence. For example, the image sequence may be a continuous rotational motion image obtained using a self-produced Photonovo image acquisition system as a single axis rotation image. The Photo Novo Image Acquisition System includes a digital camera, a rotating platen driven by a motor, and a personal computer (PC). The images can be acquired by rotating them by 1 degree from different angles. However, embodiments of the present invention are not limited to images acquired using such a photonovo image acquisition system, and the image acquisition interval may be different even if the image is acquired by another rotational motion image acquisition system or the same photoknob image acquisition system. The same may be applied to the acquired image. Alternatively, the image sequence may be an image sequence obtained by using a plurality of cameras disposed at predetermined intervals in the same direction instead of a single axis rotation image.

2 is a flowchart illustrating an example of an important image selection algorithm according to an embodiment of the present invention.

Referring to FIG. 2, first, a correspondence point between two images included in an image sequence obtained by using a camera, one of which is necessarily an important image and the other image is referred to as a 'comparative image' Extract (S101). More specifically, the feature point is extracted from the important image, and the corresponding point corresponding to the feature point is extracted from the comparative image. According to an embodiment of the present invention, the first image (important image) and the second image (comparative image) to be input are first performed after S101 according to the important image selection algorithm shown in FIG. 2. If it is determined in step S102 that the number ratio R _c of the corresponding points to the feature points is greater than or equal to a predetermined threshold value, the first image (important image) and the third image (comparative image) are sequentially returned from step S101 in FIG. 2. Proceed with the procedure. According to the embodiment of the present invention, there is no limitation in the specific method of extracting the corresponding point in step S101.

According to an embodiment of the present invention, one image must be an important image in the procedure of step S101 shown in FIG. 2. Therefore, according to the embodiment of the present invention, the first image is always selected as an important image. Then, by performing the procedure of step S101 or less, an image to be an important image next to the first image, that is, a second important image, is selected from the input image sequence (multiple comparison images). Subsequently, a process of selecting a third important image is performed using the second important image according to the same procedure as in the following embodiment.

Continuing to refer to FIG. 2, this value determines whether less than a predetermined threshold value (Threshold) by calculating the ratio of the number (R _c) of the extracted feature points against the corresponding points (S102). Here, the number ratio R _c of the corresponding points to the feature points refers to the ratio of the number of corresponding points of the second image (comparative image) or another comparison image to the number of feature points of the first image (important image). When obtaining the number ratio R _c of the corresponding points, the ratio of the number of points of correspondence between the feature point of the immediately preceding important image and the subsequent image (comparative image) of the important image is always calculated. The threshold may be determined as a predetermined value smaller than 1, and may be selected as an appropriate value in consideration of the efficiency of 3D model reconstruction, the number of important images to be selected, and / or the processing speed of the processor.

In general, when the ratio of the number of corresponding points to the feature point between the two images (the important image and the comparative image) is close to 1, it indicates that the two images are located in close proximity. Conversely, if the ratio of the corresponding points to the feature points is too low, the two images are generally unrelated images. Therefore, in the exemplary embodiment of the present invention, when the ratio of the number of corresponding points to the immediately preceding important image becomes less than or equal to a predetermined threshold value, step S103 or less proceeds, so that the image that has too much relevance to the previous important image is the next important image. To prevent selection. According to an embodiment of the present invention, as described below, only an image having a ratio of the contrast point to the feature point or less is constant (eg, 0.9) or less is selected as an important image, so that the interval between the important images selected in the image sequence is increased. It can be kept more than a certain distance. In this way, in a single image sequence, only a small number of images are selected as important images in a possible range, thereby minimizing calculation in subsequent procedures (camera calibration procedures, etc.).

Subsequently, when the number ratio R _c of the corresponding points to the feature points becomes greater than or equal to the predetermined threshold as a result of the determination in step S102, the step is performed using the image of the important image (first frame) and the next frame (eg, the third frame). S101 Repeat the following procedure. On the other hand, when the determination result in step S102 indicates that the number ratio R _c of the corresponding points is smaller than the predetermined threshold, a procedure for calculating the reprojection error is performed in step S103 or below.

When the number ratio R _c of the feature points to the corresponding points is smaller than a predetermined threshold, three-dimensional reconstruction is performed on the projected space by using an image sequence up to the corresponding comparison image (S103). A predetermined projection matrix may be used to reconstruct the 3D model from the obtained 2D planar image (comparative image). Then, re-projective errors (RE) for each image are obtained using the reconstructed three-dimensional models (S104). The reprojection error may be a coordinate difference between the reconstructed three-dimensional model (corresponding points) and the reconstructed three-dimensional model from the important image. Steps S103 and S104 may be sequentially performed on the images before the comparison image that satisfy the conditions of step S102.

Subsequently, among the reprojection errors of each of the plurality of comparison images satisfying step S102, an image having the minimum value is found (S105), and the comparison image is selected as the second important image (S106). In this case, only the comparison image of which the ratio of the contrast point to the above-described feature point is less than or equal to a predetermined value (0.9) among the comparison images having a small reprojection error may be selected as the important image.

Subsequently, it is determined whether the comparison image satisfying step S102 is the last frame in the input image sequence (S107). If it is determined that the last frame has not been performed, the procedure of step S101 or less is repeated again using the next frame as in step S108. In this case, the procedure of step S101 or below which is newly started performs the procedure of S102 to S104 based on the important image selected in step S106. As a result of the determination in step S107, when the important image selection process is performed until the last frame, the selection algorithm of the important image according to the present embodiment ends.

As a result of the selection algorithm of the important image, a predetermined number (eg, M) images are selected as the important image from the input image sequence. Hereinafter, such an important video selection algorithm will be described in detail.

Selection algorithm of important video

Interested in the construction of a fully automated system using feature extraction and camera calibration techniques for three-dimensional reconstruction from non-corrected images. For such an automated system, one of the additional technologies is a procedure for selecting an important image. The critical image should have sufficient baseline to calculate disparity, and should enable camera motion analysis and three-dimensional reconstruction that require a large amount of computation from a large number of images to a small image. . In addition, it is necessary to select as few images as possible, including overlapping points of overlap between images, and to minimize errors caused when the overall camera motion and camera parameters are estimated.

Project creative space (projective space) when turned is a point X _i = (X _i, Y _i, Z _i) ^T and thus a point x _i of the two-dimensional camera image corresponding to the plane coordinate system of the 3-dimensional space given on, the two The relationship with the points is satisfied by x _i = P ^j X _i by the 3x4 projection projection matrix. Projective Projection Matrix and Projective Restoration are obtained from Equation 1 using a factorization method.

In the case of three-dimensional reconstruction using the camera projection matrix, the accuracy of the reconstructed 3D data depends on the accuracy of the obtained projection matrix. In an embodiment of the present invention, as a method for selecting only an image having an accurate camera projection matrix, after calculating a camera projection matrix of each image in a projection space, an image having the smallest reprojection error is selected as an important image. The reprojection error can be obtained using Equation 2.

According to one embodiment of the present invention, first, the first image of the image sequence is selected as the important image. The feature point of the first important image is extracted to find a corresponding point with the next image, and the ratio R _c of the feature point to the corresponding point is calculated. When this value is larger than the threshold, the corresponding point is found again in the next image. On the other hand, when R _c is less than the threshold, it is projectively restored from the first important image to the current image. The reprojection error is calculated from each projection matrix, and the image containing the smallest error and the ratio of the corresponding point to the feature point of 90% or less is selected as the important image. Then, the above process is repeatedly performed on the continuous input image using the selected important image to select another important image.

3 is a flowchart illustrating a fully automatic calibration process according to an embodiment of the present invention. Referring to FIG. 3, a fully automatic calibration process according to an embodiment of the present invention may be performed by first performing a camera calibration process using n (<M) important images from among M important images selected according to the embodiment described with reference to FIG. 2. Using the pre-auto-calibration process and the information (such as projected projection matrix and metric homography), the camera calibration process is performed on the remaining (Mn) important images. It includes a full-auto-calibration process to be performed. As a result of the fully automatic calibration, an accurate projection matrix for each camera (important image) can be obtained.

Referring to FIG. 3, first, corresponding point extraction is performed on n important images (from the first important image to the nth important image) (S201). In operation S202, it is determined whether the ratio R _c between the feature point and the extracted corresponding point is smaller than a predetermined threshold. As a result of the determination, when the ratio R _c between the feature point and the extracted corresponding point becomes equal to or greater than a predetermined threshold value, the next frame is selected (S203), and steps S201 and S202 are repeated for the next frame.

On the other hand, if the ratio R _c between the feature point and the extracted corresponding point is smaller than the predetermined threshold, as a result of the determination in step S202, the pre-projective reconstruction step S203 is performed, and then the preliminary project is performed. The pre-projective bundle adjustment step (S204) and the pre-projective structure step (S205) are performed. There is no particular limitation on the method of performing the procedure of steps S203 to S205, and may be performed according to a conventional procedure in the art. For example, "Multiple View Geometry" by R. Hartley and A. Zisserman (Cambridge Univ. Press, 2000), "Euclidean reconstruction from image sequences with varying and unknown focal length and pricipal by A. Heyden and K. Astrom. point "( In Proc. CVPR, pp. 438-443, 1997), and / or" Bundle Adjustment-a modern synthesis, In Vision algorithms by B. Triggs, P. McLauchlan, R. Hartley and A. Fitzgibbon: Theory and practice "(LNCS 1883, pp. 298-372, 2000) can be used, but is not limited thereto. As a result of performing steps S203 to S205, a projection projection matrix can be obtained.

Subsequently, after performing a pre-metric reconstruction step (S206), a pre-metric bundle adjustment step (S208) and a pre-metric structure step (S208) are performed. Perform The algorithms disclosed in the above cited documents in steps S206 to S208 can also be used. As a result of performing steps S206 to S208, metric homography can be obtained.

3, the full automatic calibration procedure is performed using the remaining important images (ie, (n + 1) th important images to the M th important images) that are not used in the preliminary automatic calibration procedure. The projected projection matrix and the metric homography obtained in the preliminary automatic calibration procedure may be used for this full automatic calibration procedure. Step S209 shows an example of such a full automatic calibration process, and the present embodiment is not limited thereto.

According to the embodiment of the present invention, as a result of performing the preliminary automatic calibration procedures S201 to S208 and the pre-automatic calibration procedure S209, the camera calibration procedure is completed. This will be described below in more detail.

Auto calibration before camera

<Preliminary or Preliminary Auto Calibration>

Here, a method of automatically calibrating a camera for an image selected by an important image selection algorithm will be described. Camera auto-calibration is an essential process for three-dimensional reconstruction using multiple images that have been compared. As described above, FIG. 3 is a flowchart illustrating a camera automatic calibration process according to an embodiment of the present invention.

Projective reconstruction is converted to metric reconstruction by performing camera autocalibration. Camera auto-calibration uses the factorization method from the corresponding point to perform projective reconstruction, and obtains metric homography from absolute quadric estimation. Deveranay, O. Faugeras, "From Projective to Euclidean reconstruction", IEEE conf.on CVPR, pp. 264-269, 1996). In this process, we obtain the projection projection matrix and metric homography for n images. This part is defined as preliminary automatic calibration.

A point X _j in 3D space is projected by x _j = P _i ^m X _j in each image from the I-th camera projection matrix P _i ^m . The subscript m denotes the projection matrix in the metric space. The metric projection matrix can be obtained as shown in Equation 3 by the projection projection matrix obtained from the projection reconstruction and the 4 × 4 metric homography H.

<Full Auto Calibration>

Projection projection matrices and metric homography for n important images were obtained from the prior automatic correction. Next, a procedure for performing calibration on the entire important image is performed, which is called full automatic calibration. Find the correspondence point between the nth important image and the (n + 1) th important image, and use the projective structure and metric homography obtained in the previous automatic calibration process. Calculate the projection matrix. Two bundle adjustments are performed for optimized projective restoration and metric restoration. Through this process, the metric projection matrix of the important image continues. As the number of corresponding points decreases as the frame progresses, when the number falls below a predetermined reference value, the above process is repeated after projective restoration. When the last important image is performed, the automatic calibration procedure is terminated.

Estimation of Default Matrix

Next, a method of extracting feature points and estimating a basic matrix will be described. In general, the epipolar constraint is applied to calculate or modify the position of the corresponding point. In the embodiment of the present invention, algebraic derivation is used as a method for estimating the base matrix. The two equations shown in Equation 4 are a method of deriving a basic matrix from two corrected images (more precisely, projection matrices of these images).

Here, K, K 'means an internal parameter of the camera, R, t means an external parameter of the camera, P ⁺ means a pseudo-inverse of P. According to the embodiment of the present invention, since the projection matrix obtained in the previous step is used, the time for calculating the basic matrix is reduced. Once we have the correct camera projection matrix, we can estimate the exact basic matrix very quickly through algebraic derivation.

In an embodiment of the present invention, an edge that expresses a feature of a scene or an object is selected as the feature point. First, the edges of important images are extracted and matched using optical flow. The epipolar constraint is applied using the basic matrix obtained above. Corresponding points thus obtained are obtained by three-dimensional coordinates using a point triangulation method.

Experiment result

As described above, in the exemplary embodiment of the present invention, an important image is selected from a plurality of compared images, and a three-dimensional model is reconstructed through a basic matrix estimation through automatic calibration and algebraic deduction for the selected important images. .

1. Selection algorithm of important video

4 shows some of the images used to experiment the embodiment of the present invention. Table 1 compares the number and time required of the selected images according to the important image selection algorithm according to an embodiment of the present invention with those of other algorithms. As can be seen from Table 1, the Gibson method selects a relatively small number of images compared to other methods, but it is very sensitive to noise because it uses the base matrix, and iteratively computes the exact base matrix. Since it is performed, it can be seen that the time required is considerably long by many repeated executions. Nister's method takes less time than Gibson's method, but since it calculates the sharpness of each image, it is sensitive to the size of the image and it can be seen that there are more important images selected than other methods. In the case of pattern box and GOM-I image, almost all images were selected because the image was very sharp. However, the method according to the embodiment of the present invention can be seen that the time required is considerably shorter than other methods.

FIG. 5 illustrates a ratio (●) of a corresponding key image to a feature point (●) and a reprojection error (■) of each image with respect to the CG box image. 6 shows the results for a Bear-II image. Referring to FIG. 5, it can be seen that the reprojection error decreases and increases as the frame progresses, which shows that the baseline should have sufficient baseline.

2. Estimation of Base Matrix

Next, we experimented the basic matrix to find the corresponding points from the selected important images. In order to accurately estimate the image correction and the basic matrix, the image acquired using the pattern is shown in FIG. 7.

7 (e) and 7 (f) show the projection matrix of each image by using the pattern information, and the epipolar distance error (●) and the normalized 8-point algorithm for the basic matrix estimated through algebraic derivation. It shows the epipolar distance error (▲) for the basic matrix calculated by. It can be seen that the base matrix created from the exact projection matrix has a very small error.

8 (a) and 8 (b) show an epipolar distance error and a reprojection error of an image for some reconstructed points by reconstructing the image of FIG. 7, respectively. The average epipolar distance error of the base matrix calculated by the normalized 8-point algorithm is 1.26 pixels, the average epipolar distance error of the base matrix estimated by the algebraic derivation is 1.04 pixels, and the average reprojection error is 0.51 pixels. to be.

FIG. 9 is an experiment comparing a method of estimating a basic matrix for a pattern box image with a method based on algebraic derivation from various algorithms and projection matrices. 9 (a) shows the time taken to calculate the base matrix from the corresponding point, and (b) shows the average error of the base matrix estimated by various methods.

3. Fully automatic calibration

FIG. 10 shows an experimental result of a ratio (▲) of reconstructed corresponding points to feature points and an error value (◆) of reprojecting the reconstructed three-dimensional data to an image for the full-automatic calibration of the selected important image. Since the edge of the image is used, many feature points are extracted. In this experiment, the threshold of the reprojection error was set to 0.05 pixels, so that the corresponding points including the error larger than the threshold were not reconstructed. 10 (a) shows the result of the pattern box. The average reprojection error generated during the previous automatic calibration is 0.0398 pixels, and the average reprojection error generated during the full automatic calibration is 0.0374 pixels. FIG. 10 (b) shows an experimental result for the bear-I image, and the average reprojection error is 0.373 pixels at the time of the pre-automatic calibration and 0,0382 pixels at the time of the automatic calibration. It can be seen that as the frame progresses, the number of corresponding reconstruction points decreases.

Table 2 shows the number of 3D data reconstructed from the medium images selected by each method and the average reprojection error for CG and GOM-II images. In the case of CG images, according to an embodiment of the present invention, many points were reconstructed with a small number of images but a minimum reprojection error. In the case of GOM-II images, the Nister method unnecessarily reconstructed many points from Chapter 13 images and has the largest reprojection error as shown in the experimental results. In the case of the Gibson method, the baseline of the selected important image is too large to reduce the number of reconstructed 3D data compared to other methods. Since edges are used for feature point extraction, much 3D data can be obtained. 11 and 12 are three-dimensional reconstructed result images of the CG image and the bear-II image.

As described in detail above, according to an embodiment of the present invention, an important image selection algorithm for selecting an image having a minimum reprojection error for three-dimensional reconstruction from a plurality of non-corrected images, and a camera automatic correction between important images We apply the epipolar constraint by estimating the base matrix through algebraic derivation from the camera projection matrix. According to this embodiment of the present invention, as a method for minimizing the error of the reconstructed three-dimensional data, only the image having the minimum reprojection error is selected. In addition, the camera can be calibrated for the entire selected middle image even if the corresponding points disappear through the camera automatic calibration process. According to this embodiment of the present invention, it is possible to reconstruct the three-dimensional model more accurately and quickly.

The embodiments of the present invention described in detail above are merely illustrative of the technical idea of the present invention, and should not be construed as limiting the technical idea of the present invention by the embodiments. The protection scope of the present invention is specified by the claims of the present invention described later.

1 is a flowchart illustrating the entire reconstruction procedure of a three-dimensional model according to an embodiment of the present invention.

FIG. 2 is a flowchart illustrating an example of a process of selecting an important image in the flowchart of FIG. 1.

3 is a flowchart illustrating an example of a full automatic calibration process in the flowchart of FIG. 1.

4 shows a portion of a test image sequence used to test the effects of an embodiment of the present invention.

FIG. 5 is a graph showing a comparison between experimental results of an embodiment of the present invention and test results of a test image sequence CG image. FIG.

FIG. 6 is a graph showing a comparison between experimental results of an exemplary method and experimental results of an embodiment of the present invention for a test image sequence Bear-II image.

7 (a)-(d) are test images used in a basic matrix estimation process according to an embodiment of the present invention, and FIGS. 7 (e) and 7 (f) are conventional normalized 8-point algorithms It is a graph showing the epipolar error when using the algebraic derivation algorithm used in the embodiment of the invention.

8 is a graph showing the epipolar distance error and the reprojection error when using the conventional normalized eight-point algorithm and the algebraic derivation algorithm used in the embodiment of the present invention, respectively.

FIG. 9 is a diagram illustrating a result of a comparison experiment of a basic matrix estimation time and an error. FIG.

10 is a graph showing the results of a full automatic calibration experiment using the selected important image.

FIG. 11 is a diagram illustrating a 3D reconstruction result generated using an embodiment of the present invention for a test image sequence CG image.

FIG. 12 is a diagram illustrating a three-dimensional reconstruction result generated using an embodiment of the present invention for a test image sequence Bear-II image.

Claims (6)

A method for reconstructing a three-dimensional model from an image sequence, A portion of the image sequence includes a k (k is an integer of 1 or more) th major image and a plurality of comparative images subsequent to the k th important image, and Selecting a (k + 1) th important image from the comparison images by using a reprojection error of each of the comparison images together with a ratio of the corresponding points of the comparison images to the feature points of the kth important image ; Obtaining a projection matrix of each of the important images by performing an automatic camera calibration procedure using the selected important images in all of the image sequences; Obtaining a basic matrix using the projection matrix of the important images adjacent to each other; And Removing noise from each of the edges of the important images using an epipolar constraint using the basic matrix, and reconstructing the 3D model using the edges from which the noise is removed. . The 3D model reconstruction method of claim 1, wherein the key image having k is 1 is a first image of the image sequence. The method of claim 1, wherein a part of the image sequence comprises images of the image sequence from the kth important image to a comparison image in which an image of which the ratio is less than a first threshold is first generated, and wherein (k + The first important image is a reconstruction method of a 3D model, characterized in that the comparison image is a minimum reprojection error of a portion of the image sequence. The 3D model of claim 3, wherein the (k + 1) th important image is selected from comparison images in which the ratio is less than or equal to a second threshold, and wherein the second threshold is smaller than the first threshold. Method of reconstruction. The method of claim 3, wherein the reprojection error (R.E.) is calculated using the following equation (E-1).

(E-1) Here, P ⁱ is a projection projection matrix, X _j is an arbitrary point in three-dimensional space, x ⁱ _j is a point in the projection coordinate system corresponding to X _j , and d is a distance. The method of claim 1, wherein the basic matrix is calculated through algebraic derivation from the projection matrix.

Images