KR100496186B1 - Method for estimation of projection matrix from the video sequence - Google Patents

Abstract

The present invention relates to a method of determining a projection matrix used to track the trajectory of the camera from a video image captured by a moving camera or to restore a three-dimensional structure of the video image. More specifically, the present invention includes the steps of selecting a keyframe from the video image; And estimating the projected absolute secondary cone surface by removing the projection camera matrix having a large error.

In the projection matrix determination method according to the present invention, an accurate projection matrix can be determined without an optimization process such as a bunch adjustment. Therefore, it can be usefully used for camera trajectory tracking, 3D structure reconstruction and 3D image synthesis.

Description

Method for estimation of projection matrix from the video sequence

The present invention relates to a method of determining a projection matrix used to track the trajectory of the camera from a video image captured by a moving camera or to restore a three-dimensional structure of the video image. More specifically, the present invention includes the steps of selecting a keyframe from the video image; And estimating the projected absolute secondary cone surface by removing the projection camera matrix having a large error.

Computer vision technology has been widely applied to visual effects since the 1990s, especially in the field of image synthesis that effectively combines real-life film and virtual objects. In order to synthesize a virtual object into a real image, a match move technique is essential, and there are two-dimensional image synthesis and a method of synthesizing a three-dimensional object in a two-dimensional image. Two-dimensional image synthesis places a virtual image in the live image, and keeps track of this position in the next frame and synthesizes it.

However, when camera movement is large and rotational components are included, three-dimensional image synthesis is required, and a technique for restoring the position and movement of the camera from the photographed film and constructing a three-dimensional model of the scene is required. .

In order to synthesize a virtual 3D object into a real image, a virtual object is rendered according to a camera trajectory extracted from the image, and a virtual image is synthesized. In order to position the virtual object in the target image, the camera trajectory and three-dimensional structure information of the captured scene are required. In addition, the accurate three-dimensional structure information takes into account that occlusion of the virtual object by the real image occurs, and also enables the collision processing between the real object and the virtual object.

The trajectory restoration of the camera is obtained by using the information of the image changed by the movement of the camera, which is based on the coincidence information between successive images. Matching information between images often uses a feature tracking technique. This technique considers the brightness information of the area around the target point to be tracked. It is common to install a marker before capturing an image in order to obtain the matching information easily and accurately. Although the motion information of the camera can be extracted by using the position information of the marker in each image, a lot of restrictions are imposed on the shooting scene.

Another method is to use mechanical equipment such as a motion control camera. Since such equipment moves by controlling a camera attached to mechanical equipment with a computer, the movement information of the camera is recorded, and the same movement can be repeatedly shot. However, these methods are space-constrained, costly, and difficult to express various intentions of the director (for example, holding by hand).

Techniques for obtaining camera information from two or several images have been actively studied since the 1990s. Recently, research on augmented reality, such as modeling a three-dimensional structure from a video image or synthesizing a virtual object by estimating a camera trajectory using an automatic correction technology of a camera, has been conducted.

In addition, a technique of using a basic matrix to obtain a camera trajectory from a video image and extracting an external parameter of the camera from the basic matrix to implement a 3D match move is also known [HS Sawhney, Y. Guo, J. Asmuth, and R. Kumar, "Multi-View 3D Estimation and Applications to Match Move", In proc. IEEE MVIEW , pp. 21-28, 1999]. In addition, a technique for three-dimensionally modeling a structure in an image sequence has been proposed, and this has been extended to extract a camera trajectory from a video image to implement augmented reality [K. Cornelis, M. Pollefeys, M. Vergauwen, and LV Gool, "Augmented Reality from Uncalibrated Video Sequences.", In proc. SMILE 2000, Lecture Notes in Computer Science , vol. 2018, pp 144-160, 2001.]. In addition, a method for improving the feature tracking method, restoring the hierarchical matching-based projection structure, and introducing the RANSAC method in the automatic calibration process of the camera has been proposed [S. Gibson, J. Cook, T. Howard, R. Hubbold, and D. Oram, "Accurate Camera Calibration for Off-line, Video-Based Augmented Reality", In proc. IEEE and ACM ISMAR , Darmstadt, Germany, Sep. 2002.].

However, in the above studies, the coordination process is essential because the video contains a lot of images. Therefore, there is a disadvantage in that the calculation amount is very large and the calculation process is complicated.

In order to solve the above problems, the present invention proposes a method of determining a projection matrix used to restore a camera trajectory and a three-dimensional structure in a video image without an optimization process such as a bunch adjustment.

First, we select keyframes suitable for camera trajectory analysis from the target video image to reduce the overall computation time, and in the second process, estimate the projected absolute secondary cone surface for more accurate camera automatic correction (to obtain the camera's internal parameters). Remove the image that contains a lot of errors.

The projection matrix determination method according to the present invention can obtain accurate results without the process of over-adjustment.

Accordingly, an object of the present invention is to provide a method for determining a projection matrix used for tracking a camera trajectory or reconstructing a three-dimensional structure of the video image from a video image captured by a moving camera.

Another object of the present invention is to provide a method of synthesizing a 3D image using the projection matrix.

The present invention relates to a method for determining a projection matrix used to track the trajectory of the camera from a video image captured by a moving camera or to restore a three-dimensional structure of the video image.

More specifically, the present invention,

Selecting a keyframe from the video image (a);

(B) estimating a projected absolute secondary cone surface in projected space from the keyframe;

(C) obtaining a spatial projective transformation matrix H from the estimated projected absolute quadratic cone surface; And

And (d) obtaining a projection matrix from the spatial projection matrix H.

In the projection matrix determining method according to the present invention, the key frame comprises: a ratio of the number of coincidence points to the feature points; Distribution of match points for the image; And a projection error by the planar projection transformation matrix between the two images.

In particular, the key frame,

The value of the ratio of the number of matching points N _c to the number of feature points N _F N _c / N _F ;

Standard deviation of the distribution of coincidence points for the image σ _p ; And

From the values H _{a, err} of the projection error by the planar projection transformation matrix between two images

It is preferable to calculate the discriminant value of Equation 1 below, and select a frame whose discriminant value is equal to or less than a predetermined value as a keyframe:

[Equation 1]

In the above formula,

S _ij is the discriminant value,

ω ₁ , ω ₂ and ω ₃ are weights.

In the keyframe selection step (a), the projection error H _{a, err} by the planar projection transformation matrix is preferably defined by Equation 2:

[Equation 2]

Wherein x _i and x ' _i are the coincidence points between the two images,

H _a is a planar projection matrix,

d (x, y) is the distance between points x and y,

N _c is the number of matches.

Further, in the keyframe selection step (a), σ _{P, which} is a standard deviation for the distribution of coincidence points, is preferably defined by the following equation:

[Equation 3]

Where N _S is the number of subregions divided,

N _ci is the number of matching points in the i-th subregion,

N _c is the number of matches,

The horizontal size W _s and the vertical size H _s of the subregion are respectively

And

ego,

Here, the function int (.) Means integer conversion.

In the method of determining a projection matrix according to the present invention, the projected absolute second conical surface estimating step of (b) is performed by removing an image representing an error of a predetermined threshold or more from the key frames selected in the step (a). It is desirable to estimate the absolute secondary cone surface.

In particular, in step (b), it is preferable to estimate a projected absolute secondary cone surface by removing an image showing an error above the predetermined threshold by a Least Median of Square (LMedS) method.

More preferably, the step (b) is,

(E) setting the projection camera matrix of the first camera as the reference projection camera matrix in the projection space;

(F) selecting two arbitrary projection camera matrices from the projection camera matrices of the remaining cameras except the first camera;

(G) estimating a projected absolute quadratic cone surface from the two selected projection camera matrices;

(H) calculating an error between the projection of the projection camera matrix of each camera and the projection of the first projection camera matrix by the estimated projected absolute quadratic cone surface;

(I) storing an intermediate value when arranging the calculated errors in order and the estimated projected absolute secondary cone surface;

Repeating steps (f) to (i) a predetermined number of times to obtain a projected absolute secondary cone surface that minimizes the median value, and storing the minimum median value;

Setting a predetermined error threshold value from the minimum intermediate value, and removing (k) the projection camera matrix whose error value obtained in step (h) is greater than the threshold value; And

Estimating an absolute secondary cone surface using the remaining projected camera matrix.

The error threshold τ in the step (k) is preferably obtained by the following equation (4):

[Equation 4]

In the above formula,

median is the minimum median

p is the total number of data sets (since the first camera is excluded in the present invention, p is preferably the number of total camera projection matrices minus one),

q is the number of sampled data in the entire data set (the present invention samples two camera matrices, preferably 2).

In addition, the present invention is a projection determined in accordance with the present invention the frame excluded in determining the projection matrix (i.e., the frame excluded in the selection of keyframes, and the frame having a large error, excluded in estimating the projected absolute secondary cone surface). The present invention relates to a method of synthesizing an arbitrary image into a 3D image to a video image including the reconstructed frame after reconstruction by a matrix. Camera information excluded in the projection matrix determination process may be finally restored by a camera ablation technique.

Hereinafter, a general description of the projection cone secondary curve estimation process in the projection structure used in the present invention.

The image obtained by the camera can be said to be projected on a plane of three-dimensional space is two-dimensional, the action of this camera is expressed as a projection matrix of three rows and four columns (3 × 4), as shown in Equation 5 below:

[Equation 5]

As shown in Equation 5, the camera projection matrix (hereinafter referred to as "camera matrix" or "camera projection matrix") may be expressed as a product of three matrices.

Among these, the right matrix is an external parameter and means a conversion from the world coordinate system to the camera coordinate system. R _{3 × 3} and t ₃ ^T are the rotational and moving components in the rigid transformation matrix, respectively.

The middle matrix serves to project the space into the plane.

The left matrix K converts the projection onto the plane as an internal parameter into the pixel coordinate system in the image.

The process of obtaining the internal parameter K of the camera projection matrix is called camera calibration. There are two ways to calibrate the camera. One is to use three-dimensional information (coordinates), and internal parameters are obtained by using the three-dimensional information that is known and the relationship that is projected on the image. In this method, a check box or the like which makes it easy to set a world coordinate system is mainly used. The other is an automatic correction method for obtaining camera information using only information in an image without using 3D information.

The external parameters R and t in Equation 5 contain information of the camera position and the viewing direction with respect to the world coordinate system. Meanwhile, the world coordinate system is determined by the user. Therefore, if the position of the first camera and the viewing direction coincide with the world coordinate system so that the first camera is a reference, the projection matrix of the first camera becomes P ₁ = K [I | 0].

If there are other cameras with the same internal parameters, then P ₂ = K [R | t], where R and t represent the position and rotation information relative to the first camera. Since the information about K is unknown, applying a 4x4 transformation matrix to eliminate K of the first camera matrix can be summarized as in Equation 6:

[Equation 6]

x ₁ = P ₁ X = K [I | O] X = K [I | O] HH ^-1 X = [I | O] H ^-1 X = [I | O] X _p

x ₂ = P ₂ X = K [R | t] X = K [R | t] HH ^-1 X = [M | p ₄ ] H ^-1 X = [M | p ₄ ] X _p

In the above formula,

X is a three-dimensional point,

x ₁ and x ₂ are the points projected within the two images,

H is a spatial projection matrix,

X _p is the projective structure as the projection transformation of X.

The projective structure of the two images can be reconstructed through the base matrix. By obtaining the spatial projective transformation matrix H from the projected structure, the Euclidean structure can be restored [A. Zisserman, P. Beardsley, and I. Reid, "Metric Calibration of a Stereo Rig.", In proc. IEEE Workshop Representation of Visual Scenes , pp. 93-100, Cambridge, 1995.]. Also, since the camera projection matrix can be reconstructed, internal and external parameters can be extracted in Euclidean space.

Therefore, in order for the automatic correction of the camera to be accurate, the projective structure reconstruction must be accurate, and it is important to accurately extract the spatial projection transformation from the projective structure.

The relationship between the Euclidean space and the projected space is shown in FIG. 1. By applying the spatial projection transformation matrix H to the projection camera projection matrix for every image, a camera projection matrix for every image in Euclidean space can be obtained.

For example, if the number of frames in the video image is 100, there are 100 projection camera projection matrices, which can be used to estimate the projected absolute secondary cone surface. One spatial projection transformation matrix H can be obtained from the projected absolute quadratic cone surface. By applying this to the 100 projection projection matrices, 100 Euclidean camera projection matrices can be obtained, which actually represent the trajectory of the camera movement.

In the following, the projected structure reconstruction and absolute secondary cone surface estimation are described.

1. Restore the projective structure

The projective structure can be reconstructed using the base matrix from two images. The three images can be reconstructed from a trifocal tensor [R. Hartley and A. Zisserman, Mutiple View Geometry in Computer Vision , Cambrige University Press. 2000.]. In addition, a method of restoring the projective structure from several images has been proposed, which is classified into a factoriazation based method and a merging based method.

The decomposition base is less affected by error accumulation because the camera projection matrix and the three-dimensional structure are computed simultaneously. In contrast, the merge-based method relies heavily on the accuracy of the initial estimation and is suitable for application to long sequence images such as video images [A. Azarbayejani and A. Pentland, "Recursive Estimation of Motion, Structure, and Focal Length.", In proc. IEEE Trans. on Pattern Matching and Machine Intelligenec , vol. 17, no. 6, pp. 567-575, June 1995.].

The merge-based method is further divided into a sequential merging method and a hierarchical merging method. While the sequential merging method is highly dependent on initial estimates and errors are cumulative, the hierarchical method has the advantage of reducing this.

2. Absolute Quadratic Quadric Estimation

Absolute quadratic cone surfaces are cut by an infinite plane and have a curved absolute cone curve. The absolute quadratic cone surface is represented by a 4x4 symmetric matrix with a rank of three.

Also, at the Euclidean space Becomes the form of.

The relationship between the camera matrix P and the absolute quadratic cone surface Q is given by:

[Equation 7]

PQP ^T = KK ^T

According to the projected structure reconstruction theory, the relationship between the camera matrix P on the Euclidean and the camera matrix P _p on the projected structure is described by an arbitrary spatial projection transformation H, and P _p also satisfies Equation 7 above.

Regarding the internal parameters of the camera, it is assumed that the horizontal and vertical ratios of the pixel are 1, the skew of the camera is 0, and the principal point is the center of the image, and the elements are mapped to each other. From can be derived a linear equation such as

[Equation 8]

(P _p Q _p P _p ^T ) ₁₂ = 0

(P _p Q _p P _p ^T ) ₁₃ = 0

(P _p Q _p P _p ^T ) ₂₃ = 0

(P _p Q _p P _p ^T ) ₁₁ = (P _p Q _p P _p ^T ) ₂₂

In the above formula, (A) _ij means the elements of the i-th row, j-th column of the matrix A,

Q _p is a 4x4 symmetric matrix, an absolute quadratic cone surface in projected space, with 10 unknowns and 9 degrees of freedom.

Four equations can be made from the camera matrix in the projected structure, and Q _p can be estimated from at least four camera matrices. After estimating Q _p from Equation 8 in the relation between Q in Euclidean space and Q _p in the projected structure, H ⁻¹ is obtained using eigenvalue decomposition. In other words,

In, D matrix is a diagonal matrix, which can be easily divided into two by square root of each element, and H ^-1 can be obtained by multiplying by U matrix.

In this case, since the absolute secondary cone surface has a coefficient of 3, The last element of is 0.

This makes H- ¹ an irreversible matrix. therefore, Creates H ^-1 by inserting a random number into the last element of, usually replaced by 1 [B. Triggs, "Autocalibration and The Absolute Quadric", In proc. IEEE CVPR , pp. 609-614, 1997.].

Hereinafter, an embodiment of a projection matrix determining method according to the present invention will be described in detail with reference to the drawings. However, the present invention is not limited by the following examples.

2 is a flowchart of a projection matrix determining method according to the present invention.

As shown in FIG. 2, the present invention provides a method of determining a projection matrix used to track a camera's trajectory or restore a three-dimensional structure of a subject from a video image captured by a moving camera.

Selecting a keyframe from the video image (S10);

Estimating a projected absolute secondary cone surface in the projected space by removing an error-rich image from the keyframe by an LMedS method (S20);

Obtaining a spatial projective transformation matrix H from the estimated projected absolute quadratic cone surface (S30); And

Obtaining a projection matrix from the spatial projection matrix H (S40).

Hereinafter, a process of selecting a key frame suitable for camera trajectory analysis from the target video image will be described.

Keyframe selection can drastically reduce the amount of computation required for reconstruction of camera movements and scene structures in video data with a large number of frames. After the keyframe selection process, the degree of camera movement between images becomes even.

For more efficient camera correction, as few keyframes as possible should be selected, and at the same time, frames should be selected to represent the entire camera movement. In addition, it should be possible to accurately interpret the motion information of the camera from the selected images. Hereinafter, keyframe selection discrimination equations and discrimination criteria used in the present invention will be described.

a-1. Keyframe Selection Determination

In general, in order to analyze the movement of the camera from the selected image, it is necessary to effectively configure the correspondence between the target images. The greater the distance between the target scene and the camera, the more accurately the camera's movement can be interpreted. To select an image that satisfies the above conditions, three things are considered:

i) ratio of the number of coincidence points to the number of feature points;

ii) the projection error due to the planar projection transformation matrix between the two images; And

iii) Consideration of the distribution of coincidence points for the image.

First, the ratio of the number of coincidence points to the number of feature points considers whether extraction of coincidence points between images is sufficient. In a video image, the change of the image due to camera movement generally decreases the number of coincidence points over time, and this information defines the range in which the next keyframe will be selected.

Secondly, the projection error by the planar projection transformation matrix indicates the degree of movement of the camera in the current frame, and from this, whether the camera movement can be accurately interpreted.

Third, the coincidence distribution in the target image is considered to check whether the epipolar geometry can be estimated well. The epipolar geometry includes camera motion information between two images, and is represented in a logarithmic matrix and estimated from the coincidence information between the images. The base matrix is very sensitive to noise, and the more evenly matched points are distributed in the image, the more adaptive it is to noise and allowing robust base matrix estimation.

Equation 1 below is a discriminant for considering the three factors in order to select an effective keyframe:

[Equation 1]

In the above formula, S _ij is a discriminant value,

ω _i (i = 1, 2, 3) is the weight,

N _c and N _F are the number of coincident and feature points, respectively,

σ _P is the standard deviation of the distribution of coincidence points in the target image,

H _{a, err} is the projection error due to the planar projection transformation matrix.

The weight is set to a predetermined value according to the degree of importance of each term in Equation (1). In the experimental example described later, ω ₁ = 3, ω ₂ = 1, and ω ₃ = 10.

The keyframe is selected by the discrimination equation, and the first image and the last image of the video sequence are also set as keyframes. The discriminant value is calculated until the ratio of the feature points of the first image is 50% to the number of feature points of the first image, and the image having the smallest discriminant value is selected as the keyframe. Repeat the above process until the last frame and continue to get keyframes.

a-2. Planar Projection Error

The one-to-one correspondence is satisfied between the images obtained from the non-moving camera by the planar projection transformation matrix, and this property is used for the image mosaic technique. It is impossible to restore the 3D structure information in the image without the camera movement. Therefore, the smaller the plane projection conversion error, the less suitable for camera automatic correction.

The planar projection transformation matrix between images has a one-to-one relationship as the camera moves, and the larger the camera movement, the larger the projection error according to the depth information. That is, the projection error represents the motion information of the two cameras, which means the distance between the cameras. Projection errors can be obtained from at least four pieces of concordance information between two images, and four or more points are obtained using a DLT (Direct Liner Transformation Algorithm) method.

Equation 2 is an expression representing the projection error by the planar projection transformation matrix.

[Equation 2]

In the above formula, x _i and x ' _i are homogeneous coordinates as a coincidence point between the two images,

H _a is a planar projection matrix,

d (x, y) is the distance between points x and y,

N _c is the number of matches.

a-3. Standard deviation of coincidence distribution

The distribution of coincidence points greatly influences the estimation of the base matrix. Therefore, if the matching point distribution is uniform for the image, a more accurate and stable basic matrix can be estimated.

To check whether the coincidence points are distributed evenly in the image, the image is divided into sub-sized regions of constant size, and the distribution of coincidence points in the target image is analyzed by using the difference between the coincidence points density in the entire image and the coincidence points density in each subregion. Can be. If the number of divided regions equals the number of coincidence points, if there is one coincidence point for each region, the coincidence points may be considered to be most evenly distributed in the image. When dividing an image into regions of uniform size, it is preferable to divide the shape of the area into the same rectangle as the shape of the entire image.

Equation 9 is used to segment the entire image into subregions of uniform size close to the number of corresponding points:

[Equation 9]

Where int (.) Means integer conversion,

W _S and H _S are the width and length of the subregion, respectively.

N _c is the number of matches.

Using Equation 9, the size of the subarea is obtained, and the entire target image is divided into sizes in each direction of the subarea.

In order to calculate the distribution of the matching points for the target image, the number of matching points existing in each subregion is obtained, and the standard deviation of the matching point density for the area of the entire image and the density of the matching points existing in each subregion is used.

Since the subregions have a constant size, assuming that the area of the subregion is 1, the standard deviation equation for the density of the points is given by Equation 3:

[Equation 3]

Where N _S is the number of subregions divided,

N _ci is the number of matching points in the i-th subregion,

N _c is the number of matches.

Hereinafter, the step (b) of estimating an absolute secondary cone surface based on LMedS will be described.

Absolute quadratic cone surfaces can be estimated from three or more images through linear equations, but these linear equations are difficult to obtain accurate solutions due to noise effects.

In the present invention, a more accurate absolute quadratic cone surface is estimated by eliminating an error-prone camera matrix using the LMedS method based on the Monte Carlo technique.

The first projection camera matrix among the projection camera matrices is set to be the reference of another projection camera matrix. That is, all the camera matrices are multiplied by a spatial projection matrix such that the first projection camera matrix is [I | 0]. Assuming P = [M | p ₄ ], we can apply the spatial projection matrix T as shown in Equation 10 to all camera matrices so that the first camera matrix becomes [I | O]:

[Equation 10]

When projecting the projected absolute secondary cone surface on the projected structure to the first projection camera, the projected absolute secondary cone surface is reduced from 10 to 7 unknowns. This means that it can be estimated from at least two camera matrices except the first camera.

In order to remove the error-prone camera matrix from the absolute quadratic cone surface estimation, the absolute quadratic cone surface is estimated by randomly selecting two camera matrices from the set of camera matrices except the first camera matrix. The estimated absolute quadratic cone surface is projected on the remaining cameras to calculate the errors in the first camera matrix and the errors in the respective camera matrices. Equation 11 below calculates the error r _i of the i-th camera matrix:

[Equation 11]

_i = r (λ ₀ P ⁰ Q _{p p p} P ^0T - λ _i P ⁱ Q _{p p p} P ^iT) _norm

Where (.) Norm represents the magnitude of the vector,

P _p ⁰ is the projection camera matrix of the first camera,

P _p ⁱ is the projection camera matrix of the i th camera,

Q _p is the projected absolute quadratic cone surface,

Since (KK ^T ) ₃₃ = 1, λ ₀ and λ _i are 1 / (P _p ⁰ Q _p P _p ^0T ) ₃₃ and 1 / (P _p ⁱ Q _p P _p ^iT ) _33, respectively.

The above process is repeated a predetermined number of times to store the absolute secondary cone surface when the error of the intermediate position (hereinafter referred to as intermediate value) is the smallest.

The threshold for distinguishing an error-prone camera matrix using the minimum median and the absolute quadratic cone surface at that time uses Equation 4:

[Equation 4]

Where median is the minimum median,

p is the total number of camera projection matrices (that is, the total number of frames) minus one,

q is 2.

The threshold value is determined from Equation 4, the camera matrix having an error larger than the threshold value is excluded, and the absolute secondary cone surface is estimated again from the remaining camera matrix.

3 is a flow chart of step (b) of estimating the projected absolute secondary cone surface described above.

As shown in FIG. 3, estimating the projected absolute secondary cone surface, (b),

Setting the projection camera matrix of the first camera as the reference projection camera matrix in the projection space (S21);

Selecting two arbitrary projection camera matrices from the projection camera matrices of the remaining cameras except the first camera (S22);

Estimating a projected absolute quadratic cone surface from the two selected projection camera matrices (S23);

Calculating an error between the projection of the projected camera matrix of each camera and the projection of the first projection camera matrix with respect to the estimated projected absolute quadratic cone surface (S24);

Storing a median value when arranging the calculated errors in order and the estimated projected absolute secondary cone surface (S25);

Repeating the steps (S22) to (S25) a predetermined number of times (S26), obtaining a projected absolute secondary cone surface that minimizes the median value, and storing the minimum median value (S27);

Setting a predetermined error threshold value from the minimum intermediate value, and removing a projection camera matrix having an error value obtained in the step (S24) greater than the threshold value (S28); And

Estimating an absolute secondary cone surface using the remaining projected camera matrix (S29).

By using the projected absolute quadratic cone surface estimated by the method, the spatial projection transformation matrix H can be obtained as described above (S30), and through this, the projection matrix of the camera can be obtained (S40).

Using the obtained projection matrix, the trajectory of the moving camera can be tracked and the 3D Euclidean structure of the target image can be restored.

As described above, in order to track the trajectory of the camera, the camera matrix of the frame excluded from the selection of the key frame and the LMedS based second absolute cone surface estimation process may be restored in a conventional manner. It is possible to estimate the camera matrix from at least six pieces of matching information, and this process is called camera resection.

Hereinafter, an experimental example of the camera restoration method according to the present invention.

1. Keyframe Selection Process

In the present invention, three video sequences were tested, and the number of keyframes and processing time selected by the conventional method and the method according to the present invention are shown in Table 1 and Fig. 4, respectively:

Table 1. Number of keyframes extracted and processed by each method

Experimental video Keyframes Processing time (sec.) Target image Frames Image size Niceter method Gibson Method Method according to the invention Niceter method Gibson Method Method according to the invention box 621 720 × 80 23 14 15 412 416 179 desk 407 720 × 80 17 13 13 201 200 76 Fountain 134 320 × 40 28 12 12 16.63 38.59 10 cottage 100 720 × 76 27 27 24 74 102 42

The experimental environment is 1Gbytes of CPU-Pentium 4 2.3GHz RAM.

Nister et al. [David Nister, "Frame Decimation for Structure and Motion", In proc. SMILE 2000 , Dublin, Ireland, July 2000.] calculates the sharpness of each image, so it takes a lot of calculation time depending on the image size and the number of selected keyframes is relatively higher than other methods. The above-mentioned S. Gibson's method selects fewer keyframes than the varistor's method, but requires much computation time. The Gibson method requires a process of estimating a base matrix in order to consider epipolar geometry. Since the base matrix is very sensitive to noise, an iterative method is necessary to make an accurate estimate. Therefore, the Gibson's method selects less keyframes than the varistor method, and there is no significant difference in processing time.

Instead of directly estimating and using the base matrix, the method according to the present invention analyzes whether the correct base matrix can be estimated through the coincidence distribution. This can greatly reduce processing time, and the number of keyframes finally selected is similar to the Gibson method. FIG. 3 shows the connectivity of the camera information and the position of the key frame analyzed by the proposed method and the conventional method for the fountain image.

2. Application to virtual images

In order to evaluate the performance of the method according to the present invention, the experiment was performed on the virtual image. The 3D virtual object was configured as shown in FIG. 5, and the camera trajectory was raised on the y axis while rotating about the object. Internal parameters were fixed arbitrarily and noise was added randomly from -0.5 to 0.5. 5 shows structure data and a camera trajectory for the virtual object.

We performed the sequential projective restoration method and the hierarchical projective restoration method for the virtual camera. Absolute quadratic surfaces were estimated by the linear method, the bunch adjustment method and the method according to the present invention.

6 shows the camera trajectories restored by the respective methods. It can be seen that both projective restoration methods are more accurate than the linear method. In particular, it can be seen that the method according to the present invention can obtain a result similar to the actual camera trajectory without considering the complicated crowd adjustment method.

7 shows the cumulative error of each camera parameter obtained using an absolute quadratic curve.

3. Performance test on live image

On a live-action image, a box on a desk was taken to evaluate the performance of the method according to the invention. The total number of images is 621, and the size of the images is 720 × 460.

FIG. 8A illustrates three of the target video frames.

The image selected by the discriminant used in the present invention is shown in the camera motion graph (b).

The motion graph indicates a plane projection transformation error between adjacent images in each image. Fifteen keyframes were selected over the entire sequence, taking 2 minutes and 59 seconds (see Table 1 above). The portion shown by the dotted line represents the image selected as the key frame. It can be seen from the graph that a frame with a relatively large amount of motion is selected.

(c) shows the result obtained by applying linear method, group adjustment method and proposed method to sequential and hierarchical projective restoration method. The graph shows the cumulative distance error from the principal point, and the center of the image is set as the exact reference principle point. In both the sequential projective restoration method and the hierarchical projective restoration method, it can be seen that the method according to the present invention is superior to the result of applying the group adjustment, as in the result of the virtual image.

FIG. 9 illustrates a result of synthesizing a virtual object into a real image using a three-dimensional structure and a camera trajectory obtained by applying an algorithm according to the present invention to various video sequences.

As described above, the projection matrix determination method according to the present invention can determine an accurate projection matrix without an optimization process such as bunch adjustment. Therefore, it can be usefully used for camera trajectory tracking, 3D structure reconstruction and 3D image synthesis.

1 illustrates the relationship between the projective structure and the Euclidean structure,

2 is a flowchart of a projection matrix determining method according to the present invention;

3 is a flowchart of the step of estimating the projected absolute secondary cone surface in the present invention,

4 shows a comparison of the connectivity of the camera information and the position of the key frame analyzed by the conventional method and the method of the present invention.

5 illustrates a structure and a camera trajectory of a virtual image to which the method according to the present invention is applied.

Figure 6 is a camera trajectory obtained by estimating the absolute second-order surface by the linear method, the group adjustment method and the method according to the invention, respectively,

FIG. 7 illustrates the cumulative error of each camera parameter obtained using an absolute quadratic curve.

8 shows the results of applying the method according to the present invention to the live-action image,

9 illustrates a result of synthesizing a virtual object into a real image using a projection matrix obtained by applying the method according to the present invention to various video images.

Claims (11)

In the method for determining the projection matrix used to track the trajectory of the camera from the video image captured by the moving camera or to restore the three-dimensional structure of the video image, Selecting a keyframe from the video image (a); (B) estimating a projected absolute secondary cone surface in projected space from the keyframe; (C) obtaining a spatial projective transformation matrix H from the estimated projected absolute quadratic cone surface; And And (d) obtaining a projection matrix from the spatial projective transformation matrix H. The method of claim 1, In the keyframe selection step of step (a), the keyframe is, Ratio of number of coincidence points to feature points; Distribution of match points for the image; And The method is selected based on the projection error by the planar projection transformation matrix between the two images. The method of claim 2, In the keyframe selection step of step (a), the keyframe is, The value of the ratio of the number of matching points N _c to the number of feature points N _F N _c / N _F ; Standard deviation of the distribution of coincidence points for the image σ _p ; And From the values H _{a, err} of the projection error by the planar projection transformation matrix between two images A method for calculating a discriminant value of Equation 1 below and selecting a frame having a discriminant value equal to or less than a predetermined value as a key frame: [Equation 1] In the above formula, S _ij is the discriminant value, ω ₁ , ω ₂ and ω ₃ are weights. The method of claim 3, wherein the projection error H _{a, err} due to the planar projection transformation matrix is defined by Equation 2. [Equation 2] Wherein x _i and x ' _i are the coincidence points between the two images, H _a is a planar projection matrix, d (x, y) is the distance between points x and y, N _c is the number of matches. The method of claim 3, wherein σ _{P, which} is a standard deviation of the distribution of coincidence points, is defined by Equation 3 below: [Equation 3] Where N _S is the number of subregions divided, N _ci is the number of matching points in the i-th subregion, N _c is the number of matches, The horizontal size W _s and the vertical size H _s of the subregion are respectively And ego, Here, the function int (.) Means integer conversion. The method of claim 1, wherein in step (b), And estimating a projected absolute secondary cone surface by removing an image representing an error equal to or greater than a predetermined threshold value among the key frames selected in the step (a). The method of claim 6, The step (b) is characterized in that to remove the image showing the error above the predetermined threshold by the least median of square (LMedS) method. The method of claim 6, wherein step (b) (E) setting the projection camera matrix of the first camera as the reference projection camera matrix in the projection space; (F) selecting two arbitrary projection camera matrices from the projection camera matrices of the remaining cameras except the first camera; (G) estimating a projected absolute quadratic cone surface from the two selected projection camera matrices; (H) calculating an error between the projection of the projection camera matrix of each camera and the projection of the first projection camera matrix by the estimated projected absolute quadratic cone surface; (I) storing an intermediate value when arranging the calculated errors in order and the estimated projected absolute secondary cone surface; Repeating steps (f) to (i) a predetermined number of times to obtain a projected absolute secondary cone surface that minimizes the median value, and storing the minimum median value; Setting a predetermined error threshold value from the minimum intermediate value, and removing (k) the projection camera matrix whose error value obtained in step (h) is greater than the threshold value; And Estimating an absolute quadratic cone surface using the remaining projected camera matrix. The method of claim 8, The error threshold value τ in the step (k) is obtained by the following equation (4): [Equation 4] In the above formula, median is the minimum median p is the total camera projection matrix-1 q is 2. The method of claim 8, wherein the error r _i of the i-th camera matrix in the step (h) is obtained by the following equation (11): [Equation 11] _i = r (λ ₀ P ⁰ Q _{p p p} P ^0T - λ _i P ⁱ Q _{p p p} P ^iT) _norm Where (.) Norm represents the magnitude of the vector, P _p ⁰ is the projection camera matrix of the first camera, P _p ⁱ is the projection camera matrix of the i th camera, Q _p is the projected absolute quadratic cone surface, λ ₀ and λ _i are 1 / (P _p ⁰ Q _p P _p ^0T ) ₃₃ and 1 / (P _p ⁱ Q _p P _p ^iT ) _33, respectively. After restoring the frame excluded in determining the projection matrix by the projection matrix determined in claim 2 or 8, And synthesizing an arbitrary image into a 3D image to a video image including the reconstructed frame.