KR100805802B1 - Apparatus and method for camera auto-calibration in motion blurred sequence, Augmented reality system using it - Google Patents

Abstract

1. TECHNICAL FIELD OF THE INVENTION

The present invention relates to an apparatus and method for automatically correcting a camera having a severe motion blur and an augmented reality system using the same.

2. The technical problem to be solved by the invention

The present invention provides accurate external variables (camera motion information) of a camera in an image with severe motion blur for use in augmented reality, which is used in film production or video production such as computer graphics (CG) advertisement. And an automatic camera correction apparatus and method thereof for obtaining a video with severe motion blur, and an augmented reality system using the same, to obtain a value of an internal variable (camera state information).

3. Summary of Solution to Invention

The present invention provides a camera automatic correction apparatus for removing a phenomenon in which intensity is spread by camera motion by moving a Gaussian smoothing of a severe motion blur input through an image input means. Gaussian smoothing means; Camera information acquisition means for acquiring external and internal variables of camera motion in the Gaussian smoothed image; And reconstruction filtering means for filtering an image in which a motion blur phenomenon occurs using a reconstruction filter based on the obtained values of the external variable and the internal variable, wherein the Gaussian smoothing means according to the degree of motion blur of the reconstructed image. And a Gaussian smoothing process and an internal / external variable acquiring process in the camera information acquiring means.

4. Important uses of the invention

The present invention is used for augmented reality in an image in which a motion blur occurs.

Motion Blur, Gaussian Smoothing, Camera Auto Correction, Feature Point, External Variable, Internal Variable, Augmented Reality

Description

Apparatus and method for camera auto-calibration in motion blurred sequence, Augmented reality system using it}

1 is an explanatory diagram showing the shape of a rectangular parallelepiped correcting object used for conventional camera calibration;

FIG. 2 is an explanatory diagram showing an image in which natural feature points are detected that cause a decrease in system performance due to a conventional motion blur phenomenon; FIG.

3 is a diagram illustrating an embodiment of a camera automatic correction apparatus for an image having severe motion blur and an augmented reality system using the same according to the present invention;

4 is a view illustrating an image in which a motion blur phenomenon is removed due to Gaussian smoothing according to the present invention;

FIG. 5 is a flowchart illustrating an embodiment of a method for automatically correcting a camera of an image having severe motion blur according to the present invention.

* Explanation of symbols on the main parts of the drawing

10: camera automatic correction device 11: video input unit

12: Gaussian smoothing unit 13: Camera information acquisition unit

14: reconstruction filtering unit 20: augmented reality system

The present invention relates to an apparatus for automatically correcting a camera having a severe motion blur and a method thereof, and an augmented reality system using the same. More particularly, the present invention relates to a movie production or a computer graphics (CG) advertisement. Camera auto-correction device of high motion blur image which acquires the exact value of external and internal variables of the camera from the image with high motion blur for use in augmented reality used in production, and The method and an augmented reality system using the same.

In the present invention, 'augmented reality' refers to a technology of inserting a graphic object into the live-action image, 'external variable' represents the motion information of the camera, 'internal variable' is the focal length of the camera, comb (Skew), an aspect ratio, and a state point of a camera including a principal point.

In general, in order to obtain geometric information of a subject and values of camera motion information (external variable) and state information (internal variable) from a video obtained through a camera, image information obtained from a camera and A camera calibration process for estimating the parameters between the actual geometric information of the subject is required. If you want to acquire values of camera's external variables (camera's motion information) and internal variables (camera's status information) during video shooting, the easiest and most accurate method can be used by using a motion control camera. In general, a method of obtaining information of a camera is not applicable to a pre-recorded image, and there is a problem that equipment of a motion control camera (MCC) is very expensive. Therefore, methods for performing camera automatic correction based on image processing techniques have been studied. These are largely divided into 'method using artificial correction object' and 'self-correction method using natural feature'. .

Here, the method using the artificial correction object is a method of performing automatic correction by photographing a three-dimensional cuboid-shaped correction mechanism as shown in FIG. 1 and solving the geometric relationship of the rectangular parallelepiped. This method has been the most widely used to date, and has the advantage of producing relatively accurate results, but the correction object must have the characteristics of a typical cuboid, and the image of the correction object must appear in the image sequence to perform automatic correction. There is a limitation that it should be caught, which is not suitable as an automatic camera calibration method for photo-realistic compositing.

On the other hand, the 'autonomous correction method' is a method of extracting a natural feature point from an image photographed from various angles and performing correction using only information of each corresponding point. When all the frames are paired by two, use the fundamental matrix or the essential matrix, or three pairs to perform the Projective Reconstruction using the Trifucal Tensor. Later, based on the assumption of "IAC (Image of Absolute Conic)" proposed by "Richard Hartley", the camera internal variable matrix is obtained and the camera projection matrix is distanced from the projective space. This is a method of obtaining a transformation matrix transformed into metric space. This method can be useful for real-world synthesis because it doesn't use artificial correction objects as in the method of using artificial correction objects.However, in the process of finding a connection relationship between corresponding points in each frame, noise There is a high risk of added performance resulting in deterioration of performance, and in many cases the LM (Levenberg Marquardt) algorithm must be inserted in order to create a robust system. Meanwhile, in the feature point extraction step, the KLT feature tracker (Kanade-Lucas-Tomasi Feature Tracker) has been known to be the best performing consistently extracting the relationship of the correspondence points, but this method also has a motion blur ( When a motion blur occurs, a large number of erroneous feature points that may cause performance deterioration are extracted as shown in FIG. 2, and this method is also not suitable as a camera automatic correction method for real-world synthesis.

The present invention has been proposed to solve the above problems, and in the case of an image having a heavy motion blur for use in augmented reality used in producing a movie or a computer graphics (CG) advertisement Camera auto-correction apparatus and its method for aggravating motion blur, and augmented reality using the same for acquiring the values of the external variables (camera motion information) and internal variables (camera status information) of the camera The purpose is to provide a system.

Other objects and advantages of the present invention can be understood by the following description, and will be more clearly understood by the embodiments of the present invention. Also, it will be readily appreciated that the objects and advantages of the present invention may be realized by the means and combinations thereof indicated in the claims.

In the apparatus of the present invention for achieving the above object, in the camera automatic correction device, the intensity of the motion blur (Motion) by the camera motion (movement) by Gaussian smoothing the image with a heavy motion blur input through the image input means (Intensity) Gaussian smoothing means for removing the phenomenon of smearing; Camera information acquisition means for acquiring external and internal variables of camera motion in the Gaussian smoothed image; And reconstruction filtering means for filtering an image in which a motion blur phenomenon occurs using a reconstruction filter based on the obtained values of the external variable and the internal variable, wherein the Gaussian smoothing means according to the degree of motion blur of the reconstructed image. And a Gaussian smoothing process and an internal / external variable acquiring process in the camera information acquiring means.

In addition, the present invention, in the augmented reality system for producing an image using the internal / external variables obtained by the camera automatic correction device, the external variable representing the motion information of the camera obtained through the camera automatic correction device and Based on an internal variable representing the state information of the camera, the graphic object is inserted into an image having severe motion blur.

On the other hand, the present invention, in the automatic camera correction method, the Gaussian smoothing step of removing the phenomenon that the intensity (Intensity) is spread by the camera motion (movement) by Gaussian smoothing the severe motion blur input through the image input means ; Camera information obtaining step of obtaining external and internal variables of the camera from the Gaussian smoothed image; A reconstruction filtering step of filtering an image in which a motion blur phenomenon occurs using a reconstruction filter based on the obtained values of the external variable and the internal variable; And repeatedly performing the Gaussian smoothing step and the camera information acquiring step according to the degree of motion blur of the reconstructed image.

The above objects, features and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, whereby those skilled in the art may easily implement the technical idea of the present invention. There will be. In addition, in describing the present invention, when it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 3 is a diagram illustrating an embodiment of a camera automatic correction apparatus for an image having severe motion blur and an augmented reality system using the same.

As shown in FIG. 3, the camera automatic correction apparatus 10 for an image having severe motion blur includes a Gaussian image input unit 11 for receiving an image and an image input to the image input unit 11. From the Gaussian smoothing image by Gaussian smoothing portion 12 and Gaussian smoothing portion 12 by Gaussian smoothing to remove the blurring of intensity due to camera motion (movement) by Gaussian smoothing Camera information acquisition unit 13 for acquiring the external variables (camera motion information) and internal variables (camera state information) of the camera motion, and external variables (motion information of the camera) acquired by camera information acquisition unit 13 ), And a filter using a restoration filter to remove the motion blur from the original image with heavy motion blur based on the values of the internal variable (camera state information). And a reconstruction filtering unit 14 for ringing, the Gaussian smoothing unit 12 and the camera information acquisition unit 13 repeatedly performing the Gaussian smoothing process and the internal / external variable acquiring process according to the degree of motion blur of the reconstructed image. do.

On the other hand, the augmented reality system 20 is a motion blur by using an external variable (camera motion information) and an internal variable (camera status information) of the camera obtained by the camera automatic correction device 10, Insert graphic objects into severe images.

는 특정 영상을 가우시안 스무딩시킨 주파수 응답을 나타낸다.Here, the Gaussian smoothing unit 12 performs Gaussian smoothing on the input image using a filter as shown in Equation 1 below. Equation 1 below represents a kind of low pass filter using a Gaussian function. At this time,

Denotes a frequency response obtained by Gaussian smoothing a specific image.

는 2차원 영상에 대응되는 주파수 영역의 2차원 공간에서의 좌표를 나타내고,

는 해당 주파수 위치에서의 명암도(Intensity)를 나타낸다. 그리고,

는 가우시안 함수의 폭을 조절하는 파라미터로서, 스무딩(Smoothing)되는 정도를 조절하는데, 이때

는 입력 영상의 모션 블러(Motion Blur)의 정도에 따라 적절하게 설정한다. 다만, 도 4에서는 5 픽셀(pixel)로 설정하였다.In [Equation 1],

Represents coordinates in the two-dimensional space of the frequency domain corresponding to the two-dimensional image,

Denotes the intensity at the frequency position. And,

Is a parameter to adjust the width of Gaussian function, and it controls the degree of smoothing.

Is appropriately set according to the degree of motion blur of the input image. In FIG. 4, however, it is set to 5 pixels.

By passing the image input to the image input unit 11 through a low pass filter using the Gaussian function of Equation 1, it is confirmed that the motion blur phenomenon, which was shown as a comb pattern, is removed. (See FIG. 4 below).

On the other hand, the camera information acquisition unit 13 basically uses the "Metric Reconstruction" method proposed by "Richard Hartley" in the Gaussian Smoothing image of the camera external variables (camera motion information) and internal variables (camera Get the value of). However, instead of the "Metric Reconstruction" method suggested by Richard Hartley, the camera's external variables (camera motion information) and internal variables (camera status information) can be changed using other correction methods, such as the method suggested by "Marc Polleyfeys". A value can be obtained (see FIG. 5 below).

Hereinafter, an operation process of the camera information acquisition unit 13 will be described in more detail.

Equation 2 shows a camera projection matrix assumed in the present invention.

는 카메라 내부변수를 나타내는 3 * 3 행렬이고,

는 카메라의 회전을 나타내는 3 * 3 행렬이고,

는 단위행렬(Identity Matrix)이며,

는 카메라의 초점 위치를 나타내는 벡터이다. In [Equation 2],

Is a 3 * 3 matrix representing camera internal variables,

Is a 3 * 3 matrix representing the rotation of the camera,

Is the identity matrix,

Is a vector representing the focus position of the camera.

The camera information acquisition unit 13 extracts the correspondence of the feature points from each frame using a KLT feature tracker (Kanade-Lucas-Tomasi Feature Tracker), and extracts two sheets at frame intervals according to the correspondence of the extracted feature points. The 8-point algorithm is used to calculate the fundamental matrix. Then, the projection matrix is calculated in the projective space as shown in Equation 3 below from the fundamental matrix.

는 두 영상 위의 특징점들 간의 대응관계로부터 사영공간(Projective Space)상의 기하학적인 관계를 설정해주는 3 * 3 행렬로서, 공간상의 한 점이 영상 1, 영상 2 위에서 좌표

를 갖는 경우에

으로 성립된다. 또한,

는 한 영상 위의 특징점 좌표를 다른 영상 위의 단일 직선(Epipolar Line)으로 매핑(mapping)시키는 것을 나타낼 수도 있다. 또한,

는 한 쪽의 카메라 초점을 다른 쪽의 카메라 영상 위로 사영시켰을 경우의 좌표(epipole)를 나타내는 벡터이고,

는

와

를 외적시키는 연산이며,

는 도 5의 카메라 정보 획득 단계(103)에서 양정부호(Positive Definite) 문제를 해결하기 위해서 행렬의 계수(rank)를 늘려주기 위한 3 * 1 임의의 값을 갖는 벡터이고,

은 임의의 상수(스칼라)이다.In Equation 3, a fundamental matrix is represented.

Is a 3 * 3 matrix that sets the geometric relationship in the projective space from the correspondence between the feature points on the two images.

If you have

Is established. Also,

May represent mapping of feature point coordinates on one image to a single straight line (Epipolar Line) on another image. Also,

Is a vector representing the coordinates (pipipole) when one camera focuses over the other camera image,

Is

Wow

Is an operation that crosses

Is a vector having a 3 * 1 random value to increase the rank of the matrix to solve the positive definite problem in the camera information acquisition step 103 of FIG.

Is any constant (scalar).

를 계산하고, 이렇게 구한

들의 집합(set)이 일관되게 위치하도록 사영공간(Projective Space)의 축을 맞추어준다. 이때, 사영복원(Projective Reconstruction)된 결과에서는 도형의 직선성만 보존되고, 평행성, 길이비율 및 크기는 보존되지 않는다.In this way, two-frame pairing is used to project 3D coordinates on the projective space of the feature points.

And calculate this

Align the axis of the projective space so that the set of fields is consistent. In this case, only the linearity of the figure is preserved in the result of projective reconstruction, and the parallelism, length ratio, and size are not preserved.

를 하기의 [수학식 5]에 따라 구한다.Equation 4 below is used to obtain Absolute Quadric Q using SVD (Singular Value Decomposition), and from this, Projection Matrix in Projective Space is calculated from Distance Space. Projective Transform Converts to Projection Matrix in Space

Obtained according to Equation 5 below.

는 i번째 카메라의 내부변수를 나타내는 행렬이며,

는 i번째 카메라의 사영행렬(Projection Matrix)을 나타낸다. 사영변환(Projective Transform)

를

에 곱하면 직선성뿐만 아니라 평행성, 길이비율까지 보존되는 거리복원(Metric Reconstruction)이 수행된다. 이때, 크기는 보존되지 않으므로 대상물체의 모양은 알 수 있지만 실제 길이의 척도는 알 수 없다.In [Equation 4],

Is a matrix representing the internal variables of the i-th camera,

Denotes the projection matrix of the i-th camera. Projective Transform

To

When multiplying by, the Metric Reconstruction is performed, which preserves not only linearity but also parallelism and length ratio. At this time, since the size is not preserved, the shape of the object may be known, but the actual length may not be measured.

The camera information acquisition unit 13 obtains values of an external variable (camera motion information) and an internal variable (camera state information) of the camera through the above processes ([Equation 2] to [Equation 5]). .

Meanwhile, the reconstruction filtering unit 14 models an image in which a motion blur phenomenon occurs due to a rapid movement of the camera as shown in Equation 6 below.

는 모션 블러링을 거친 영상을 나타내고,

는 모션 블러링을 거친 영상의 주파수 성분을 나타낸다. 그리고

는 모션 블러(Motion Blur) 현상으로 인한 필터효과(즉, 모션 블러링에 의한 주파수 응답)를 나타내고,

값은 현재부터 과거

시간까지 카메라 평면상에서 고정된 사물의 위치점 (x, y)가 이동한 값이며, 이는

시간동안 카메라의 움직임에 의해 결정되므로 (

)에 대한 정보는 상기 카메라 정보 획득부(13)에 의해 획득한 카메라의 움직임에 대한 근사값(카메라의 움직임정보(외부변수))을 이용하여 획득한다. 이렇게 모션 블러링(Motion Blurring), 가우시안 스무딩(Gaussian Smoothing) 과정을 거친 영상의 주파수 응답을 하기의 [수학식 7]과 같이 모델링한다.In Equation 6 above

Represents the motion blurred image,

Denotes the frequency component of the image which has undergone the motion blur. And

Denotes the filter effect (ie, the frequency response due to motion blur) due to the motion blur phenomenon,

Values are from now to past

This is the value by which the location point (x, y) of a fixed object moved on the camera plane by time, which is

Is determined by the camera's movement over time

) Is obtained by using an approximation value (motion information of the camera (external variable) of the camera) obtained by the camera information acquisition unit 13. The frequency response of the image that has undergone Motion Blurring and Gaussian Smoothing is modeled as shown in Equation 7 below.

는 모션 블러링(Motion Blurring)과 가우시안 스무딩(Gaussian Smoothing)을 모두 거쳤을 경우의 주파수 응답을 나타내고,

는 모션 블러링(Motion Blurring)과 가우시안 스무딩(Gaussian Smoothing)을 하나로 묶은 필터로 보았을 경우의 주파수 응답을 나타낸다.In [Equation 7],

Shows the frequency response when both Motion Blurring and Gaussian Smoothing are performed.

Shows the frequency response when viewed with a filter that combines Motion Blurring and Gaussian Smoothing.

From Equation 7, the response of the simplified minimum mean square error is obtained as shown in Equation 8 below. Here, the minimum mean square error represents a Wiener filter, but at this time, a restoration filter other than the Wiener filter may be used.

는 더 작은 값으로 설정한다. The Gaussian smoothing unit 12 performs Gaussian smoothing on the reconstructed filtered image in order to remove a motion blur phenomenon still remaining in the reconstructed filtered image by the reconstruction filtering unit 14. At this time,

Sets to a smaller value.

FIG. 5 is a flowchart illustrating an embodiment of a method for automatically correcting a camera of an image having severe motion blur according to the present invention.

First, an image of severe motion blur is input from the camera through the image input unit 11 of the camera automatic correction apparatus 10 (101), and the Gaussian smoothing unit 12 is input to the image input unit 11. Gaussian smoothing is performed by passing an image through a low pass filter using a Gaussian function (102).

Subsequently, an external variable (camera motion information of the camera) is obtained by using the "Metric Reconstruction" method proposed by "Richard Hartley" of the Gaussian smoothing image by the Gaussian smoothing unit 12 in the camera information acquisition unit 13. ) And initial values of internal variables (state information of the camera) are obtained (103).

The reconstruction filtering unit 14 may perform a motion blur based on values of an external variable (camera motion information) and an internal variable (camera state information) of the camera acquired by the camera information acquisition unit 13. The severe image is filtered using a restoration filter (104).

Next, the Gaussian smoothing unit 12 performs Gaussian smoothing on the image filtered by the reconstruction filtering unit 14 (105), and the camera information acquisition unit 13 performs Gaussian smoothing. After obtaining values of the external variables (camera motion information) and the internal variables (camera status information) of the camera from the image (106), the user has a motion blur phenomenon in the Gaussian smoothing image. It is determined whether it has been sufficiently removed (107).

As a result of the determination 107, if the motion blur phenomenon is not sufficiently removed, the process proceeds to step 104, where the Gaussian Smoothing step 105 and the camera information obtaining step 106 are repeated. Perform.

In this way, when the motion blur phenomenon is sufficiently removed, the external variables (camera motion information) and the internal variables (camera motion of the camera) acquired in the camera information acquisition step 106 by the augmented reality system 20 State information) (108).

Then, in the augmented reality system 20, the external variables (camera motion information) and the internal variables (camera movement of the camera) obtained in the camera information acquisition step 106 when producing a movie, a computer graphics (GC) advertisement, etc. Based on the state information), the graphic object is inserted into the image with severe motion blur.

As described above, although the present invention has taken an example of acquiring an external variable (camera motion information) and an internal variable (camera state information) of a correct camera in an image with severe motion blur, the present invention is used for augmented reality. It can be applied to other imaging fields.

As described above, the method of the present invention may be implemented as a program and stored in a recording medium (CD-ROM, RAM, ROM, floppy disk, hard disk, magneto-optical disk, etc.) in a computer-readable form. Since this process can be easily implemented by those skilled in the art will not be described in more detail.

The present invention described above is capable of various substitutions, modifications, and changes without departing from the technical spirit of the present invention for those skilled in the art to which the present invention pertains. It is not limited by the drawings.

The present invention as described above, by acquiring the values of the correct external and internal variables of the camera in the image with severe motion blur, the change of the image background in the synthesis of photorealistic images and graphic objects such as action scenes And there is an effect that can be applied to augmented reality without awkwardness.

Claims (9)

In the camera automatic correction device, Gaussian smoothing means for removing a phenomenon in which intensity is spread by camera motion (movement) by Gaussian smoothing an image having a heavy motion blur input through the image input means; Camera information acquisition means for acquiring external and internal variables of camera motion in the Gaussian smoothed image; And Restoration filtering means for filtering an image in which a motion blur phenomenon occurs using a restoration filter based on the obtained values of the external variable and the internal variable Including, but according to the degree of motion blur of the reconstructed image, the Gaussian smoothing means and the camera information obtaining means repeats the Gaussian smoothing process and the internal / external variable acquisition process, characterized in that the automatic camera of the motion blur severe image Correction device. The method of claim 1, The camera information obtaining means, The apparatus of claim 1, wherein the values of the external variable and the internal variable are calculated by using a metric reconstruction method proposed by Richard Hartley on the Gaussian smoothed image. The method of claim 1, The reconstruction filter, By using the obtained external variable of [Equation] of (

Camera auto-correction apparatus of the image with severe motion blur characterized in that it is determined. [Equation]

(here,

Represents the motion blurred image,

Represents the frequency components of the image after motion blurring,

Denotes the filter effect (ie, the frequency response due to motion blur) due to the motion blur phenomenon,

Values are from now to past

The position (x, y) of the fixed object on the camera plane is moved by time.) The method according to any one of claims 1 to 3, And the external variable represents motion information of the camera, and the internal variable represents state information of the camera. The method of claim 4, wherein Status information of the camera, Camera automatic correction device for a severe motion blur image, characterized in that it comprises at least one of the focal length, comb pattern, aspect ratio, pub information of the camera. In the augmented reality system for producing an image using the internal / external variables obtained by the camera automatic correction device of claim 1, And a graphic object is inserted into an image having severe motion blur based on an external variable indicating camera motion information acquired by the camera automatic correction device and an internal variable indicating camera state information. In the automatic camera calibration method, A Gaussian smoothing step of removing a phenomenon in which intensity is spread by camera motion (movement) by Gaussian smoothing an image having a heavy motion blur input through the image input means; Camera information obtaining step of obtaining external and internal variables of the camera from the Gaussian smoothed image; A reconstruction filtering step of filtering an image in which a motion blur phenomenon occurs using a reconstruction filter based on the obtained values of the external variable and the internal variable; And Repeatedly performing the Gaussian smoothing step and acquiring the camera information according to the motion blur of the reconstructed image Camera auto-correction method of a severe motion blur including a. The method of claim 7, wherein The external variable represents the motion information of the camera, and the internal variable represents the state information of the camera. The method of claim 8, Status information of the camera, Camera auto-correction method of the image with a severe motion blur, characterized in that it comprises at least one of the focal length, comb pattern, aspect ratio, pub information of the camera.

Images