KR100421208B1 - Wethod for synthesizing three-dimensional image - Google Patents

Abstract

PURPOSE: A method for synthesizing a 3D image is provided to photograph an object in the front of a background of a single color, automatically segment the object, and synthesize the segmented object with a different image. CONSTITUTION: An image with multi-resolution is generated from an image of a photographed object(201,202). The optical flow of a lower resolution image is firstly obtained among the images of the multi-resolution, and the optical flow of an upper resolution image is obtained according to obtain optical flow information to obtain the optical flow of the entire original image(203). Operation and structure information obtained with respect to a previous image is updated according to operation and structure information obtained through the optical flow obtained with respect to a current image, and 3D structure and operation information of an object is extracted(204,205). A successive image is 3-dimensionally synthesized with a different image according to 3D structure and operation information of the object(206).

Description

{Wethod for synthesizing three-dimensional image}

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method of synthesizing a three-dimensional image, and more particularly, to a method of synthesizing a three-dimensional image by using an optical flow and an extended Kalman filter or a numerical method for a monocular image sequence. And a method of synthesizing a three-dimensional image by updating the information to synthesize the three-dimensional image.

In general, computer graphics plays an important role in the production of movies and television. Authors want to create more free images, so they use computer graphics images to synthesize actors and subjects freely within the image. However, in order to synthesize, it is necessary to take an actor or a subject and separate it from the background. For this purpose, the chroma-key technique has been used most frequently.

1, first, the subject 102 is photographed in the background of a single-color blue screen 101, and information about the pan, tilt, and zoom of the camera . The photographed image generally includes a distinctive object from a single background. The method of separating the subject from the background is to consider the three primary colors R, G, and B. The color range corresponding to the background color The subject is separated from the input image by presence or absence of a color signal. Generally, considering the color value of the photographed image, the blue background is assigned a value of 0 and the subject is assigned a value of 1 to separate the subject from the background. A new synthesized image can be obtained by synthesizing the image of the separated object with the other background image 106 already having in consideration of the operation information 107 of the camera.

The conventional method using the chroma key, which is a method of synthesizing the image, uses the sensor to estimate the motion of the subject, so that only the motion of the subject can be measured and the structure of the pixel can not be extracted in the image. Since the three-dimensional structure of the subject can not be known, it is not possible to synthesize a three-dimensional image considering the depth between the subject and the background. Since scenes at other viewpoints other than the photographed scene must be separately photographed, It is not possible to reproduce the same arbitrary image.

The object of the present invention is to automatically perform a series of steps of shooting a subject in front of a background of a single color and then segmenting the subject and synthesizing the subject again with another image, Dimensional image by using an image to extract a three-dimensional structure, thereby obtaining a three-dimensional model, synthesizing an image based on computer graphics technology, and generating an image at an arbitrary viewpoint.

1 is a flowchart of image synthesis using a conventional chroma-key.

2 is a flowchart of a video synthesis method according to an embodiment of the present invention.

FIG. 3 illustrates a forward wavelet transform.

Figure 4 shows an inverse wavelet transform.

Figure 5 shows a system based on an Extended Kalman Filter.

According to an aspect of the present invention, there is provided an image synthesizing method of photographing a subject on a screen background and synthesizing the same with other images according to operation information of the subject, the method comprising: (1) obtaining an optical flow of the entire captured image And (2) extracts the three-dimensional structure and motion information of the subject by updating the motion and structure information obtained for the previous image according to the motion and structure information obtained through the optical flow obtained for the current image with respect to the continuous image And (3) three-dimensionally synthesizing the three-dimensional structure and operation information of the object with other images.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings.

FIG. 2 schematically shows an overall flow of an embodiment of the present invention. The optical flow of a photographed image is obtained, a motion of the subject in the three-dimensional space and a three-dimensional structure of the subject itself are extracted, It is shown that an arbitrary composite image is created by compositing with another object image or background image.

First, the subject is photographed in the background of a single color screen or a screen having a texture capable of extracting singularity (201). In order to maintain compatibility with existing systems, the present invention can use an image capturing system such as a general TV or a movie camera.

Generally, in order to extract a three-dimensional structure of a subject existing in an image, a method of obtaining a three-dimensional structure from a photographed image using a stereo camera, a method of using a feature point when a monocular camera is used, There are various methods such as a method using a flow.

Next, the optical flow of the entire captured image 203 is described. In the present invention, an optical flow is used to extract motion and three-dimensional structure while using a monocular camera. The theoretical background for obtaining the optical flow is as follows. The position of a pixel in an image, Let's call it time. first Time constant is a continuous space-time intensity distribution, the following Equation 1 is established, and this relation is referred to as an optical flow condition (Optical Flow Constraint).

[Equation 1]

The following equation (2) can be obtained by using the chain law of differentials.

&Quot; (2) "

In Equation (2) ego to be.

However, in the above Equation 2, and There is only one relationship, so there is no unique solution. Therefore, other conditions (constraints) are needed to solve this problem.

As another condition above, it is assumed that the second differential relationship as shown in the following Equation 3 is satisfied.

&Quot; (3) "

Then, a solution such as the following Equation 4 can be obtained.

&Quot; (4) "

Next, it is assumed that the motion around a certain pixel does not change. Then you can use Is defined as the following equation (5) .

&Quot; (5) "

The above error To and The following relational expression (6) can be obtained.

&Quot; (6) "

Solving the above equation yields the following equation (7).

&Quot; (7) "

In Equation (7)

ego,

to be.

Next, an operation field in which the variation between pixels is minimum is found. First, the error of the optical flow condition is defined as the following equation (8).

&Quot; (8) "

Then, the error of the inter-pixel variation of the velocity vector is defined as the following equation (9).

&Quot; (9) "

At this time, the solution to the error of the optical flow condition and the variation of the velocity vector given by the following equation (10) is minimized.

&Quot; (10) "

Generally, this can be solved using the calculus of variation and the solution can be obtained in an iterative manner.

In order to obtain an optical flow, an optical flow constraint is generally assumed, and a neighboring sampling method or a second differential method or a Horn and Schunk method or a block operation model is used. It is assumed that the neighborhood will have the same optical flow in one pixel to obtain the optical flow in this assumption. In other words, Is a velocity vector satisfying the optical flow condition as the following Equation (11)

&Quot; (11) "

To Centered around A spatiotemporal window including a number of pixels satisfies the following expression (12). &Quot; (12) "

&Quot; (12) "

Therefore, by estimating the above relational expression using least square estimation, the following equation (13) can be obtained.

&Quot; (13) "

Minimizing (13) The following equation (14) can be obtained.

&Quot; (14) "

In Equation (14) Is given by the following equation (15).

&Quot; (15) "

Using this optical flow, the result obtained is given in pixel units, the motion of the subject can be extracted from the obtained result, and the relationship between the optical flow and the motion and structure of the object can be obtained. From the optical flow, Can be obtained. On the other hand, since the optical flow assumes smoothness, segmentation between the subjects is difficult, and since the relational expression for obtaining the optical flow uses primary or secondary differentials, it is susceptible to noise and occlusion occurs. In order to solve this problem, a multi-resolution method is used (202) because it takes a long time to obtain an optical flow. By using the multi-resolution method, a low-pass filtered image can be obtained. As a result, noise can be reduced, and a resolution of a required image can be obtained, so that an image can be used in accordance with the required precision.

In general, a method for obtaining multiple resolutions includes a subband method, a wavelet transform method, a Gaussian or laplacian pyramid method, and the like. It is preferable to use wavelet transform in the present invention. A flow chart of the wavelet transformation is given in Fig. 3 and Fig. The theory is described in detail in "Dense Structure From A Dense Optical Flow Sequence", Yalin Xiong and Steven A. Shaffer, International Symposium on Computer Vision, Coral Gables, FL, As shown in FIG. 3, the calculation process of the optical flow using the multi-resolution is performed by wavelet-transforming the original image to obtain the multi-resolution image and obtaining the optical flow from the n-th resolution image.

By obtaining the optical flow as described above, the resolution image can be selected in accordance with the required time, the optical flow can be obtained, and the noise-free optical flow can be obtained.

In general, And translation operation And rotation motion (Refer to Refinement of Optical Flow Estimation and Detection of Motion Edges, Andrea Giachetti and Vincent Torre, Proc. ECCV'96, Cambridge, UK, April 1996).

&Quot; (16) "

A schematic diagram of the system using the above equation and the Extended Kalman Filter (EKF) is given in FIG. Before deriving the extended Kalman filter using the above equation, it is necessary to define variables and functions as shown in Equation 17 below.

&Quot; (17) "

, The above equation can be expressed by the following equation (18).

&Quot; (18) "

In order to use the extended Kalman filter, three steps are required as shown in FIG.

The first step is a step 501 of obtaining an initial operation. In this step, the initial motion information on the current image frame is estimated using the current optical flow information 500 and the estimated structure information.

First, we add a depth map And the current operation And velocity vector Depth map that can be calculated by The initial operation can be estimated by obtaining a value that minimizes the relational expression as shown in the following Equation 19. " (19) "

&Quot; (19) "

In the above relation The Depth map with matrix The structure uncertainty of The And the depth uncertainty caused by the current flow uncertainty caused by the flow uncertainty caused by the flow.

The second process is an update process 502 by an extended Kalman filter. In this process, the following structure and operation information are obtained by utilizing current flow information, previous structure information and initial motion estimation value. The structure is represented by the preceding coordinates. The simplest derivation is as follows.

first Is the number of pixels of an image, Is the depth in each pixel, the state vector at that time is given by the following equation (20).

&Quot; (20) "

And the measurement shows the flow velocity of each pixel as follows. Lt; / RTI >

In the above equation Is the measured flow rate at each pixel. Then, , The error covariance of Of the diagonal block matrix.

In the above equation Of the flow velocity in each pixel Error covariance.

And the Jacobian matrix of the measurement equations is given by

In this third step 503, a prior depth map and structural information from the coordinates are moved to posterior coordinates through interpolation, rotation, and translation. .

As described above, in the present invention, the two-dimensional projection information of the three-dimensional operation information of the object is obtained by utilizing the pixel information of the image. The obtained information is called an optical flow, and the motion of the object in the image and the three-dimensional structure can be obtained. However, since the optical flow obtained between the two images can not obtain sufficient data, a complete three-dimensional structure is obtained by obtaining the optical flow using the continuous image and updating the information using the extended Kalman filter. By using the 3D information of the object finally obtained and other image data already possessed, it is possible to synthesize real images.

In the step of photographing a subject, an image is taken using a stereo camera instead of a monocular camera, depth is calculated using triangulation, and the processes described above are applied to only one image Obtain depth and motion. Then, the depth information can be updated by adding the two depth data.

The present invention relates to a method and apparatus for acquiring a three-dimensional composite image and applying it to the field of image synthesis such as virtual space synthesis such as virtual studio and virtual reality requiring a three-dimensional image, This method has the effect of reducing the cost of image processing and obtaining more free image synthesis than the conventional method through synthesis between real images and real images and computer graphics.

Claims (4)

An image synthesizing method for photographing a subject on a screen background and compositing the same with another image in accordance with operation information of the subject, (a) generating a multi-resolution image of an image of a photographed subject; (b) obtaining an optical flow of the lower resolution image from the multi-resolution image first, obtaining an optical flow of the upper resolution image according to the information, and obtaining an optical flow of the entire original image; (c) extracting the three-dimensional structure and motion information of the subject by updating the motion and structure information obtained for the previous image according to the motion and structure information obtained through the optical flow obtained for the current image, ; And (d) synthesizing three-dimensional images with other images according to the three-dimensional structure and operation information of the subject The method of claim 1, wherein the step (a) Wherein a plurality of resolutions are obtained using a wavelet transform method, a subband transform method, or a Gaussian or Laplacian pyramid method. The method of claim 1, wherein the step (b) Wherein the optical flow is obtained by using a neighboring sampling method or a second differential method or a Horn and Schunk method or a block operation model. The method of claim 1, wherein the step (c) Characterized in that an extended Kalman filter or a general Kalman filter is used.