KR100924716B1 - Method for Rendering Virtual View for 2D/3D Free View Video Generation - Google Patents

Abstract

Disclosed is a 2D / 3D virtual view synthesis method for free view image reproduction. The disclosed method includes the steps of (a) acquiring N images corresponding to N viewpoints; Estimating the disparity with respect to selected reference images of the N images; Warping the variation related information of the reference image with respect to images other than the reference image among N images; Estimating a variation for images other than a reference image by using the warped variation related information. According to the disclosed method, there is an advantage of minimizing operational duplication occurring during virtual view image synthesis.

Video, composite, variation

Description

Method for Rendering Virtual View for 2D / 3D Free View Video Generation}

The present invention relates to a 2D / 3D virtual view synthesis method for reproducing a free view image, and more particularly, to a method capable of reducing the amount of computation during virtual view image synthesis and providing an image according to a free view even for a 3D image. It is about.

Due to the rapid development of the multimedia processing field, 3D TV is expected to become one of the products attracting attention in the next generation broadcasting system market. In order to commercialize three-dimensional TV, technologies for acquiring and processing multi-view images, compressing and transmitting huge amounts of data must be developed.

Virtual view synthesis is one of the important technologies in various image fields as well as 3D TV. Many methods have been proposed based on image-based rendering.

Virtual collimation synthesis is a technology that provides a variety of viewpoints with only a limited number of camera images, which is very useful for reducing data volume and cost in developing 3D TV systems.

As a technique for synthesizing virtual viewpoints, a method of reproducing an intermediate viewpoint from a stereo image has been conventionally proposed. The intermediate viewpoint was generally synthesized using a weighted average according to the image quality of the projected image.

Meanwhile, for virtual view synthesis, disparity information needs to be obtained with respect to given reference images.

For example, when reference images of N viewpoints are given, variation of N images should be estimated to obtain depth information of each image.

Such a variation estimation method for synthesizing a virtual viewpoint has a problem that requires a large amount of computation because a variation estimation has to be performed for a plurality of reference images.

In addition, conventionally, many methods have been proposed for a method of synthesizing a 2D image corresponding to a designated virtual view, but there is a problem in that a 3D image cannot be provided as a free view.

The present invention proposes a 2D / 3D virtual view synthesis method for reproducing a free view image for minimizing operational duplication occurring during the synthesis of the virtual view image.

Another object of the present invention is to propose a 2D / 3D virtual view synthesis method for reproducing a free view image, which does not need to obtain depth information from all N view images when an N view image is given when synthesizing a virtual view image.

Another object of the present invention is to propose a 3D virtual view synthesis method that can freely select and play a view even for a 3D image.

In order to achieve the above object, according to an aspect of the present invention, a method for estimating the variance for image synthesis in a virtual view, the method comprising the steps of: (a) obtaining N images corresponding to N views; Estimating the disparity with respect to selected reference images of the N images; Warping the variation related information of the reference image with respect to images other than the reference image among N images; A method for estimating a disparity at the time of synthesizing a virtual view image, comprising the step (d) of estimating the disparity for images other than the reference image using the warped disparity related information.

(B) estimating the variation of the reference image comprises: calculating a cost per pixel function of the reference image; Setting an energy function for the cost per pixel function; Calculating an optimal cost function that minimizes the energy function; And estimating the variation by an optimization process using the optimal cost function.

Images other than the reference image are divided into target images and semi-target images.

The semi-target image is the leftmost image and the rightmost image among the N images, and the reference image is selected not to be adjacent to each other among the images except the semi-target image.

The variation related information may include an optimal cost function of the reference image.

The front and rear warping of the optimal cost function of the adjacent reference images are performed on the target image, and the front or rear warping of the optimal cost function of the adjacent reference image is performed on the target image.

(E) detecting a hidden pixel of the first reference image and a hidden pixel of the target image when warping the optimal cost function of the reference image to the target image; Setting (f) a preset visibility function value for occluded pixels of a target image; (G) warping an optimal cost function of the first reference image using the visibility function value; And repeating steps (e) to (f) with respect to the second reference image and the target image.

The pixel that is not matched when the pixel of the first reference image and the target image is matched is set as a hidden pixel of the target image.

When there is a pixel that overlaps the pixel matching between the first reference image and the target image, when there is a pixel that satisfies both the first condition based on the variation and the second condition based on the optimal cost function, the pixel is selected as the first pixel. If a pixel that satisfies both the first condition and the second condition does not exist, both the pixel of the target image that is overlapped and the pixel of the target image that do not match are masked. Set to candidate pixels.

The visibility function is set to a value of '1' or '0', '0' is assigned to the masked pixel of the target image and the masked candidate pixel of the target image, so that the optimal cost of the first reference image is expressed by the following equation. A function is warped forward to the target image.

Where E is the optimal cost function, d is the displacement, I is the x-coordinate of the first reference image, and O is the visibility function.

Detecting a masked pixel of a reference image adjacent to the semi-target image and a masked pixel of the target image when warping the optimal cost function of the reference image to the semi-target image; Setting a preset visibility function value for occluded pixels of the semi-target image; And setting an optimal cost function of the occluded pixels of the semi-target image by the following equation using the visibility function value.

Wherein O is a visibility function, E is an optimal cost function, w is a weight function, and λ is a weight constant.

In the case of pixel matching between the semi-target image and the reference image adjacent to the semi-target image, pixels that are not matched are set as hidden pixels of the semi-target image.

When there is a pixel where overlapping matching occurs in pixel matching between the semi-target image and the reference image adjacent to the semi-target image, when there is a pixel that satisfies both the first condition based on the variation and the second condition based on the optimal cost function If the corresponding pixel is set as a masked pixel of the reference image, and there is no pixel satisfying both the first condition and the second condition, the pixel of the semi-target image that is overlapped and the semi-target image that do not match Set all of the pixels to be hidden candidate pixels.

According to another aspect of the present invention, the method includes: obtaining N images corresponding to N viewpoints (a); Estimating the disparity with respect to selected reference images of the N images; Warping the variation related information of the reference image with respect to images other than the reference image among N images; Estimating a disparity for images other than a reference image using the warped disparity related information; (E) projecting all points in the real image into 3D space by using the estimated disparity information of the N viewpoint images; Converting the coordinates of the real image into arbitrary virtual viewpoint image coordinates (f); And (g) projecting three-dimensional coordinates onto a two-dimensional plane of the virtual viewpoint image.

According to another aspect of the invention, the method comprising the steps of (a) obtaining N images corresponding to N viewpoints; Estimating disparity information for the N images; Synthesizing a two-dimensional image corresponding to at least two or more viewpoints associated with the arbitrary virtual viewpoint using the disparity information of the acquired image adjacent to the arbitrary virtual viewpoint when an arbitrary virtual viewpoint is designated; And synthesizing a three-dimensional image by using the at least two two-dimensional images.

According to the present invention, there is an advantage of minimizing the operational duplication that occurs when synthesizing a virtual viewpoint image.

In addition, according to the present invention, when the N-view image is given when synthesizing the virtual view image, it is not necessary to obtain depth information from all N view images.

In addition, according to the present invention, there is an advantage that a viewpoint can be freely selected and reproduced even for a 3D image.

Hereinafter, with reference to the accompanying drawings will be described in detail a preferred embodiment of the 2D / 3D virtual view synthesis method for free view image playback according to the present invention.

1 is a view showing the overall flow of a virtual view synthesis method for free view image playback according to an embodiment of the present invention.

Referring to FIG. 1, N view image data photographed from N cameras is acquired (step 100).

When the N-view image is obtained, the depth information of the N images is calculated by estimating the variation of the N images (step 102).

Conventionally, when N N view image data is acquired, the depth of N images is calculated by estimating the variation of the N image data one by one for image synthesis of the virtual view. Such a conventional method is called an N-view & N-depth method.

 However, in the embodiment of the present invention, the depth information is calculated only for M (M <N) preset images among N images without calculating depth information for N images, and for the remaining (NM) images. Use depth warping to obtain depth information.

Such a method proposed in the present invention will be represented by a semi-N-view & N-depth method, and a method of obtaining depth information by the semi-N-view & N-depth method will be described in detail with reference to a separate drawing. Shall be.

When depth information of N viewpoint images is obtained by a semi-N-view & N-depth method, image synthesis for an arbitrary viewpoint is performed.

If an arbitrary viewpoint is set, all points in the real image are back-projected in 3D space using the disparity information of predetermined viewpoint images associated with the arbitrary viewpoint (step 104).

After reverse projection is performed on the 3D image, the coordinates of the actual image are converted into coordinates corresponding to an arbitrary viewpoint (step 106).

When the coordinate transformation is completed, the 3D coordinates are projected onto the 2D plane of the virtual image (step 108).

Detailed methods of the above-described step 104 to step 106 will be described in detail with reference to the separate drawings.

In the above, the general procedure for synthesizing a 2D virtual view for playing a free view image according to an embodiment of the present invention has been described. Hereinafter, the suggestions of the present invention will be described in detail.

I. Semi N-Point & N-Depth Method

Prior to describing the semi-N-view & N-depth scheme of the present invention, the conventional N-view & N-depth scheme will be described.

2 is a view for explaining a conventional N-view & N-depth structure.

According to the conventional method, when images of N viewpoints are given, N depth maps have to be acquired to synthesize virtual viewpoints. The N-view & N-depth method proposed by Zitnick estimates N depth maps and synthesizes intermediate viewpoints through disparity estimation for N images.

As shown in FIG. 2, when seven viewpoint images are given, seven disparity estimation processes should be performed using all seven viewpoint images as reference images.

Since the variation estimation process requires a great amount of computation, it is generally performed in advance offline. The biggest reason for the high complexity of the estimation of the variation in the conventional method is that the variation estimation is performed in the same manner for all images.

Although the seven viewpoint images shown in FIG. 2 are mostly overlapping images except for some hidden regions, unnecessary computational overlap is generated by performing the same disparity estimation for each of the seven images.

This embodiment proposes a semi-N-view & N-depth method that can minimize such computational overlap and obtain depth information from an N-view image more quickly.

The proposed semi N-view & N-depth scheme is based on the fact that the depth information in adjacent images does not differ except for the hidden region.

3 is a diagram illustrating two images acquired from different adjacent viewpoints and a depth map of each image.

3 shows an image pair in which break dancers are viewed at different points of view. Referring to FIG. 3, it can be seen that the depth maps of the two images are similar and there is a difference in only the hidden region.

4 illustrates a semi-N-view & N-depth structure according to an embodiment of the present invention.

According to a preferred embodiment of the present invention, when an image of N viewpoints is given, N images are divided into three types. Some images of the N images are classified as reference images, some images are classified as target images, and other images are classified as semi-target images.

Referring to FIG. 4, when seven viewpoint images are given, second, fourth, and sixth images are set as reference images. Also, the first and last images are set as semi target images. The remaining image is set as the target image.

According to a preferred embodiment of the present invention, the depth information of the N images is calculated only for the reference images. Depth information of the target image and the semi-target image is calculated by warping information associated with the variation information of the reference image, and depth information is not independently calculated.

The information associated with the variation information of the warped reference image may be optimal cost function information of the reference image.

When a resource required to calculate depth information of one image is C, the N-view & N-depth method as shown in FIG. 2 requires a calculation amount of 7C. However, in the case of the semi-N-view & N-depth method as shown in FIG. 4, since the depth information is calculated only for three images, an operation amount of 3C + a is required.

When an image of N viewpoints is given, a reference image, a target image, and a semi-target image may be set as follows.

When an image of N viewpoints is given, images corresponding to left and right ends are set as semi-target images.

In addition, the reference image may be selected from some of the images other than the semi-target image, and not selected to be adjacent to each other. The target image is set to images except for the semi-target image and the reference image.

In FIG. 4, images 1 and 7 corresponding to the left and right ends of the image are set as semi-target images, and images 2, 4 and 6, which are not adjacent to each other, are set as reference images. Images 3 and 5 except for the semi-target image are set as target images.

5 is a diagram illustrating an example of setting a reference image, a target image, and a semi-target image when N is an even number according to an embodiment of the present invention.

Referring to FIG. 5, a case in which eight images are given is illustrated, and images 1 and 8 corresponding to left and right ends are set as semi-target images. FIG. 5 illustrates a case where images 2, 5, and 7 of images 2 through 7 are set as reference images. Unlike FIG. 5, images 2, 4, and 7 may be set as reference images.

Given N images, the number of semi-target images, the number of reference images and the number of target images may be generalized as shown in Equation 1 below.

However, Equation 1 is merely an embodiment for generalizing the number of semi-target images, reference images, and target images, and it will be apparent to those skilled in the art that the number of images may be set differently as necessary.

If N = 4

4 and 5, asymmetric warping is performed in which only one of front warping and rear warping is performed on the semi-target image. Meanwhile, the target image is symmetrically warped using the front and rear reference images.

6 is a flowchart illustrating a process of estimating the variation using the semi-N-view & N-depth method according to an embodiment of the present invention.

Referring to FIG. 6, an optimal cost function is calculated for reference images (step 600). The optimal cost function is first calculated for the estimation of the variance for the reference images. As mentioned above, the optimal cost function may be warped to the target image or the semi target image.

When the optimal cost function is calculated for the reference images, the variation of the reference images is estimated through an optimization process using the same (step 602). Various methods such as Winner-Takes All (WTA) and graph cuts can be used as optimization methods, and the final variation is estimated through optimization.

When the variance estimation for the reference images is completed, symmetrical warping or asymmetrical warping for the target image and the semi-target images is performed to warp the optimal cost function of the reference image (step 604).

The variance for the target image and the semi target image is estimated using the warped optimal cost function (step 606). In the estimation of the variation of the target image and the semi-target image, pixel processing is required for the masked region that does not exist in the reference image. Pixel processing of the masked region will be described in detail with reference to the accompanying drawings.

II. Calculate Optimal Cost Function of Reference Image

In general, at least two or more images are used to estimate the variation. When the i-1, i, i + 1th images are called left, center, and right images, the difference image of the center image is calculated by moving the left (right) image to the right (left) and obtaining a difference.

The difference image e _{i , j} (p, d) of the i-th image is calculated as Equation 2 below as the i, j-th image.

In Equation 2, p and d represent the 2D position and variation of the pixel, I represents the pixel value, and T represents the upper limit of the matching cost. The cost-per-pixel function ei (p, d) is calculated by taking into account the visibility of the point of the center image using e _{i , i + 1} and e _{i , i- 1} .

When the cost per pixel function is defined as e (p, d) and the optimal cost function is defined as E (p, d), the cost per pixel function and the optimal cost function have the following relationship: .

In Equation 3 above, n means noise. In the following, E (p, d) is expressed as E (p), because the same procedure is performed for all the variations.

The method of obtaining the cost function is based on the cost function calculation method described in Korean Patent Application No. 2007-0136756 filed by the inventor.

We set the energy function for the calculation of the optimal cost function and compute the optimal cost function that minimizes the energy function.

Equation 4 below is an equation representing an energy function used in the present invention.

In Equation 4 above, w is a weighting function and weights a pixel value difference from neighboring pixels. w causes a smaller weight to be assigned as the difference between the pixel values between p and p + n is greater, and a larger weight to be assigned as the difference between the pixel values becomes smaller. In other words, weights are inversely proportional to the difference in pixel values.

In addition, a cost function difference value between the pixel and the neighboring pixel is calculated, and the cost function difference value is multiplied by the aforementioned weight.

은 서로 직교하는 2차원 벡터이다. M은 에너지 함수에서 고려되는 이웃 픽셀 집합의 사이즈를 나타낸다.In Equation 3 above, n and

Are two-dimensional vectors orthogonal to each other. M represents the size of the neighboring pixel set considered in the energy function.

 The optimal cost function is a value that minimizes an energy function as shown in Equation 4, and the following Equation 5 can be obtained by firstly differenting Equation 4 to obtain a cost function.

To simplify the above equation, we redefine the set of neighboring pixels. When p is (x, y), a set of neighboring pixels can be expressed as in Equation 6 below.

Using Equation 6 above, Equation 5 may be expressed as Equation 7 below.

An iteration may be used to obtain the optimal cost function E (p) from Equation 7, and the solution of the (k + 1) th iteration is calculated as in Equation 8 below.

Equation 8 includes a normalized cost function per pixel and a cost function obtained by weighted averages of costs of neighboring pixels. By performing an iterative calculation, the cost function E (p) is normalized by the cost function of neighboring points.

According to an embodiment of the present invention, an asymmetric Gaussian weight function using CIE-Lab may be used. r _c and r _s are weight constants for the color and spatial distance, respectively. If C _i is the color distance calculated in the i-th image, the weight function may be expressed as in Equation 9 below.

A Gaussian-sider acceleration scheme is proposed in this embodiment to accelerate the computation of the cost function by equation (8).

The biggest problem of slowing down the calculation speed in the iteration calculation by Equation 8 is that the information already calculated is used only after one iteration is completed.

In order to solve this problem, N (p), which is a set of neighboring pixels, is classified by Nc (p), which is a causal part, and Nn (p), which is an in causal part, and calculated by Equation 10 below. Can be accelerated.

To compute a more reliable optimal cost function, it is necessary to gather information from farther pixels. This means that a significant number of iterations must be performed to estimate the correct optimal cost function.

In order to reduce the number of iterative calculations, a multiresolution approach is proposed along with a Gaussian-sider acceleration.

The multiresolution approach is to perform the same operation at a smaller resolution than the resolution of the original image, and to use the result of the calculation at a higher resolution. For convenience of explanation, hereinafter, the resolution of the original image is referred to as a first resolution, and the resolution corresponding to 1/2 of the first resolution is referred to as the second resolution, and the resolution corresponding to 1/2 as the second resolution is referred to as the third resolution. Let it be.

7 illustrates a frame structure of a multi-resolution approach according to an embodiment of the present invention.

In Equation 8, in general, the optimal cost function E (p) can be initialized to the cost function e (p) per pixel. This embodiment proposes a multiresolution approach that utilizes the final computed value at lower resolutions as an initial value at higher resolutions, which can accelerate convergence in iterative computations.

Referring to FIG. 7, the cost function e ₂ (d) per pixel is calculated at the third resolution, which is 1/4 of the resolution of the original image, and the calculated e ₂ (d) is used as an initial value to calculate the cost at the third resolution. The function E ₂ (d) is calculated.

The calculated E ₂ (d) at the third resolution is utilized as the initial value at the second resolution. In the second resolution, the cost function E ₁ (d) is calculated by using the final value at the third resolution as an initial value, and the calculated E ₁ (d) is used as the initial value at the first resolution.

That is, the final value calculated at the low resolution of the Nth resolution is used as the initial value of the higher resolution of the (N-1) th resolution. This approach can greatly reduce the amount of computation at the final original resolution.

The final calculated cost function at low resolution is upsampled and used as a high resolution initial value, which requires interpolation to minimize errors in the boundary region during upsampling.

It is preferable to perform adaptive interpolation according to Equation 11 below, rather than linear interpolation that propagates an error in the boundary region during upsampling.

In Equation 11, w denotes a weighting function and λa is a weighting factor and may be set to 15, for example.

III. Handle to pixels in obscured areas

As described above, the optimal cost function computed for the reference image is warped to the target image and the semi target image. In the target image, since the symmetric warping is performed using two reference images, the hidden region is compensated and there are only a few holes.

However, in the case of the semi-target image, since the asymmetric warping is performed in which only one of the front warping and the rear warping is performed, processing of the hidden region is required.

For example, in the case of the image at the leftmost point of view, since only rear warping is performed, an area covered by the left portion of the image occurs. In this embodiment, a method for allocating an optimal cost function to the hidden region of the semi-target image is proposed.

First, the masked region processing for the target image is described. For the target image, both front warping and rear warping are performed. First, front warping will be described.

First, a function St (j) for a target image composed of points of a reference image is defined. St (j) is defined as in Equation 12 below.

i and j represent x-coordinates of the reference image and the target image, and W represents the horizontal size of the image.

8 and 9 are diagrams illustrating a case where an area that is hidden occurs when forward warping is performed between a reference image and a target image.

Referring to FIG. 8, the fourth to seventh pixels of the reference image are overlapped with the pixels of the target image. Two pixels are pixels of an image existing in both the reference image and the target image, and two pixels are pixels only in the reference image. On the other hand, the pixels of the reference image do not match to the eighth and ninth pixels of the target image, which are pixels of the image existing only in the target image.

In FIG. 8, the fourth and fifth pixels marked with a dotted line actually exist only in the reference image, and the hidden pixels of the reference image have a smaller variation than the pixels that are not normally masked as shown in FIG. 8.

Meanwhile, referring to FIG. 9, the second to fifth pixels of the reference image are overlapped with the pixels of the target image. In addition, pixels of the reference image do not match to the eighth and ninth pixels of the target image, which are pixels of the image existing only in the target image.

In Fig. 9, the fourth and fifth pixels indicated by the matching relationship with dotted lines are pixels that are actually present only in the reference image, and in this case, the hidden pixels have a larger variation than the pixels that are not.

A first condition based on the variation and a second condition based on the optimal cost function are used to determine whether the pixel is hidden or unhidden in the reference image.

The first condition utilizes the feature that the shifted pixels are smaller in variation than the unhidden pixels, and the second condition utilizes the feature that the cost is greater than the hidden pixels.

If there is a pixel that satisfies both the first condition and the second condition among the overlapped matching pixels, it is determined as the hidden pixel of the reference image, and only the cost function of the unobscured pixel is warped.

However, when there are no pixels satisfying both the first condition and the second condition among the pixels that are overlapped, it may not be determined which pixel information of the reference image should be warped.

In this case, both the non-matching pixel and the overlapping matching pixel in the target image are set as hidden candidate pixels.

For example, in FIG. 9, the second and third pixels and the eighth and ninth pixels of the target image are both set as hidden candidate pixels.

In the target image, a visibility function Ot (j) is set for each pixel, and the visibility function has a value of '0' or '1'. The value of the visibility function of the hidden pixel and the hidden candidate pixel of the target image may be set to '0', and the value of the visibility function of the other pixels may be set to '1'.

When the value of the visibility function is set for each pixel in the target image, the cost function of the reference image is warped as shown in Equation 13 using the visibility function in the target image.

That is, only the optimal cost function of the pixels present in the target image is warped from the reference image to the target image.

In Equation 13, the y coordinate is omitted because the same process is repeated for each line. Back warping is also performed in a similar manner to front warping, and in back warping, St (j) is defined as in Equation 14 below.

Back warping may be performed using Equation 15 below using the visibility function Ot (j) of the target image.

When both front and rear warping are performed on the target image, the front and rear warping may be overlapped with the pixels of the target image. In this case, it is preferable to select a warping having a small optimal cost function for the corresponding variation. .

On the other hand, in the case of the semi-target image, since only one of the front warping and the rear warping is performed, the hidden area must be processed. In addition, in the case of the target image, holes may occur after the front and rear warping, and processing for these holes is necessary.

The masked region processing of the semi-target image and the hole processing of the target image are performed using the following equation (16) using the semi-target image or the visibility function Ot of the target image.

IV. Synthesis of Virtual Viewpoint Image

In the present embodiment, texture information of the real image is mapped to the virtual image, and this process is performed for all real images. 10 is a diagram illustrating a process of synthesizing a virtual viewpoint image.

Referring to FIG. 10, a process of synthesizing a virtual viewpoint image through (1) reverse projection, (2) coordinate transformation of a camera, and (3) projection is illustrated.

FIG. 11 is a diagram illustrating a movement of a virtual camera for synthesizing a virtual viewpoint image at the virtual viewpoint a.

The rotation of the virtual camera may generate many holes in the synthesized image, so the parallel movement is considered in this embodiment.

The two closest images (i th and (i + 1) th) for a given virtual viewpoint are selected and projected onto the virtual viewpoint. The point mi (x _i , y _i ) having the variation di in the i-th image is converted into a point M _i in 3D space as shown in Equation 17 below.

(x ₀ , y ₀ ) means the base point of the image. By converting Mi into coordinates of the virtual camera using parallel movements (Tx, Ty, and Tz), and projecting onto the image surface of the virtual camera, the relationship between the corresponding points in the real and virtual images can be calculated.

는 다음의 수학식 18과 같이 계산된다. Virtual point

Is calculated as in Equation 18 below.

To simplify the equation, normalized coordinates (a _x , a _y , a _z ) = (Tx, Ty, Tz) / B are used, and the distance between cameras can be defined as 1. When and are regarded as actual and projected images, the following relation is established.

When forward mapping is performed to transfer texture information from an actual image to a virtual image, some problems may occur. In the case of equation (x), one-to-one mapping is not performed, and multiple projections or holes occur at the virtual viewpoint. The reason for multiple projections at the virtual point of view is depth discontinuity and image resampling.

Points that exist at depth discontinuities are projected to the same point in the virtual view even though they have different variations. In this case, only the point with the largest shift among the projected points is preserved because the point close to the camera wraps the remaining points.

는 정수 단위가 아닐 수도 있다. 이러한 문제점을 해결하기 위해 가상 시점 합성 시 가상 영상에서 실사 영상으로 투영하는 역방향 맵핑과 선형 보간법이 이용될 수 있다. Another problem is caused by the resampling of the image. When the size of the object changes during the virtual view reproduction, multiple projections and holes may occur even for points having the same variation. In particular, the point of the virtual view

May not be an integer unit. In order to solve this problem, reverse mapping and linear interpolation, which project from a virtual image to a real image, may be used in synthesizing a virtual view.

Reverse mapping prevents degradation of image quality that may occur in image resampling. The virtual camera moves along the x and z axes, including left, right, front and back. Since the hole can be generated at the virtual point of view, the movement on the y axis is preferably limited.

는 가상 카메라의 실제 위치를 의미한다. V(p)를 가상 시점의 점이 실제 영상에서 보이는지 여부를 판별하는 가시 함수라고 할 때, 가상 시점은 투영된 영상의 내삽을 통해 다음의 수학식 20과 같이 연산된다.

Means the actual position of the virtual camera. When V (p) is a visual function for determining whether a point of the virtual view is visible in the real image, the virtual view is calculated as shown in Equation 20 through interpolation of the projected image.

V. Generation of 3D Virtual View Image

The present invention proposes a method of generating a 3D image corresponding to a virtual viewpoint. According to a preferred embodiment of the present invention, when a virtual viewpoint is given, a two-dimensional image of two or more viewpoints associated with the virtual viewpoint may be synthesized to generate a three-dimensional image corresponding to the virtual viewpoint.

FIG. 11 illustrates a case where a 3D image corresponding to a virtual viewpoint is generated using two virtual viewpoint images a and b together with the movement of the camera for the virtual viewpoint image at the viewpoint a.

In FIG. 11, a is a virtual viewpoint designated by the user. When the virtual view is designated, the 2D image at the specified view and the 2D image at b, which is a view moved by a predetermined distance from the specified view, are synthesized.

At this time, the distance between the two viewpoints is shown as Bs, the length of Bs can be selected by those skilled in the art.

That is, a first two-dimensional image is synthesized at a designated virtual viewpoint a, and a view b second two-dimensional image moved by Bs from the viewpoint a is synthesized. By synthesizing the 3D image using the 2D image as the stereo image, the 3D image may be synthesized with respect to the virtual viewpoint.

FIG. 11 illustrates a case in which a 3D image is synthesized using a 2D image at a virtual view designated by a user and a 2D image at a time shifted by a predetermined distance from the designated virtual view, but it may be modified in various ways. Do.

For example, the 3D image may be synthesized by synthesizing the 2D image from the viewpoint moved to the right of the predetermined distance and the view moved to the left of the predetermined distance from the virtual viewpoint designated by the user.

As another example, three-dimensional images may be obtained by acquiring a larger number of two-dimensional images rather than two two-dimensional images.

That is, according to the spirit of the present invention, a 3D image may be synthesized by obtaining a 2D image from a plurality of viewpoints associated with a designated virtual view.

According to an embodiment of the present invention, a shift sensor scheme may be used for high quality 3D image synthesis. 12 is a diagram illustrating an example of a shift sensor method.

In order to do zero parallax setting (ZPS), the sensor of the virtual camera is moved by h parallel to the position of the lens. This allows the convergence distance Zc to be selected when the three-dimensional scene is reproduced on the display device. The shifting of the sensor is calculated through the movement of the camera's base point. When the horizontal movement of the base point is defined as h, the point of the virtual view point is calculated by the following equation (21). Here, the moved right and left images.

1 is a view showing the overall flow of a virtual view synthesis method for free view image playback according to an embodiment of the present invention.

2 is a view for explaining a conventional N-view & N-depth structure.

3 is a diagram illustrating two images acquired at different adjacent viewpoints and a depth map of each image.

4 illustrates a semi-N-view & N-depth structure in accordance with an embodiment of the present invention.

5 is a diagram illustrating an example of setting a reference image, a target image, and a semi-target image when N is an even number according to an embodiment of the present invention.

FIG. 6 is a flowchart illustrating a process of estimating a variation using a semi-N-view & N-depth method according to an embodiment of the present invention. FIG.

7 illustrates a frame structure of a multi-resolution approach according to an embodiment of the present invention.

8 and 9 are diagrams illustrating a case where an area that is hidden occurs when forward warping is performed between a reference image and a target image.

10 is a diagram illustrating a process of synthesizing a virtual viewpoint image.

11 is a diagram illustrating a movement of a virtual camera for synthesizing a virtual viewpoint image at the virtual viewpoint a.

Claims (26)

A variation estimation method for image synthesis at a virtual viewpoint, Acquiring N images corresponding to N viewpoints (a); Estimating the disparity with respect to selected reference images of the N images; Warping the variation related information of the reference image with respect to images other than the reference image among N images; And estimating a disparity for images other than a reference image by using the warped disparity-related information. The method of claim 1, Estimating the variation of the reference image (b), Calculating a cost per pixel function of the reference image; Setting an energy function using the cost per pixel function; Computing an optimal cost function that minimizes the energy function; And And estimating a disparity by an optimization process using the optimal cost function. According to claim 1, And an image other than the reference image is divided into a target image and a semi-target image. The method of claim 3, The semi-target image is the leftmost image and the rightmost image among N images, and the reference image is selected so as not to be adjacent to each other among images except the semi-target image. Way. The method of claim 3, And the disparity related information is a function of an optimal cost of the reference image. The method of claim 5, The front and rear warping of the optimal cost function of adjacent reference images is performed on the target image, and the front or rear warping of the optimal cost function of the adjacent reference image is performed on the target image. Disparity estimation method for image synthesis from virtual viewpoint. The method of claim 3, When warping the optimal cost function of the reference image to the target image, (E) detecting the hidden pixels of the first reference image and the hidden pixels of the target image; Setting (f) a preset visibility function value for occluded pixels of a target image; (G) warping an optimal cost function of the first reference image using the visibility function value; And And repeating steps (e) to (f) with respect to the second reference image and the target image. The method of claim 7, wherein The method of estimating disparity for image synthesis at the virtual view, characterized in that for matching the pixel between the first reference image and the target image, a pixel that is not matched is set as a hidden pixel of the target image. The method of claim 8, When there is a pixel that overlaps the pixel matching between the first reference image and the target image, when there is a pixel that satisfies both the first condition based on the variation and the second condition based on the optimal cost function, the pixel is selected as the first pixel. Set to the hidden pixels of the reference image. If there are no pixels satisfying both the first condition and the second condition, both the pixels of the target image which are matched in duplicate and the pixels of the target image which do not match are set as the candidate candidate masks. Disparity estimation method for image synthesis from virtual viewpoint. The method of claim 9, The visibility function is set to a value of '1' or '0', '0' is assigned to the masked pixel of the target image and the masked candidate pixel of the target image, so that the following reference equation And an optimal cost function is forward warped to the target image.

In the above equation, E is the optimal cost function, d is the displacement, i is the x-coordinate of the first reference image, O is the visibility function, subscript t means the target shape, and subscript r indicates the reference image. Meaning. The method of claim 3, When warping the optimal cost function of the reference image to the semi-target image, Detecting a masked pixel of the reference image adjacent to the semi-target image and a masked pixel of the target image; Setting a preset visibility function value for occluded pixels of the semi-target image; And And setting an optimal cost function of the occluded pixels of the semi-target image by the following equation using the visibility function value.

In the above equation, O is a visibility function, E is an optimal cost function, w is a weight function, λ is a weight constant, p is a pixel coordinate, m is a temporary variable set for sigma operation, N (p) Is the set of neighboring pixels, N _c (p) is the causal part of the set of neighboring pixels, N _n (p) is the non-causal part of the set of neighboring pixels, and the superscript k + 1 means k + 1 th in the iterative operation And the superscript k means the kth in the iterative operation, and the subscript t means the semi-target image. The method of claim 11, The method of estimating disparity for image synthesis in a virtual view according to claim 11, wherein the pixel that is not matched when the pixel is matched between the semi-target image and the reference image adjacent to the semi-target image is set as a hidden pixel of the semi-target image. The method of claim 12, When there is a pixel where overlapping matching occurs when pixel matching between the semi-target image and the reference image adjacent to the semi-target image, when there is a pixel that satisfies both the first condition based on the variation and the second condition based on the optimal cost function Set that pixel as the hidden pixel of the base image. If there are no pixels satisfying both the first condition and the second condition, all of the pixels of the semi-target image that are matched repeatedly and the pixels of the semi-target image that do not match are set as the candidate candidate masks. Disparity estimation method for image synthesis in a virtual view. Acquiring N images corresponding to N viewpoints (a); Estimating the disparity with respect to selected reference images of the N images; Warping the variation related information of the reference image with respect to images other than the reference image among N images; Estimating a disparity for images other than a reference image using the warped disparity related information; (E) projecting all points in the real image into 3D space by using the estimated disparity information of the N viewpoint images; Converting the coordinates of the real image into arbitrary virtual viewpoint image coordinates (f); And And (g) projecting three-dimensional coordinates onto a two-dimensional plane of the virtual viewpoint image. The method of claim 14, Images other than the reference image are divided into target images and semi-target images. The method of claim 15, The semi-target image is the leftmost image and the rightmost image among the N images, and the reference image is selected so as not to be adjacent to each other among the images except the semi-target image. The method of claim 16, And the variation related information is a function of an optimal cost of the reference image. The method of claim 17, The front and rear warping of the optimal cost function of adjacent reference images is performed on the target image, and the front or rear warping of the optimal cost function of the adjacent reference image is performed on the target image. Image Synthesis in Virtual View. Acquiring N images corresponding to N viewpoints (a); Estimating disparity information for the N images; If an arbitrary virtual viewpoint is specified, a first two-dimensional image corresponding to a first viewpoint associated with the arbitrary virtual viewpoint and a second viewpoint different from the first viewpoint and corresponding to the second virtual viewpoint (C) synthesizing a two-dimensional 2D image using the obtained disparity information of the image; Comprising a step (d) of synthesizing a three-dimensional image by using the first two-dimensional image and the second two-dimensional image, The first viewpoint and the second viewpoint associated with the arbitrary virtual viewpoint are views of the arbitrary virtual viewpoint or a predetermined distance shifted from the arbitrary virtual viewpoint. delete The method of claim 19, At least two or more viewpoints associated with the arbitrary virtual viewpoint are: And a viewpoint moved to the left of the predetermined distance from an arbitrary virtual viewpoint and a viewpoint moved to the right of the predetermined distance from the arbitrary virtual viewpoint. The method of claim 19, Step (b) is, Estimating the variation with respect to the selected partial reference images of the N images; Warping the variation-related information of the reference image with respect to images other than the reference image among N images; Estimating variation for images other than a reference image by using the warped variation-related information. The method of claim 22, And images other than the reference image are divided into target images and semi-target images. The method of claim 23, wherein The semi-target image is the leftmost image and the rightmost image among N images, and the reference image is selected so as not to be adjacent to each other among the images except the semi-target image. The method of claim 24, And the variation related information is a function of an optimal cost of the reference image. The method of claim 25, The front and rear warping of the optimal cost function of adjacent reference images is performed on the target image, and the front or rear warping of the optimal cost function of the adjacent reference image is performed on the target image. 3D image synthesis method in virtual view.

Images