KR20180005859A - Apparatus and method for generating virtual viewpoint image - Google Patents

Abstract

An apparatus and method for generating a virtual viewpoint image are disclosed. The apparatus for generating a virtual viewpoint image according to the present invention comprises: a segmented image generating unit generating a standard disparity image and a reference disparity image by segmenting each of a standard image and a reference image corresponding to different viewpoints by a unit of segmentation; a disparity calculating unit calculating a disparity corresponding to the standard image and reference image segmented by the unit of segmentation to generate the standard disparity image and reference disparity image; an image mapping unit mapping the standard disparity image and reference disparity image to a virtual viewpoint using the disparity; and a virtual viewpoint generating unit generating a virtual viewpoint image corresponding to the virtual viewpoint by mixing a first mapping image obtained by mapping the standard disparity image to the virtual viewpoint and a second mapping image obtained by mapping the reference disparity image to the virtual viewpoint.

Description

[0001] APPARATUS AND METHOD FOR GENERATING VIRTUAL VIEWPOINT IMAGE [0002]

The present invention relates to a technique for generating a virtual viewpoint image, and more particularly, to a technique for generating an image for a virtual viewpoint using a geometric relationship between a reference image and a reference image corresponding to different viewpoints.

The virtual viewpoint image refers to an image corresponding to a virtual viewpoint not photographed by the photographing apparatus, and can be generated using an existing image photographed by the photographing apparatus.

Basically, the virtual viewpoint image is mapped to the position of the target virtual viewpoint using the geometric information of the reference image acquired by the photographing apparatus, and the occluded area that is not mapped is mapped through another reference image . Then, interpolation is performed using the similar information of the surroundings, and finally a virtual viewpoint image corresponding to the position of the virtual viewpoint is generated. Here, the geometry information of the reference image may include photographing apparatus calibration information, depth and variation information of individual pixels, and the like.

In the prior art, it is assumed that the reference images have the same size, depth information indicating the geometric information between the reference images, and the mutated images are mapped on a one-by-one basis.

If the depth information or the variation information of one pixel is not correct due to mapping on a pixel basis, a lot of noise is generated on a pixel basis. In general, it is difficult to obtain accurate results when using the stereo matching method to obtain correspondence based on the similarity of all pixels of two images.

Also, it takes a long time to obtain the depth information and the variation information of all the pixels on a pixel-by-pixel basis, and to generate an image using the information. Therefore, when a virtual viewpoint image is generated in real time, a problem arises due to a trade-off between quality and computation amount.

That is, it is necessary to develop a technique that can reduce the time required for generating a virtual viewpoint image by studying a technique other than the conventional pixel-by-pixel mapping method.

Korean Patent Publication No. 10-2014-0022300, published on February 24, 2014 (name: method and apparatus for generating multi-view image)

It is an object of the present invention to provide a virtual viewpoint image corresponding to a point in time when a shot is not taken.

It is also an object of the present invention to reduce the amount of calculation as compared with a conventional technique of mapping in pixel units.

It is also an object of the present invention to provide a robust virtual viewpoint image by grouping and processing adjacent pixels having local characteristics.

It is another object of the present invention to prevent blocking phenomenon that may occur when mapping is performed on a block-by-block basis, and to generate a natural virtual viewpoint image.

According to an aspect of the present invention, there is provided an apparatus for generating a virtual viewpoint image, the apparatus comprising: a divided image generating unit for dividing each of a reference image and a reference image corresponding to different viewpoints in units of division; A disparity calculation unit for calculating a disparity corresponding to the reference image to generate a reference disparity image and a reference disparity image, an image mapping unit for mapping the reference disparity image and the reference disparity image to a virtual point in time using the disparity, And generating a virtual viewpoint image corresponding to the virtual viewpoint by mixing a first mapping image in which the reference mutation image is mapped to the virtual viewpoint and a second mapping image in which the reference mutation image is mapped to the virtual viewpoint, And an image generating unit.

At this time, the image mapping unit may perform backward mapping based on the reference mutation image and the reference mutation image.

At this time, the image mapping unit may map a block unit larger than or equal to the division unit.

At this time, the image mapping unit may perform mapping by assigning mapping weights according to the distance between the center point of the reference mutation image and the reference mutation image.

At this time, the image mapping unit may perform mapping using at least two images among the reference image, the reference mutation image, the reference image, and the reference mutation image.

In this case, when the mapping is performed using three images among the reference image, the reference mutation image, the reference image, and the reference mutation image, the image mapping unit assigns a mixed weight to each of the reference image and the reference image So that the mapping can be performed.

The mixed weights may be calculated using at least one of a distance between the reference image and the virtual viewpoint, and a distance between the reference image and the virtual viewpoint.

In this case, the virtual viewpoint image generator may generate the virtual viewpoint image by assigning a composite weight to each of the first and second mapped images.

Here, the composite weight may be set using at least one of a distance between the reference image and the virtual viewpoint, and a distance between the reference image and the virtual viewpoint.

The composite weight may be set using an error value between the pixel value of the reference image mapped to the same pixel and the pixel value of the reference image.

In this case, the image interpolation unit may interpolate the empty region of the virtual viewpoint image.

At this time, the image interpolator can interpolate the empty area using pixel values of an adjacent area based on at least one of a spatial axis and a time axis corresponding to the empty area.

At this time, the virtual viewpoint is located between the viewpoint of the reference image and the viewpoint of the reference image, and may be a value set by the user or a predetermined value.

In this case, the divided image generating unit may perform at least one of a regular image segmentation and an irregular image segmentation to divide the reference image and the reference image.

According to another aspect of the present invention, there is provided a virtual viewpoint image generating method performed by a virtual viewpoint image generating apparatus, comprising: dividing each of a reference image and a reference image corresponding to different viewpoints in units of division; Calculating a variation corresponding to the reference image and the reference image to generate a reference variation image and a reference variation image; mapping the reference variation image and the reference variation image to a virtual time point using the variation; And generating a virtual viewpoint image corresponding to the virtual viewpoint by mixing a first mapping image in which the reference mutation image is mapped to the virtual viewpoint and a second mapping image in which the reference mutation image is mapped to the virtual viewpoint, .

At this time, mapping to the virtual viewpoint may perform backward mapping based on the reference mutation image and the reference mutation image.

In this case, the mapping to the virtual viewpoint may be performed on a block basis, which is larger than or equal to the division unit.

In this case, the mapping to the virtual viewpoint may be performed by assigning mapping weights according to the distance between the center point and the pixel of the reference mutation image and the reference mutation image.

At this time, mapping to the virtual viewpoint may perform mapping using at least two images among the reference image, the reference mutation image, the reference image, and the reference mutation image.

In this case, when the mapping is performed using the three images of the reference image, the reference mutation image, the reference image, and the reference mutation image, mapping to the virtual viewpoint may be performed on each of the reference image and the reference image, The mapping can be performed by applying a mixed weight.

The mixed weights may be calculated using at least one of a distance between the reference image and the virtual viewpoint, and a distance between the reference image and the virtual viewpoint.

The generating of the virtual viewpoint image may generate the virtual viewpoint image by assigning a composite weight to each of the first mapping image and the second mapping image.

Here, the composite weight may be set using at least one of a distance between the reference image and the virtual viewpoint, and a distance between the reference image and the virtual viewpoint.

The composite weight may be set using an error value between the pixel value of the reference image mapped to the same pixel and the pixel value of the reference image.

The method may further include interpolating a blank region of the virtual viewpoint image.

The interpolation of the free area may interpolate the free area using pixel values of the adjacent area based on at least one of a spatial axis and a time axis corresponding to the free area.

At this time, the virtual viewpoint is located between the viewpoint of the reference image and the viewpoint of the reference image, and may be a value set by the user or a predetermined value.

In this case, the dividing step may divide the reference image and the reference image by performing at least one of a regular image division and an irregular image division.

According to the present invention, it is possible to provide a virtual viewpoint image corresponding to a point in time when the photographing is not performed.

Further, according to the present invention, it is possible to reduce the amount of calculation as compared with the conventional technique of mapping in pixel units.

According to the present invention, a robust virtual viewpoint image can be generated by grouping and processing adjacent pixels having local characteristics.

Also, according to the present invention, it is possible to prevent blocking phenomenon that may occur when mapping is performed on a block-by-block basis, and to generate a natural virtual viewpoint image.

1 is a block diagram illustrating a configuration of a virtual viewpoint image generating apparatus according to an embodiment of the present invention.
2 is a flowchart illustrating a method of generating a virtual viewpoint image according to an embodiment of the present invention.
3 is a view for explaining the division of a reference image and a reference image according to an embodiment of the present invention.
4 is a view for explaining a virtual viewpoint according to an embodiment of the present invention.
5 is a diagram illustrating a process of mapping a disparity image to a virtual point of view in step S230 of FIG.
6 is a diagram for explaining a mapping weight according to an embodiment of the present invention.
7 is a diagram for explaining a process of generating a virtual viewpoint image in step S240 of FIG.
8 is a block diagram illustrating a computer system in accordance with an embodiment of the present invention.

The present invention will now be described in detail with reference to the accompanying drawings. Hereinafter, a repeated description, a known function that may obscure the gist of the present invention, and a detailed description of the configuration will be omitted. Embodiments of the present invention are provided to more fully describe the present invention to those skilled in the art. Accordingly, the shapes and sizes of the elements in the drawings and the like can be exaggerated for clarity.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating a configuration of a virtual viewpoint image generating apparatus according to an embodiment of the present invention.

1, the virtual viewpoint image generation apparatus 100 includes a partitioned image generation unit 110, a variation calculation unit 120, an image mapping unit 130, a virtual viewpoint image generation unit 140, As shown in FIG.

First, the divided image generating unit 110 divides each of reference images and reference images corresponding to different viewpoints into segment units.

At this time, the viewpoint of the reference image and the viewpoint of the reference image are different from each other. The time corresponding to the virtual viewpoint image to be generated by the virtual viewpoint image generating apparatus 100 is located between the reference image and the reference image.

In addition, the divided reference image is generated by dividing the reference image by the divided image generating unit 110, and the divided reference image is generated by the divided image generating unit 110 by dividing the reference image into divided units will be.

The divided image generating unit 110 may perform at least one of a regular image segmentation and an irregular image segmentation to divide a reference image and a reference image. Particularly, the divided image generation unit 110 may perform fixed image segmentation for dividing the reference image and the reference image at regular intervals.

In addition, the divided image generating unit 110 may divide the reference image and the reference image into a variable by a meaningful unit using an image segmentation technique. The divided image generating unit 110 may perform an irregular image segmentation to divide the reference image and the reference image into units of a similar unit.

Next, the variation calculation unit 120 calculates a variation corresponding to the divided reference image and the divided reference image, and generates a reference variation image and a reference variation image.

The variation calculation unit 120 calculates a variation corresponding to a reference image divided into segment units and a divided reference image. Here, the disparity means a correspondence relationship between a reference image, which is a video shot at different viewpoints of the same scene, and the same point included in the reference image, and a point included in the reference image and a point included in the reference image The difference may be a value expressed in pixels.

For example, when the variation of a specific pixel included in the reference image is 10, the correspondence corresponding to the specific pixel on the reference image is shifted left or right by 10 on the basis of the coordinates of the specific pixel in the reference image It is located at the place where it moved.

That is, when the reference image is set as the left image and the reference image is set as the right image, if the variation of a specific pixel included in the reference image is 10, the corresponding point of the specific pixel in the reference image is It is located on the left side by 10 in reference.

In general, stereo matching is used to process stereo images such as left and right images. The stereo matching technique is a method of selecting pixels having the highest similarity by comparing the pixels of the reference image and the pixels of the candidate group of the reference image.

However, the variation calculation unit 120 of the virtual viewpoint image generation apparatus 100 according to an embodiment of the present invention estimates the representative variation by comparing the characteristics of the pixels included in the reference image with candidate groups of the reference image.

In addition, the variation calculation unit 120 generates the reference variation image and the reference variation image using the calculated variation.

The image mapping unit 130 maps the reference mutation image and the reference mutation image to the virtual time point using the mutation.

Here, the virtual viewpoint is located between the viewpoint of the reference image and the viewpoint of the reference image, and may be a value set by the user or a predetermined value.

When the left image is a reference image and the right image is a reference image, the reference image may be set to 0 and the reference image may be set to 1. The virtual point of view, which is an intermediate point between the reference image and the reference image, can be expressed as a relative position based on the reference image and the reference image.

At this time, one or more positions of the virtual viewpoint corresponding to the virtual viewpoint image to be generated can be set. The virtual time point may be a predetermined value or a value input from the user.

In particular, the virtual point of view can be set to 0.5, which means the center of the reference image and the center of the reference image. When a plurality of virtual viewpoints are set, a plurality of virtual viewpoints may be set so that a time interval of a reference image, a plurality of virtual viewpoints, and an interval between viewpoints of a reference image are uniform.

Also, the image mapping unit 130 performs a backward mapping based on the reference mutation image and the reference mutation image.

The image mapping unit 130 may generate a first mapping image by mapping the reference mutation image to a virtual viewpoint in a backward direction, and generate a second mapping image by mapping the reference mutation image to a virtual viewpoint in a reverse direction.

The image mapping unit 130 may map a block unit larger than or equal to the division unit. For example, if the size of the division unit is 4x4, the image mapping unit 130 may map the 4x4 adjacent pixels by performing mapping to 8x8 size, which is a block unit larger than 4x4. Accordingly, the image mapping unit 130 solves the blocking phenomenon and can synthesize images naturally.

The image mapping unit 130 may generate the first mapping image using the reference image and the reference variation image, or may generate the first mapping image using the reference image, the reference variation image, and the reference image. Similarly, when generating the second mapping image, the image mapping unit 130 generates a second mapping image using the reference image and the reference variation image, or generates a second mapping image using the reference mapping image, Can be generated.

When the image mapping unit 130 generates the first and second mapping images using three images, the image mapping unit 130 assigns mixed weights to the reference image and the reference image to generate a first mapping image and a second mapping image, A second mapping image can be generated.

Here, the mixed weight may be set using at least one of the distance between the reference image and the virtual viewpoint, and the distance between the reference image and the virtual viewpoint. That is, when the distance difference of the viewpoint is smaller, the image mapping unit 130 may generate a mapping image by giving a larger weight value.

The image mapping unit 130 may perform mapping by assigning mapping weights according to the distance between the center point of the reference mutation image and the reference mutation image and the pixel distance.

Here, the mapping weight is a weight according to a distance between a center point of a reference variation image and a center point of a reference variation image and a pixel to be mapped. For example, when the reference transition image is 4x4, the four pixels closest to the center point of the reference transition image give the first mapping weight. The second mapping weight is assigned to 12 pixels having a distance of 2 pixels from the center point of the reference transition image. At this time, the first mapping weight has a larger value than the second mapping weight.

Next, the virtual viewpoint image generator 140 combines the first mapping image in which the reference mutation image is mapped to the virtual viewpoint and the second mapping image in which the reference mutation image is mapped to the virtual viewpoint, .

The virtual viewpoint image generator 140 may generate a virtual viewpoint image by assigning composite weights to the first and second mapping images, respectively. Here, the composite weight may be calculated using at least one of the distance between the reference image and the virtual viewpoint, and the distance between the reference image and the virtual viewpoint.

For example, it is assumed that the reference image represents the left image, the reference image represents the right image, and the virtual viewpoint is set to 0.4. In this case, since the virtual viewpoint is closer to the reference image than the reference image, the virtual viewpoint image generator 140 sets the first composite weight given to the first mapped image to be larger than the second composite weight given to the second mapped image .

That is, when the distance difference between the viewpoints is smaller, the virtual viewpoint image generator 140 may generate a virtual viewpoint image by combining the first and second mapping images with a larger composite weight value.

The composite weight may also be a calculated error value when performing the mapping. Here, the error value may be a value set based on a difference between pixel values extracted for the same point from the reference image and the reference image when the backward mapping is performed.

For example, when a pixel value extracted from a reference image is referred to as a PL and a pixel value extracted from a reference image is referred to as a PR, a difference | PL-PR | between two pixel values may mean an error value.

When the reference deviation image is correct, the error value, which is the difference between PL and PR, is small. Therefore, the virtual viewpoint image generator 140 can give a larger composite weight value. On the other hand, when the reference image is inaccurate, the error value has a larger value, and the virtual view image generator 140 gives a smaller composite weight.

For the sake of convenience of explanation, it has been described that the composite weight is given using the reference mutation image. However, the present invention is not limited to this, and the virtual view image generator 140 may assign the composite weight using the reference mutation image, The virtual viewpoint image may be generated using both the corresponding composite weight and the composite weight corresponding to the reference disparity image.

Finally, the image interpolator 150 interpolates the holes of the generated virtual viewpoint image.

At this time, the image interpolating unit 150 can interpolate the empty area using the pixel values of the adjacent area based on at least one of the spatial axis and the time axis corresponding to the empty area.

Hereinafter, a virtual viewpoint image generation method performed by the virtual viewpoint image generation apparatus according to an embodiment of the present invention will be described in more detail with reference to FIG. 2 through FIG.

2 is a flowchart illustrating a method of generating a virtual viewpoint image according to an embodiment of the present invention.

First, the virtual viewpoint image generating apparatus 100 divides the reference image and the reference image (S210).

The virtual viewpoint image generating apparatus 100 divides the input reference image and the reference image into respective divided units. For the convenience of explanation, it is assumed that the reference image means the left image and the reference image means the right image.

3 is a view for explaining the division of a reference image and a reference image according to an embodiment of the present invention.

3, the virtual viewpoint image generating apparatus 100 may perform at least one of a regular image segmentation 310 and an irregular image segmentation 320 to divide a reference image and a reference image.

The virtual viewpoint image generating apparatus 100 may generate a divided reference image and a divided reference image having a uniform size as shown in FIG. 3 by performing a uniform image segmentation 310 for dividing the reference image and the reference image at regular intervals have.

For example, it is assumed that the virtual viewpoint image generation apparatus 100 sets a 4x4 division unit of 4 pixels wide by 4 pixels wide. In this case, when the size of the reference image and the reference image is FHD (1920x1080), the reference image and the reference image can be divided into 480 divided images and 270 divided images.

In addition, the virtual viewpoint image generating apparatus 100 can perform the atypical division of the reference image and the reference image into meaningful units by using an image segmentation technique. The virtual viewpoint image generating apparatus 100 may generate reference images and reference images that are grouped in similar units.

In general, the image segmentation technique can be performed by repeating a process of grouping peripheral pixels having high similarity with a reference pixel into one group. When the virtual viewpoint image generation apparatus 100 performs the atypical segmentation using the image segmentation technique, the grouped images may be generated among the pixels having the degree of similarity equal to or higher than the threshold, such as the atypical image segmentation 320 of FIG.

Next, the virtual viewpoint image generation apparatus 100 calculates the variation of the disparity image, and generates the reference disparity image and the reference disparity image (S220).

The virtual viewpoint image generating apparatus 100 calculates a disparity representing a correspondence between a reference image and a reference image, which are images captured from different viewpoints of the same scene.

At this time, the virtual viewpoint image generating apparatus 100 searches for points included in the reference image and the reference image, which are images captured at different viewpoints, from the same scene. The virtual viewpoint image generating apparatus 100 may calculate a corresponding relationship between pixels corresponding to points included in the common viewpoint and calculate a variation by expressing the distance between the corresponding pixel of the reference image and the corresponding pixel in pixel units.

For example, it is assumed that the left image is the reference image, the right image is the reference image, and the variation of one pixel included in the reference variation image is 10. At this time, the pixel A 'in the reference image corresponding to the pixel A in the reference image is located at a position shifted leftward by 10 from the position of the pixel A.

Next, the virtual viewpoint image generation apparatus 100 maps the mutated image to a virtual viewpoint (S230).

4 is a view for explaining a virtual viewpoint according to an embodiment of the present invention.

4, the virtual viewpoint image V _i is located between the viewpoint of the reference image 410 and the reference image 420, and the virtual viewpoint image generating apparatus 100 generates a virtual viewpoint V _i can be set, or the virtual time point V _i can be set using a preset value.

The virtual viewpoint V _i may have a value between 0 and 1 when the viewpoint of the reference image 410 is 0 and the viewpoint of the reference image 420 is 1. The virtual point of view V _i may refer to a relative position of the reference image 410 and the reference image 420.

When the virtual viewpoint image generation apparatus 100 desires to generate a virtual viewpoint image corresponding to one virtual viewpoint, the virtual viewpoint image generation apparatus 100 may set 0.5 to the virtual viewpoint V _i . In addition, when the virtual viewpoint image generating apparatus 100 desires to set a plurality of virtual viewpoints, the virtual viewpoint image generating apparatus 100 may set the viewpoint of the reference image, the plurality of virtual viewpoints, A plurality of virtual viewpoints may be set.

The virtual viewpoint image generation apparatus 100 performs a backward mapping based on the reference variation image D _L 415 and the reference variation image D _R 425.

5 is a diagram illustrating a process of mapping a disparity image to a virtual point of view in step S230 of FIG.

Is referred to as shown in Fig. 5, the left image is the reference image (I _L) 510 means, and the reference image (I _L) based on a disparity image of 510 disparity image (D _L), (515) a. At this time, it is assumed that the reference image 510 is divided into a 4x4 size division unit 540. And the right image means a reference image (I _R ) 520 and the virtual time point V _0.4 is 0.4.

5, when the reference image (I _L ) 510 is uniformly divided into 4 × 4 division units 540, a reference variation image D _L ) 515 is generated. That is, the transition value of one pixel of the reference transition image (D _L ) 515 may correspond to the variation of the block of the 4x4 size of the reference image (I _L )

Assuming that the deviation value D _L (x) of the point x of the reference variation image (D _L ) 515 is d _L for convenience, d _L is a pixel of the point x of the reference image (I _L ) And a point x 'of the reference image I _R 520 corresponding to the reference image I _R. That is, when d _L is 10, the point x 'of the reference image (I _R ) 520 corresponds to the pixel shifted by 10 in the left direction in the pixel of the point x of the reference image (I _L )

만큼 이동한 곳에 위치한다. Assuming that the viewpoint of the reference image I _L 510 is 0 and the viewpoint of the reference image I _R 520 is 1, the virtual viewpoint V _0.4 is the reference image I _L 510, To the reference image (I _R ) 520

As shown in FIG.

만큼 떨어진 곳에 위치한 픽셀을 기준 영상(I_L)(510)으로부터 가지고 오는 역방향 매핑(후방사상)을 수행한다. Accordingly, the virtual visual point image generating device 100 in the rightward direction when creating a virtual viewpoint image 530, the reference image (I _L) (510) at the position of the virtual viewpoint image is desired to produce

(Backward mapping), which takes a pixel located at a distance from the reference image (I _L )

만큼 떨어진 곳에 위치한 픽셀을 참조 영상(I_R)(520)으로부터 가지고 오는 역방향 매핑을 수행한다. . Since the variation value means the relationship between the reference image I _L 510 and the reference image I _R 520 in the same way as the reference image I _R 520)

(I _R ) 520 from the reference image (I _R ) .

The virtual viewpoint image generation apparatus 100 generates a virtual viewpoint image using at least two of the reference image I _L 510, the reference variation image D _L 515 and the reference image I _R 520, (I _L ) < RTI ID = 0.0 > 510 < / RTI >

When the virtual viewpoint image generation apparatus 100 performs the backward mapping using the reference image I _L 510 and the disparity image I _R 520, the reference image I _L 510 ) And the reference image (I _R ) 520, respectively, so as to perform mapping. At this time, the disparity images 515 and 525 may be either the reference disparity image (D _L ) 515 or the reference disparity image.

Here, the mixed weight may be set using at least one of the distance between the reference image and the virtual viewpoint, and the distance between the reference image and the virtual viewpoint. That is, when the distance difference of the viewpoint is smaller, the virtual viewpoint image generating apparatus 100 can generate a mapping image by assigning a larger mixture weight value.

The virtual viewpoint image generating apparatus 100 assigns different mixed weights to the pixels taken from the reference image I _L 510 and the pixels taken from the reference image I _R 520 as shown in Equation 1 .

은 기준 영상(

)으로부터 가지고 온 픽셀 값을 의미하고,

는 참조 영상(

)으로부터 가지고 온 픽셀 값을 의미하며, 0.6은

에 부여되는 제1 혼합 가중치를 의미하고, 0.4는

에 부여되는 제2 혼합 가중치를 의미하며,

는 매핑 가중치를 의미한다. here,

The reference image (

), ≪ / RTI >

The reference image (

), And 0.6 means

, And 0.4 represents a first mixed weight

Lt; RTI ID = 0.0 > a < / RTI > second mixed weights,

Means a mapping weight.

The virtual viewpoint image generating apparatus 100 may assign a first mixed weight to a pixel value obtained from the reference image I _L 510 and calculate a pixel value obtained from the reference image I _R 520, The second mixed weights are added and combined. Then, the virtual viewpoint image generating apparatus 100 assigns a mapping weight to the summed value to determine a final pixel value corresponding to the position of the virtual viewpoint image.

Herein, the first and second weight values are calculated based on the difference between the time of the reference image (I _L ) 510 and the time of the virtual image (V _0.4 ), the time of the reference image (I _R ) _0.4 ). ≪ / RTI > For example, when the viewpoint of the reference image I _L 510 is 0, the virtual viewpoint V _0.4 is 0.4, and the viewpoint of the reference image I _R 520 is 1, (100) may set the first mixing weight to 0.6 and the second mixing weight to 0.4. The virtual viewpoint image generating apparatus 100 can obtain a larger weight value of the reference image I _L 510 and the reference image I _R 520 from the virtual viewpoint V _0.4 .

The first mixed weight and the second mixed weight are set such that the time of the reference image (I _L ) 510 and the time of the virtual viewpoint (V _0.4 ) and the difference between the time of the reference image (I _R ) 520 and the virtual time point (V _0.4 ).

6 is a diagram for explaining a mapping weight according to an embodiment of the present invention.

)는 매핑을 수행하는 변이 영상의 중심점으로부터 해당 픽셀의 거리에 따라 설정된 값을 의미할 수 있다. 이때, 매핑 가중치는 해당 픽셀과 중심점의 거리가 가까울수록 더 큰 값으로 설정될 수 있다. As shown in Fig. 6, the mapping weight (

) May mean a value set according to the distance of the corresponding pixel from the center point of the shifted image performing the mapping. At this time, the mapping weight can be set to a larger value as the distance between the pixel and the center point becomes closer.

The virtual viewpoint image generating apparatus 100 according to an exemplary embodiment of the present invention may generate a more natural virtual viewpoint image by performing mapping by assigning a larger weight to pixels having a distance to the center point of the disparity image.

Since the mutation value of one pixel of the mutation image corresponds to the mutation of the 4x4 block of the reference image, if the reverse mapping is performed with reference to one mutation value, the reference image may be a block having a minimum size of 4x4. At this time, when mapping is performed in units of blocks, blocking phenomenon may occur. Therefore, the virtual viewpoint image generating apparatus 100 can perform block mapping on a block basis, which is larger than or equal to the size of the divided unit, thereby preventing a blocking phenomenon and generating a natural virtual viewpoint image.

The virtual viewpoint image generating apparatus 100 can map the adjacent blocks to be overlapped with each other by mapping the blocks in unit of block equal to or larger than 4x4 as the division unit. 4, when the size of the block unit 550 is set to 8x8, the virtual viewpoint image generation apparatus 100 can generate a block unit 550 of 8x8 size when the original 4x4 size division unit is mapped You can map to adjacent pixels at once by mapping them together.

)를 각각의 픽셀에 부여하여 매핑을 수행할 수도 있다. At this time, as shown in FIG. 6, the virtual viewpoint image generating apparatus 100 generates a mapping weight (for example,

) To each pixel to perform mapping.

In addition, the virtual viewpoint image generating apparatus 100 generates a virtual viewpoint image (S240).

The virtual viewpoint image generating apparatus 100 blends the first mapping image in which the reference mutation image is mapped to the virtual viewpoint and the second mapping image in which the reference mutation image is mapped to the virtual viewpoint, And generates an image.

7 is a diagram for explaining a process of generating a virtual viewpoint image in step S240 of FIG.

As shown in FIG. 7, the virtual viewpoint image generation apparatus 100 generates a virtual viewpoint image 750 by combining the first mapping image 730 and the second mapping image 740. The first mapping image 730 is generated as a result of mapping using the reference image 710 and the reference mutation image 715. The second mapping image 740 is generated by mapping the reference image 720 and the reference mutation image 715. [ 725).

In addition, the virtual viewpoint image generating apparatus 100 may generate a virtual viewpoint image 750 by giving composite weights to the first mapping image 730 and the second mapping image 740, respectively.

은 제1 매핑 영상을 의미하고,

은 제2 매핑 영상을 의미하며,

는 제1 매핑 영상에 부여되는 제1 합성 가중치를 의미하고,

는 제2 매핑 영상에 부여되는 제2 합성 가중치를 의미하며,

는 가상 시점 영상을 의미한다. here,

Denotes a first mapping image,

Represents a second mapping image,

Denotes a first composite weight given to the first mapping image,

Denotes a second composite weight value given to the second mapping image,

Represents a virtual viewpoint image.

In this case, the first synthesis weight and the second synthesis weight may be values set reflecting the reference image, the reference image, and the virtual viewpoint, or a value reflecting the error calculated during mapping.

Here, the error value may be a value set based on the difference between the pixel value fetched from the reference image and the pixel value fetched from the reference image when the pixel value is taken from the reference image and the reference image by performing the backward mapping.

When the first synthesis weight and the second synthesis weight are set based on the time of the reference image, the time of the reference image, and the distance between the virtual viewpoints, the first synthesis weight and the second synthesis weight are respectively set to a first mixture weight and a second mixture weight And may have the same value as the weight.

Finally, the virtual viewpoint image generating apparatus 100 interpolates the virtual viewpoint image (S250).

The virtual viewpoint image generation apparatus 100 interpolates a virtual viewpoint image when a hole remains on the virtual viewpoint image. At this time, the virtual viewpoint image generating apparatus 100 can interpolate the virtual viewpoint image based on similar pixel values of the adjacent regions based on at least one of the space axis and the time axis.

8 is a block diagram illustrating a computer system in accordance with an embodiment of the present invention.

Referring to FIG. 8, embodiments of the present invention may be implemented in a computer system 800, such as a computer readable recording medium. 8, a computer system 800 includes one or more processors 810, a memory 830, a user input device 840, a user output device 850, and a storage 850, which communicate with one another via a bus 820. [ 860 < / RTI > In addition, the computer system 800 may further include a network interface 870 connected to the network 880. The processor 810 may be a central processing unit or a semiconductor device that executes the processing instructions stored in the memory 830 or the storage 860. [ Memory 830 and storage 860 may be various types of volatile or non-volatile storage media. For example, the memory may include ROM 831 or RAM 832.

Thus, embodiments of the invention may be embodied in a computer-implemented method or in a non-volatile computer readable medium having recorded thereon instructions executable by the computer. When computer readable instructions are executed by a processor, the instructions readable by the computer are capable of performing the method according to at least one aspect of the present invention.

As described above, the apparatus and method for generating a virtual viewpoint image according to the present invention are not limited to the configuration and method of the embodiments described above, but rather, All or a part of the above-described elements may be selectively combined.

100: virtual viewpoint image generating device
110: divided image generating unit 120:
130: video mapping unit 140: virtual viewpoint image generation unit
150: Image interpolator 310: Orthographic image segmentation
320: Atypical Image Segmentation 410, 510, 710: Reference Image
415, 515, 715: reference variation image
420, 520, 720: reference image
425, 725: reference mutation image
430, 530, 750: virtual viewpoint image
540: Division unit 550: Block unit
730: first mapping image 740: second mapping image
800: Computer system
810: Processor 820: Bus
830: Memory 831: ROM
832: RAM 840: User input device
850: User output device 860: Storage
870: Network interface 880: Network

Claims (20)

A divided image generating unit for dividing each of the reference image and the reference image corresponding to different viewpoints in units of division,
A disparity calculation unit for calculating a disparity corresponding to the reference image and the reference image divided by the dividing unit to generate a reference disparity image and a reference disparity image,
An image mapping unit that maps the reference variation image and the reference variation image to a virtual viewpoint using the variation;
A first mapping image in which the reference mutation image is mapped to the virtual viewpoint and a second mapping image in which the reference mutation image is mapped to the virtual viewpoint are mixed to generate a virtual viewpoint image corresponding to the virtual viewpoint, Generating unit
And a virtual viewpoint image generating device. The method according to claim 1,
The image-
Wherein the backward mapping is performed based on the reference transition image and the reference transition image. 3. The method of claim 2,
The image-
Wherein the virtual viewpoint image is mapped on a block basis, which is larger than or equal to the division unit. 3. The method of claim 2,
The image-
Wherein mapping is performed by assigning a mapping weight according to a distance between a center point of the reference variation image and the reference variation image and a pixel distance. The method according to claim 1,
The image-
Wherein the mapping is performed using at least two or more images among the reference image, the reference mutation image, the reference image, and the reference mutation image. 6. The method of claim 5,
The image-
Wherein mapping is performed by applying a mixed weight to each of the reference image and the reference image when performing mapping using three images among the reference image, the reference mutation image, the reference image, and the reference variation image, A virtual viewpoint image generation device. The method according to claim 6,
The mixed weights may be,
Wherein the virtual viewpoint image is calculated using at least one of a viewpoint of the reference image, a distance between the virtual viewpoints, and a distance between the reference viewpoint and the virtual viewpoint. 3. The method of claim 2,
The virtual viewpoint image generation unit may include:
Wherein the virtual viewpoint image generating unit generates the virtual viewpoint image by assigning a composite weight to each of the first mapping image and the second mapping image. 9. The method of claim 8,
The composite weighting value,
Wherein the virtual viewpoint image is generated using at least one of a viewpoint of the reference image, a distance between the virtual viewpoints, and a distance between the reference viewpoint and the virtual viewpoint. 9. The method of claim 8,
The composite weighting value,
Wherein an error value between the pixel value of the reference image mapped to the same pixel and the pixel value of the reference image is set. The apparatus of claim 1, further comprising an image interpolator interpolating a free region of the virtual viewpoint image. 12. The method of claim 11,
Wherein the image interpolating unit comprises:
Wherein the interpolation unit interpolates the empty area using a pixel value of an adjacent area based on at least one of a spatial axis and a time axis corresponding to the empty area. The method according to claim 1,
The virtual point-
Wherein the virtual viewpoint image is located between the viewpoint of the reference image and the viewpoint of the reference image and is set by a user or is a preset value. The method according to claim 1,
Wherein the divided image generating unit comprises:
Wherein the reference image and the reference image are divided by performing at least one of a stereoscopic image segmentation and an unstructured image segmentation. A virtual viewpoint image generation method performed by a virtual viewpoint image generation apparatus,
Dividing each of a reference image and a reference image corresponding to different viewpoints into a division unit,
Generating a reference variation image and a reference variation image by calculating a variation corresponding to the reference image and the reference image divided by the division unit,
Mapping the reference variation image and the reference variation image to a virtual viewpoint using the variation; and
Generating a virtual viewpoint image corresponding to the virtual viewpoint by mixing a first mapping image in which the reference mutation image is mapped to the virtual viewpoint and a second mapping image in which the reference mutation image is mapped to the virtual viewpoint,
And generating a virtual viewpoint image. 16. The method of claim 15,
Wherein mapping to the virtual time point comprises:
Wherein the backward mapping is performed based on the reference transition image and the reference transition image. 17. The method of claim 16,
Wherein mapping to the virtual time point comprises:
Wherein the mapping unit is mapped on a block basis, the size of which is greater than or equal to the division unit. 17. The method of claim 16,
Wherein mapping to the virtual time point comprises:
Wherein mapping is performed by assigning a mapping weight according to a distance between a center point of the reference mutation image and the reference mutation image and a distance between pixels. 16. The method of claim 15,
Wherein mapping to the virtual time point comprises:
Performing a mapping using at least two images among the reference image, the reference mutation image, the reference image, and the reference mutation image,
Wherein mapping is performed by applying a mixed weight to each of the reference image and the reference image when performing mapping using three images among the reference image, the reference mutation image, the reference image, and the reference variation image, A virtual viewpoint image generation method. 16. The method of claim 15,
Wherein the generating the virtual viewpoint image comprises:
Generating a virtual viewpoint image by assigning a composite weight to each of the first mapping image and the second mapping image,
The composite weighting value,
A reference value of the reference image, a distance between the reference point and the virtual point, a distance between the reference point of the reference image and the virtual point of view, or a pixel value of the reference image mapped to the same pixel, Wherein the error value is set using an error value between pixel values of an image.

Images