KR20100019927A - Method for generating vitual view image and apparatus thereof - Google Patents

Abstract

PURPOSE: A method for generating a virtual view image and an apparatus thereof are provided to obtain a more high definition virtual view image by compensating a non-processed region of a virtual view image in a current frame through a pixel value of a virtual view image of a previous frame. CONSTITUTION: The base view images for plural base view cameras are synthesized, and a virtual view image for the virtual view camera is created(401). The depth information images for the plural base view camera are synthesized and created(403). A first non-processed region of the non-processed region in the depth information image is compensated. Based on the depth information image, a second non-processed region which is the non-processed region of the virtual view image in the current frame is compensated(405).

Description

Method for generating virtual viewpoint image and its device {METHOD FOR GENERATING VITUAL VIEW IMAGE AND APPARATUS THEREOF}

The present invention relates to a virtual viewpoint image generation method and apparatus thereof, and more particularly, to a virtual viewpoint image generation method for compensating an unprocessed region of a virtual viewpoint image of a current frame by using a pixel value of a virtual viewpoint image of a previous frame, and a device thereof. Relates to a device. The present invention is derived from the research conducted as part of the IT source technology development project of the Ministry of Knowledge Economy [Task management number: 2008-F-011-01, Task name: Development of next-generation DTV core technology (standardized connection)-Glasses-free person Type 3D broadcasting technology development].

In recent years, as the interactive application and multi-view 3D display industry is activated, a virtual image for synthesizing a virtual image of a viewpoint which is not obtained from a real camera using a reference viewpoint camera image (stereo image or a multiview image) and a corresponding depth / shift image is used. Techniques for viewpoint image synthesis have been actively studied.

1 is a view for explaining a virtual viewpoint image.

The reference viewpoint cameras 101, 107, and 111 acquire an actual image. The virtual viewpoint cameras 103, 105, 109, 113, 115, and 117 are cameras of virtual viewpoints which do not actually capture and acquire an image. The virtual viewpoint image is an image viewed from the viewpoint of the virtual viewpoint cameras 103, 105, 109, 113, 115, and 117. The virtual view image is generated by synthesizing the real image acquired from the reference view cameras 101, 107, and 111, that is, the reference view image and the depth / difference information image using a 3D warping technique or an image interpolation technique. In this case, the depth represents the degree to which the photographed object is separated from the camera, and the disparity represents the position difference of the object photographed in each reference camera.

The viewpoints of the virtual viewpoint cameras 103, 105, and 109 on the optical axis are linearly interpolated from the reference viewpoint cameras 101, 107, and 111 and located on the optical axis, and the virtual viewpoint cameras 113, 115, and 117 outside the optical axis. The point of view is located at a point outside the optical axis by applying rotational or translational motion to the virtual viewpoint cameras 103, 105, and 109 on the optical axis. Reference viewpoint images of the reference viewpoint cameras 101, 107, and 111 may be synthesized to generate images of any one of the virtual viewpoint cameras 103, 105, 109, 113, 115, and 117, that is, images of various virtual viewpoints. .

2 and 3 are views illustrating a synthesized virtual viewpoint image, and the virtual viewpoint image of FIG. 2 is a virtual viewpoint on an optical axis obtained by synthesizing the reference viewpoint images of two reference viewpoint cameras 107 and 111 by a three-dimensional warping method. An image of the camera 109 and the virtual viewpoint image of FIG. 3 are images of the virtual viewpoint camera 115 other than the optical axis line synthesized from the reference viewpoint images of the two reference viewpoint cameras 107 and 111 by a three-dimensional warping method. For the subjective picture quality evaluation, Breaker Dancer images were used among the test images provided by the Moving Picture Expert Group (MPEG-4) video field. All of the images shown below are break dancer images provided in the MPEG-4 video field.

As illustrated in FIGS. 2 and 3, an unprocessed region shown in white exists in the virtual viewpoint image. Unprocessed areas occur due to the problem of integerization of depth / variation information, occlusion areas, and the like. The occlusion area is defined by the viewpoint 115 of the virtual viewpoint camera. It is an area that is visible from the viewpoint 115 of the virtual viewpoint camera but not visible from the viewpoint of the reference viewpoint cameras 107, 111 due to the difference in viewpoints of the reference viewpoint cameras 107, 111. More unprocessed regions exist in the virtual viewpoint image of FIG. 3, which is the virtual viewpoint image of the virtual viewpoint camera 115 outside the optical axis, than the virtual viewpoint image of FIG. 2.

Since the unprocessed region degrades the image quality of the virtual viewpoint image, the unprocessed region needs to be compensated with a predetermined pixel value. The unprocessed region due to the integerization of the depth / variation information can be compensated relatively easily and accurately by interpolating the effective pixel values around the unprocessed region. Here, the meaning of 'effective' corresponds to a processing region rather than an unprocessed region. However, when compensating the unprocessed area due to the occlusion area by interpolating effective pixel values around the unprocessed area, an error occurs in the compensating process because the occlusion area does not exist in the reference view image. Thus, in order to compensate for the unprocessed region, at least one reference viewpoint image of the reference viewpoint camera 101 is added to compensate for the unprocessed region of the virtual viewpoint image.

However, in this case, the unprocessed region of the virtual viewpoint image by the virtual viewpoint camera 109 on the optical axis may be compensated, but the unprocessed region of the virtual viewpoint image by the virtual viewpoint camera 115 outside the optical axis is difficult to compensate. If a reference viewpoint camera existing outside the optical axis is added, an unprocessed region of the virtual viewpoint image by the virtual viewpoint camera 115 outside the optical axis may be compensated, but in this case, the synthesis process of the virtual viewpoint image may be too complicated. In addition, the cost increases as the reference camera is added.

The present invention has been proposed to solve the above problems, and an object of the present invention is to provide a virtual view image generating method and apparatus for generating a higher quality virtual view image by compensating more precisely the unprocessed area of the virtual view image.

According to an aspect of the present invention, a virtual viewpoint image synthesizing step of synthesizing a reference viewpoint image for a plurality of reference viewpoint cameras to generate a virtual viewpoint image for a virtual viewpoint camera; A depth information image is synthesized by synthesizing depth information images of the plurality of reference view cameras, and generating a depth information image of the virtual view image, and compensating a first unprocessed region which is an unprocessed region of the depth information image of the virtual view image. step; And a virtual view unprocessed area compensation step of compensating a second unprocessed area that is an unprocessed area of the virtual view image of the current frame based on a depth information image of the current and at least one previous frame's virtual view image. Provides an image generation method.

In addition, the present invention for achieving the above object is a virtual viewpoint image synthesis step of generating a virtual viewpoint image for the virtual viewpoint camera by synthesizing the reference viewpoint image for a plurality of reference viewpoint camera; A disparity information image synthesizing step of synthesizing disparity information images of the plurality of reference view cameras to generate disparity information images of the virtual view image, and compensating an unprocessed region of the disparity information image for the virtual view image; And a virtual viewpoint unprocessed region compensation step of compensating an unprocessed region of the virtual viewpoint image of the current frame based on the disparity information image of the virtual viewpoint image of the current and at least one previous frame. do.

In addition, the present invention for achieving the above object is a virtual viewpoint image synthesizer for generating a virtual viewpoint image for the virtual viewpoint camera by synthesizing the reference viewpoint image for a plurality of reference viewpoint camera; A depth information image is synthesized by synthesizing depth information images of the plurality of reference view cameras, and generating a depth information image of the virtual view image, and compensating a first unprocessed region which is an unprocessed region of the depth information image of the virtual view image. part; And a virtual viewpoint unprocessed region compensator for compensating a second unprocessed region, which is an unprocessed region of the virtual viewpoint image of the current frame, based on a depth information image of the virtual view image of the current and at least one previous frame. Provide a generator.

According to the present invention, a higher quality virtual viewpoint image can be obtained by compensating for the unprocessed area of the virtual viewpoint image of the current frame using the pixel value of the virtual viewpoint image of the previous frame.

In addition, the reduction of the reference camera has an advantageous effect in terms of cost.

DETAILED DESCRIPTION Hereinafter, the most preferred embodiments of the present invention will be described with reference to the accompanying drawings so that those skilled in the art may easily implement the technical idea of the present invention.

4 is a flowchart illustrating a virtual view image generating method according to an embodiment of the present invention.

4, in accordance with the present invention The virtual viewpoint image generating method includes a virtual viewpoint image synthesis step 401, a depth information image synthesis step 403, and a virtual viewpoint unprocessed region compensation step 405. Hereinafter, a case of generating a virtual viewpoint image for the positions of the reference viewpoint cameras 107 and 111 and the virtual viewpoint camera 115 of FIG. 3 will be described as an embodiment.

In the virtual view image synthesizing step 401, reference view images of the plurality of reference view cameras 107 and 111 are synthesized to generate a virtual view image of the virtual view camera 115. The virtual viewpoint image of the virtual viewpoint camera 115 may be generated by synthesizing the reference viewpoint image and the depth information image obtained from the plurality of reference viewpoint cameras 107 and 111 by using a 3D warping technique or an image interpolation technique. have. Since the viewpoint of the virtual viewpoint camera 115 is higher than that of the reference viewpoint cameras 107 and 111, the virtual viewpoint image generated in the virtual viewpoint image synthesis step 401 may not be captured by the reference viewpoint cameras 107 and 111. There is an untreated area by the occlusion area. In addition, the virtual viewpoint image has an unprocessed area due to an error occurring during the process of integerization of depth / variation information. The unprocessed region of the virtual view image generated in the virtual view image synthesis step 401 is compensated in the virtual view unprocessed region compensation step 405 described later.

In the depth information image synthesis step 403, depth information images of the plurality of reference view cameras 107 and 111 are synthesized by 3D warping or image interpolation to generate a depth information image for the virtual view image. do. The depth information image generated in the depth information image synthesizing step 403 also includes an unprocessed region as in the virtual view image generated in the virtual view image synthesizing step 401. In the depth information image synthesis step 403, an unprocessed region of the depth information image for the virtual view image is compensated.

The images generated in the virtual view image synthesizing step 401 and the depth information image synthesizing step 403 mean each frame image which is continuous from the past to the present. A detailed description of the depth information image synthesis step 403 will be described later with reference to FIG. 6.

In the virtual view raw region compensation step 405, the raw region of the virtual view image of the current frame generated in the virtual view image synthesis step 401 is compensated. In this case, the depth information image of the virtual viewpoint image of the current and at least one previous frame compensated in the depth information image synthesis step 403 is used. In the virtual view raw region compensation step 405, the raw region of the virtual viewpoint image of the current frame is compensated by the pixel value of the virtual viewpoint image of the previous frame. In this case, it is necessary to determine whether the pixel value of the virtual viewpoint image of the previous frame is a pixel value representing the same object as the virtual viewpoint image of the current frame. ) Uses the depth information image compensated in) as the criterion.

A virtual point unprocessed region compensation step 405 will be described in more detail with reference to FIG. 5. FIG. 5 illustrates virtual view images 501 and 505 and depth information images 503 and 507 of previous and current frames generated in the virtual view image synthesis step 401 and the depth information image synthesis step 403 of FIG. 4. Figure is shown. FIG. 5 illustrates a virtual viewpoint image and a depth information image of one frame immediately before the current frame. Hereinafter, as an embodiment, the virtual viewpoint image and the depth information image of one frame immediately before the current frame are used.

In the virtual viewpoint images 501 and 505, the part shown by a red circle is an unprocessed area generated by the occlusion area. As illustrated in FIG. 5, the unprocessed region of the virtual viewpoint image 505 of the current frame may not be an unprocessed region of the virtual viewpoint image 501 of the previous frame. More specifically, the portion of the virtual viewpoint image 501 of the previous frame corresponding to the position of the pixel of the unprocessed region of the portion shown by the red circle in the virtual viewpoint image 505 of the current frame is not an unprocessed region.

Therefore, it can be seen that the unprocessed region of the virtual viewpoint image 505 of the current frame can be compensated by the virtual viewpoint image 501 of the previous frame. In this case, as described above, the object in which the pixel value to be compensated from the virtual viewpoint image 501 of the previous frame is present should be recognized as the object of the unprocessed area of the virtual viewpoint image 505 of the current frame. Depth information images 503 and 507 are used as a reference. Recognition of equality of depth values means that they are located at the same distance from the camera and that they are likely to be the same object. The larger the depth value, the closer the object is to the camera. The smaller the depth value, the farther the object is from the camera.

Meanwhile, as described above, an unprocessed region exists in the depth information images 503 and 507, and an unprocessed region of the depth information images 503 and 507 is finally compensated for in the depth information image synthesis step 403.

FIG. 6 is a detailed flowchart illustrating the depth information image synthesizing step 403 of FIG. 4.

Referring to FIG. 6, the depth information image synthesis step 403 includes a synthesis step 601, a horizontal search step 603, a horizontal compensation step 605, a vertical search step 607, and a vertical compensation step 609. . In the synthesizing step 601, the depth view images of the plurality of reference view cameras 107 and 111 are synthesized by 3D warping or image interpolation and are generated in the virtual view image synthesizing step 401. A depth information image for the image is generated.

In the horizontal search operation 603, when it is determined that an unprocessed region exists in the depth information image generated in the synthesis step 601, an effective depth value is searched for the unprocessed region of the depth information image in units of horizontal scan lines. . The inter-pixel distance having the effective depth value of the unprocessed area boundary among the searched effective depth values is calculated in units of horizontal scan lines. For example, if the position of any A pixel having the effective depth value of the unprocessed area boundary is (5, 1) and the position of the corresponding B pixel horizontally is (10, 1), the A and B pixels in the horizontal scan line unit The distance is five.

Next, in the horizontal compensation step 605, it is determined whether the distance between the pixels for the unprocessed area is smaller than the maximum shift value calculated in units of horizontal scan lines. The calculated disparity value at this time indicates a disparity value calculated between the base view image and the virtual view image, not the disparity value calculated between the reference view cameras. The closer the object is from the reference viewpoint cameras 107 and 111, the larger the variation value, and since the unprocessed area due to the occlusion area is generated by being blocked by the predetermined object, the distance between pixels of the unprocessed area boundary is the variation of the predetermined object (pixel). Less than the value Therefore, based on the maximum shift value, if the distance between the pixels with respect to the unprocessed area is smaller than the maximum shift value, the unprocessed area may be accurately compensated if the unprocessed area is compensated. On the other hand, there may be a case where the interpixel distance with respect to the unprocessed region is larger than the maximum shift value, which will be described later in FIG. 8.

The maximum shift value may be a maximum shift value in a horizontal scan line unit, a block unit, or an entire virtual viewpoint image.

If the distance between the pixels to the unprocessed area is smaller than the maximum shift value, the effective depth value of the boundary of the unprocessed area found in the horizontal search step 603 is used to determine the unprocessed area of the depth information image in units of horizontal scan lines. Compensated by the depth value. The predetermined depth value may be the minimum value of the effective depth values around the boundary of the unprocessed area of the depth information image. As described above, the unprocessed area by the occlusion area is caused by being blocked by a predetermined object. The predetermined depth value may be an average value, a maximum value, or the like of an effective depth value around a boundary of an unprocessed area of the depth information image according to a required accuracy.

In the horizontal compensation step 605, the unprocessed region is not compensated if the interpixel distance with respect to the unprocessed region is larger than the maximum shift value.

Next, in the vertical search step 607, the effective depth value is searched for the unprocessed area of the depth information image in units of vertical scan lines to compensate for the unprocessed area not compensated in the horizontal compensation step 605. The inter-pixel distance having the effective depth value of the unprocessed area boundary among the searched effective depth values is calculated in units of vertical scan lines.

Next, in the vertical compensation step 609, it is determined whether the distance between the pixels for the unprocessed area is smaller than the maximum shift value calculated in units of vertical scan lines. If the interpixel distance with respect to the unprocessed area is smaller than the maximum shift value, the effective depth value of the boundary of the unprocessed area found in the vertical search step 607 is used to determine the unprocessed area of the depth information image in units of vertical scan lines. Compensated by the depth value. If the distance between the pixels with respect to the unprocessed area is larger than the maximum shift value, the unprocessed area is not compensated.

Meanwhile, the vertical search step 607 and the vertical compensation step 609 may be performed first, and then the horizontal search step 603 and the horizontal compensation step 605 may be performed.

7 and 8 illustrate a depth information image of a virtual viewpoint image in which an unprocessed region is compensated by performing a depth information image synthesis step 403 according to the present invention.

FIG. 7 illustrates an image in which an unprocessed area of a depth information image of a virtual viewpoint image of previous and current frames illustrated in FIG. 5 is compensated according to the present invention.

FIG. 8 illustrates second to fourth images 803 and 805 which compensate for the unprocessed region of the first image 801 and the first image 801 which are depth information images generated in the depth information image synthesis step 403, respectively. 807).

The second image 803 is an image obtained by compensating for the unprocessed area of the first image 801 only in the horizontal scan line unit in the depth information image synthesis step 403. The third image 805 is an image obtained by compensating for the unprocessed area of the first image 801 only in the vertical scan line unit in the depth information image synthesis step 403. The fourth image 807 is an image obtained by compensating for the unprocessed area of the first image 801 in units of horizontal and vertical scan lines in the depth information image synthesis step 403.

In the first image 801, the portion corresponding to the portion indicated by the red circle of the second image 803 may be considerably larger in distance between pixels on the boundary of the unprocessed region compared to other portions in the horizontal scan line unit. In addition, the portion of the first image 801 corresponding to the portion indicated by the red circle of the third image 805 may be considerably larger in distance between pixels on the boundary of the unprocessed region compared to other portions in the vertical scan line unit. That is, in each of the portions indicated by the red circles in the second and third images 803 and 805, the distance between the pixels of the unprocessed region boundary detected in each of the vertical and horizontal searching steps 603 and 607 of FIG. 6 is larger than the maximum shift value. It's a big part. Looking at the second and third images 803 and 805, it can be seen that the image of the portion shown by the red circle is incorrectly compensated.

According to the present invention, the portion indicated by the red circle of the second image 803 is compensated through the vertical compensation step 609, and the portion indicated by the red circle of the third image 805 through the horizontal compensation step 605. Is compensated. Compared with the second and third images 803 and 805, it can be seen that the portion indicated by the red circle in the fourth image 807 according to the present invention is compensated more accurately and naturally.

9 is a detailed flowchart illustrating the virtual viewpoint unprocessed region compensation step 405 of FIG. 4.

Referring to FIG. 9, the virtual view unprocessed region compensation step 405 includes a search step 901, a determination step 903, and a setting step 905.

In the search step 901, the depth value of the depth information image for the virtual view image of the current and previous frame, corresponding to each pixel in the unprocessed region of the virtual view image generated in the virtual view image synthesis step 401, is searched for. . Although there is an unprocessed area in the depth information image of the virtual viewpoint image of the current and previous frames, the unprocessed area is compensated by the depth information image synthesizing step 403.

In the determination step 903, it is determined whether the depth value of the depth information image of the virtual viewpoint image of the previous frame found in the search step 901 is the same as the depth value of the depth information image of the virtual viewpoint image of the current frame. The sameness may be recognized when the depth value of the depth information image of the virtual viewpoint image of the previous frame and the depth value of the depth information image of the virtual viewpoint image of the current frame are exactly the same or the difference is within a predetermined range.

In the setting step 905, if the sameness of the result of the determination step 903 is recognized, the unprocessed region of the virtual viewpoint image of the current frame is compensated with a valid pixel value of the virtual viewpoint image of the previous frame. A valid pixel value of the virtual viewpoint image of the previous frame is a pixel value corresponding to each pixel in the unprocessed area. At this time, it does not matter whether the unprocessed area of the virtual viewpoint image of the previous frame is compensated. In the setting step 905, the part of which the sameness is not recognized as the result of the determining step 903 may be compensated by interpolating the boundary effective pixel value of the unprocessed region of the virtual viewpoint image of the current frame.

Meanwhile, in the virtual view image generating method according to the present invention, the disparity information image may be used instead of the depth information image of the virtual view image. In the same way that the depth information is used, the same object in the virtual viewpoint image of the previous frame and the virtual viewpoint image of the current frame has little variation in the variation value, so that the depth information image synthesis step 403 and the virtual viewpoint unprocessed area compensation step 405 ), The disparity information image may be used to compensate the virtual view image.

FIG. 10 is a view illustrating a virtual view image and a virtual view image generated by applying the virtual view image generating method according to the present invention, in which an image is compensated for an unprocessed region recognized for the sameness at the setting step 905. .

As described above, an error may occur when the unprocessed area by the occlusion area is compensated by the pixel value interpolation, etc., so that the unprocessed area by the occlusion area before the unprocessed area is compensated by the pixel value interpolation method has a small occupancy rate. The higher the quality of the final virtual view image can be. The unprocessed area shown by the red circle is an unprocessed area by the occlusion area. In the case of the virtual viewpoint image of the current frame to which the present invention is applied, the unprocessed area is compensated by the intended pixel value. Therefore, compared to the prior art, the virtual viewpoint image generation method according to the present invention can generate a virtual viewpoint image of better image quality without adding a reference viewpoint camera.

11 is a block diagram of a virtual viewpoint image generating apparatus according to an embodiment of the present invention.

Referring to FIG. 11, the apparatus for generating a virtual viewpoint image according to the present invention includes a virtual viewpoint image synthesizer 1101, a depth information image synthesizer 1103, and a virtual viewpoint unprocessed region compensator 1111.

The virtual view image synthesizer 1101 generates a virtual view image of the virtual view camera including the unprocessed region by synthesizing the reference view images of the plurality of reference view cameras. In this case, 3D warping or image interpolation may be applied.

The depth information image synthesizer 1103 synthesizes depth information images of the plurality of reference view cameras to generate a depth information image of the virtual view image generated by the virtual view image synthesizer 1101, and then adds the depth information image to the virtual view image. Compensates for the unprocessed region of the depth information image. The depth information image synthesizing unit 1103 includes a combining unit 1105, a horizontal searching unit 1106, a horizontal compensating unit 1107, a vertical searching unit 1108, and a vertical compensating unit 1109.

The synthesizing means 1105 synthesizes depth information images of the plurality of reference viewpoint cameras to generate depth information images of the virtual viewpoint images generated by the virtual viewpoint image synthesis unit 1101.

The horizontal search unit 1106 searches for an effective depth value of the unprocessed region of the depth information image generated by the combining unit 1105 in units of horizontal scan lines, and has an effective depth value of the unprocessed region boundary among the searched effective depth values. The distance between pixels is calculated in units of horizontal scan lines. The horizontal compensating unit 1107 is an unprocessed region of the depth information image as the minimum depth value among the effective depth values of the boundary of the unprocessed region searched in units of horizontal scan lines when the interpixel distance with respect to the unprocessed region is smaller than the maximum variation value. To compensate. The horizontal compensation means 1107 may compensate the unprocessed area of the depth information image with an average value, a maximum value, and the like of the effective depth value of the boundary of the unprocessed area according to the required accuracy.

The vertical search means 1108 and the vertical compensation means 1109 search and compensate for the unprocessed area like the horizontal search means 1106 and the horizontal compensation means 1107, but search and compensate for the unprocessed area by the vertical scan line.

The virtual view unprocessed region compensator 1111 generates the virtual view image synthesizer 1101 based on the depth information image of the virtual view image of the current and at least one previous frame generated by the depth information image synthesizer 1103. Compensates for the unprocessed area of the virtual view image of the current frame. The virtual viewpoint unprocessed region compensator 1111 includes a search unit 1113, a determination unit 1115, and a setting unit 1117.

The search unit 1113 may determine the depth of the depth information image of the virtual view image of the current and at least one previous frame corresponding to each pixel in the unprocessed region of the virtual view image of the current frame generated by the virtual view image synthesis unit 1101. Search for a value. The determining unit 1115 determines whether or not the depth value of the depth information image is identical to the current and at least one previous frame random virtual viewpoint image searched by the searching unit 1113. The identity may be recognized when the depth values are the same or when the difference between the depth values is within a predetermined range. The setting means 1117 may determine that the virtual view image of the previous frame corresponds to a pixel in the unprocessed area of the virtual viewpoint image of the current frame when the sameness is recognized. Compensate with the pixel value of. The setting unit 1117 may compensate for the unprocessed region by interpolating an effective pixel value of the boundary of the unprocessed region in the case of the unprocessed region where the sameness is not recognized as the determination result of the determining unit 1115.

On the other hand, the virtual view image generating method according to the present invention as described above can be created by a computer program. And the code and code segments constituting the program can be easily inferred by a computer programmer in the art. In addition, the written program is stored in a computer-readable recording medium (information storage medium), and read and executed by a computer to implement the method of the present invention. And the recording medium includes all types of recording media (intangible medium such as a carrier wave as well as tangible media such as CD and DVD) readable by a computer.

Although the present invention has been described by means of limited embodiments and drawings, the present invention is not limited thereto and is intended to be equivalent to the technical idea and claims of the present invention by those skilled in the art to which the present invention pertains. Of course, various modifications and variations are possible.

1 is a view for explaining a virtual view image,

2 and 3 are views illustrating a synthesized virtual view image;

4 is a flowchart illustrating a method of generating a virtual viewpoint image according to an embodiment of the present invention;

FIG. 5 illustrates virtual view images 501 and 505 and depth information images 503 and 507 of previous and current frames generated in the virtual view image synthesis step 401 and the depth information image synthesis step 403 of FIG. 4. Figure,

6 is a detailed flowchart illustrating a depth information image synthesizing step 403 of FIG. 4;

7 and 8 illustrate a depth information image of a virtual viewpoint image in which an unprocessed region is compensated by performing a depth information image synthesis step 403 according to the present invention;

9 is a detailed flowchart illustrating the virtual viewpoint unprocessed region compensation step 405 of FIG. 4;

10 is a view illustrating a virtual view image and a virtual view image generated by applying the virtual view image generating method according to the present invention;

11 is a block diagram of a virtual viewpoint image generating apparatus according to an embodiment of the present invention.

Claims (10)

A virtual viewpoint image synthesizing step of synthesizing reference viewpoint images for a plurality of reference viewpoint cameras to generate a virtual viewpoint image for the virtual viewpoint camera; A depth information image is synthesized by synthesizing depth information images of the plurality of reference view cameras, and generating a depth information image of the virtual view image, and compensating a first unprocessed region which is an unprocessed region of the depth information image of the virtual view image. step; And Compensating for the virtual unprocessed region compensation step of compensating for the second unprocessed region which is an unprocessed region of the virtual viewpoint image of the current frame based on the depth information image of the virtual view image of the current and at least one previous frame. Virtual viewpoint image generation method comprising a. The method of claim 1, The depth information image synthesis step, Synthesizing a depth information image of the virtual view image; A horizontal search step of searching for an effective depth value for the first unprocessed area in a horizontal scan line unit; A horizontal compensation step of compensating the second unprocessed area by using an effective depth value of the boundary when the distance between pixels having an effective depth value of the first unprocessed area boundary in a horizontal scan line unit is smaller than a maximum shift value; A vertical search step of searching for an effective depth value of the unprocessed area existing as a result of the horizontal compensation step in a vertical scan line unit; And When the interpixel distance having the effective depth value of the unprocessed area boundary of the vertical search step in the vertical scan line unit is smaller than the maximum shift value, the vertical compensation of the unprocessed area of the vertical search step is performed using the effective depth value of the boundary. Compensation stage Virtual viewpoint image generation method comprising a. 3. The method of claim 2, The horizontal and vertical compensation step Compensating the unprocessed area to the minimum of the effective depth value of the boundary Virtual viewpoint image generation method. The method of claim 1, The virtual point unprocessed region compensation step A search step of searching for a depth value of a depth information image of a virtual viewpoint image of the current and at least one previous frame corresponding to each pixel in the second unprocessed region; A determination step of determining whether or not each of the depth values searched in the search step is identical; And A setting step of setting a pixel value of at least one virtual viewpoint image of at least one previous frame corresponding to each pixel in the second unprocessed region to a pixel value in the second unprocessed region when the sameness is recognized as a result of the determining step. Virtual viewpoint image generation method comprising a. The method of claim 4, wherein The determining step Determination of equality on the basis of the case that each of the depth value is the same or the difference between each depth value is within a predetermined range Virtual viewpoint image generation method. A virtual viewpoint image synthesizing step of synthesizing reference viewpoint images for a plurality of reference viewpoint cameras to generate a virtual viewpoint image for the virtual viewpoint camera; A disparity information image synthesizing step of synthesizing disparity information images of the plurality of reference view cameras to generate disparity information images of the virtual view image, and compensating an unprocessed region of the disparity information image of the virtual view image; And Compensating for the unprocessed area of the virtual view image of the virtual view image of the current frame based on the disparity information image of the virtual view image of the current and at least one previous frame. Virtual viewpoint image generation method comprising a. A virtual viewpoint image synthesizer configured to generate a virtual viewpoint image for the virtual viewpoint camera by synthesizing the reference viewpoint images for the plurality of reference viewpoint cameras; A depth information image is synthesized by synthesizing depth information images of the plurality of reference view cameras, and generating a depth information image of the virtual view image, and compensating a first unprocessed region which is an unprocessed region of the depth information image of the virtual view image. part; And A virtual point unprocessed area compensator for compensating a second unprocessed area that is an unprocessed area of the virtual view image of the current frame based on a depth information image of the virtual view image of the current and at least one previous frame. Virtual viewpoint image generating apparatus comprising a. The method of claim 7, wherein The depth information image synthesizer Synthesizing means for synthesizing a depth information image with respect to the virtual viewpoint image; Horizontal search means for searching for an effective depth value for the first unprocessed area in a horizontal scan line unit; Horizontal compensation means for compensating the first unprocessed area by using an effective depth value of the boundary when the distance between pixels having an effective depth value of the first unprocessed area boundary in a horizontal scan line unit is smaller than a maximum shift value; Vertical search means for searching for an effective depth value of the unprocessed region existing as a result of the horizontal means in a vertical scan line unit; And When the interpixel distance having the effective depth value of the unprocessed area boundary of the vertical search means in the unit of vertical scan line is smaller than the maximum shift value, the vertical compensation of the unprocessed area of the vertical search means is performed using the effective depth value of the boundary. Compensation Means Virtual viewpoint image generating apparatus comprising a. The method of claim 8, The horizontal and vertical compensation means Compensating the unprocessed area to the minimum of the effective depth value of the boundary Virtual viewpoint image generating device. The method of claim 7, wherein The virtual point unprocessed area compensation unit Search means for searching for a depth value of a depth information image of a virtual viewpoint image of the current and at least one previous frame, corresponding to each pixel in the second unprocessed region; Determination means for determining whether or not each of the depth values searched by the search means is identical; And Setting means for setting a pixel value of the virtual viewpoint image of the at least one or more previous frames corresponding to each pixel in the second unprocessed region to a pixel value in the second unprocessed region when the result of the determination means is confirmed to be the same. Virtual viewpoint image generating apparatus comprising a.

Images