KR20120062477A - Method and apparatus for generating multi-view depth map - Google Patents

Abstract

PURPOSE: A multi-view depth image generating method and an apparatus thereof are provided to minimize error of matching function in consideration of matching error and to generate a depth image of high correlation. CONSTITUTION: A depth image generating unit(370) generates a second depth image based on two color images for a second point. A right side depth image generating unit(350) generates two color images, a first depth image, and a third depth image. The third point is between the first point and the second point.

Description

METHOD AND APPARATUS FOR GENERATING MULTI-VIEW DEPTH MAP}

The present invention relates to a method and an apparatus for generating a multiview depth image.

Three-dimensional television system using three-dimensional video has been in the spotlight as the next generation broadcasting system because it can provide a realistic image to the user through the content reconstructed from the real world through computer simulation and three-dimensional stereoscopic display. Here, the 3D television system is an extended concept of a conventional system for photographing a scene using a single camera, and refers to a system that can acquire 3D information of a scene using a plurality of cameras or sensors. Viewers can watch a scene from multiple points of view using three-dimensional video, allowing viewers to watch stereoscopic movies, stereoscopic television broadcasts, or experience games, exhibits, or events based on realistic three-dimensional images. . In addition, 3D video can be used in an immersive virtual surgery education program to provide effective education.

The three-dimensional video is expressed in three-dimensional through the depth image. The depth image refers to an image in which 3D distance information of objects existing in the image is represented by 8 bits, and the pixel value of the depth image represents depth information of each corresponding pixel. In particular, in order to implement a free view function of a 3D video, a mid-view image synthesizing technique for synthesizing virtual mid-view images existing between views using an image of several views is essential. In order to synthesize the mid-view image, a depth image is required. Since the accuracy of the depth image determines the quality of the synthesized mid-view image, it is very important to generate an accurate depth image. However, since the 3D television system independently searches for a depth image at each viewpoint of a multiview image, a correlation between viewpoints may be degraded.

Recently, the 3D video coding group within the Moving Picture Experts Group (MPEG), an international standardization group, realizes the importance of encoding multiview video and depth video, and implements depth image search software that can estimate depth images. And distributed. The original purpose of the software was to acquire depth images for stereo images. However, since the 3D video format suitable for the 3D video coding group is not a still image but a multiview image rather than a stereo image, it is not suitable to obtain a depth image by applying the software as it is. As a result, depth search algorithms can cause temporal and point-of-view correlation degradation problems.

An object of the present invention is to provide a multi-view depth image generation method and apparatus for generating a depth image having a high correlation between viewpoints.

A method of generating a multiview depth image using a plurality of color images in a 3D image generating apparatus according to an embodiment of the present invention, based on at least two color images for a first viewpoint among the plurality of color images Generating a first depth image corresponding to the first viewpoint, and generating a second depth image corresponding to the second viewpoint based on at least two color images for a second viewpoint among the plurality of color images And based on at least two color images, a first depth image, and a second depth image, for a third viewpoint, which is between the first viewpoint and the second viewpoint, among the plurality of color images. Generating a corresponding third depth image.

The generating of the third depth image may include obtaining corresponding pixels corresponding to pixels of the third depth image from the first depth image and the second depth image, and based on depth information of each of the corresponding pixels. The method may include generating the third depth image by determining a final depth value of the third depth image.

The obtaining of the corresponding pixels may include obtaining the first depth image and the second depth image by projecting them to the third viewpoint.

The method may further include correcting the first depth image and the second depth image by changing depth information of the corresponding pixel to the final depth value.

The method may further include transmitting the corrected first depth image, the corrected second depth image, and the third depth image to a 3D image reproducing apparatus.

The generating of the third depth image may include searching for a corresponding pixel between at least two color images for the third viewpoint, and among the searched pixels using the first depth image and the second depth image. Determining a matching pixel; and obtaining a final depth value corresponding to a position difference of the matching pixel, and generating the third depth image including the final depth value.

The searching of the corresponding pixel may include selecting one color image from among at least two color images for the third viewpoint as a reference image, and each of the reference image in another color image for the third viewpoint. Searching for a pixel corresponding to the pixel.

The determining of the matching pixel may include obtaining first corresponding pixels and second corresponding pixels corresponding to the respective pixels from the first depth image and the second depth image, and the first corresponding pixel and the second corresponding pixel. Determining a pixel corresponding to each pixel based on a depth difference between a depth pixel of each neighboring pixel including a corresponding pixel and a neighboring pixel that is a difference between depth information of each pixel, wherein the depth of each pixel is determined. The information may be information calculated based on variation information of each pixel and the searched pixel.

The method may further include obtaining a difference between pixel values of each pixel and the searched pixel, and determining a pixel corresponding to each pixel may include determining a difference between the pixel value and a depth difference between the neighboring pixels. And determining the searched pixel when the sum is the minimum as a pixel corresponding to each pixel.

The method may further include obtaining a difference between pixel values of each pixel and the searched pixel, and obtaining a temporal depth difference that is a difference between depth information of each pixel and depth information corresponding to each pixel in a previous image. The determining of the pixel corresponding to each pixel may include: searching for a pixel when the sum of the difference of the pixel value, the temporal depth difference, and the depth difference with the neighboring pixel becomes minimum; And determining the pixel to be.

The method may include obtaining a difference between pixel values of each pixel and the searched pixel, obtaining a temporal depth difference that is a difference between depth information of each pixel and depth information corresponding to each pixel in a previous image, and And determining a depth difference between views, which is a difference between depth information of each pixel and depth information of an adjacent view, wherein determining a pixel corresponding to each pixel may include a difference in the pixel value, the temporal depth difference, The method may include determining the searched pixel as the pixel corresponding to each pixel when the sum of the depth difference between the viewpoints and the depth difference with the neighboring pixel becomes the minimum.

The generating of the first depth image may include generating the first depth image based on depth determination information including shift information between at least two color images for the first viewpoint.

The depth determination information may further include depth information of adjacent pixels.

A method of generating a multi-view depth image in a 3D image reproducing apparatus according to another embodiment of the present invention, the method comprising: receiving a plurality of color images from the 3D image generating apparatus, for a first view from among the plurality of color images Generating a first depth image corresponding to the first view based on at least two color images, and corresponding to the second view based on at least two color images for a second view among the plurality of color images Generating a second depth image, and at least two color images, a first depth image, and the second depth image for a third viewpoint between the first viewpoint and the second viewpoint among the plurality of color images; And generating a third depth image corresponding to the third viewpoint based on the image.

The generating of the first depth image may include generating the first depth image based on depth determination information including shift information between at least two color images for the first viewpoint.

The depth determination information may further include depth information of each adjacent pixel.

The generating of the third depth image may include projecting the first depth image and the second depth image to the third viewpoint, and of the third depth image in the projected first depth image and the second depth image. Obtaining corresponding pixels corresponding to the pixels, and determining a final depth value of the third depth image based on depth information of each of the corresponding pixels to generate the third depth image.

The method may further include correcting the first depth image and the second depth image by changing the final depth value to depth information of the corresponding pixel.

The generating of the third depth image may include generating an angle of the third depth image based on depth determination information obtained using at least two color images for the third viewpoint, the first depth image, and the second depth image. The method may include obtaining depth information corresponding to a pixel and generating the third depth image by mapping the depth information to each pixel.

The depth determination information includes variation information of each pixel and depth information of each pixel adjacent to each pixel, and the neighboring pixel corresponds to each pixel in the first depth image and the second depth image. The pixel may include a first corresponding pixel, a second corresponding pixel, and an adjacent pixel of each pixel.

An apparatus for generating a multiview depth image according to another embodiment of the present invention includes an image storage unit storing a plurality of color images, and the first view based on at least two color images for a first view from among the plurality of color images. A first depth image generator configured to generate a first depth image corresponding to the second depth image generator based on at least two color images for a second viewpoint among the plurality of color images; A second depth image generating unit and at least two color images for the third viewpoint between the first viewpoint and the second viewpoint among the plurality of color images, the first depth image, and the second depth image; And a third depth image generator configured to generate a third depth image corresponding to the third viewpoint.

The first depth image generation unit generates the first depth image based on depth determination information including variation information between at least two color images for the first viewpoint and depth information of an adjacent pixel, and the second depth image. The image generator may generate the second depth image based on the depth determination information including the shift information between the at least two color images for the second viewpoint and the depth information of the adjacent pixel.

The third depth image generation unit obtains corresponding pixels corresponding to the pixels of the third depth image from the first depth image and the second depth image projected to the third viewpoint, and based on depth information of each of the corresponding pixels. The third depth image may be generated by determining a final depth value of the third depth image.

The apparatus for generating a multiview depth image may further include an inter-correlation control unit for correcting depth information of the first depth image and the second depth image based on the depth information of the third depth image.

The inter-view correlation controller may change depth information of a pixel corresponding to the third depth image in the first depth image and the second depth image to be the same as the depth information of the third depth image.

According to an exemplary embodiment of the present invention, in the process of searching for depth information, the correlation between adjacent viewpoints may be increased by minimizing the error of the matching function in consideration of the matching error of adjacent viewpoints, and generating a multiview depth image having high correlation. The image quality of the synthesized image can be improved. In addition, in multi-view depth image encoding using an inter-view prediction technique, efficiency may be increased by using a depth image having a high inter-view correlation according to an embodiment of the present invention.

1 is a view schematically showing a three-dimensional video system used according to an embodiment of the present invention.
2A and 2B are diagrams illustrating a method of generating a depth image according to an exemplary embodiment of the present invention, respectively.
3 is a flowchart illustrating a method of generating a multiview depth image according to an embodiment of the present invention.
4 is a flowchart illustrating a method of generating a multiview depth image according to an embodiment of the present invention.
5 is a block diagram illustrating an apparatus for generating a multiview depth image according to an embodiment of the present invention.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and like reference numerals designate like parts throughout the specification.

Throughout the specification, when a part is said to "include" a certain component, it means that it can further include other components, without excluding other components unless specifically stated otherwise.

A method and apparatus for generating a multiview depth image according to an exemplary embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

1 is a view schematically showing a three-dimensional video system used according to an embodiment of the present invention.

Referring to FIG. 1, a 3D video system includes a 3D image generating apparatus 100 for generating 3D video and a 3D image reproducing apparatus 200 for reproducing a 3D image. In this case, at least one of the 3D image generating apparatus 100 and the 3D image reproducing apparatus 200 includes a multi-view depth image generating apparatus 300.

The 3D image generating apparatus 100 acquires an image through a depth camera using a color camera and a sensor including a stereoscopic camera or a multi-view camera. The 3D image generating apparatus 100 transmits the acquired color video, depth image, and various metadata to the 3D image reproducing apparatus 200. In this case, the 3D image generating apparatus 100 may generate a multiview image of the 3D scene using a plurality of cameras located at various viewpoints in the same time zone.

The 3D image reproducing apparatus 200 reproduces the 3D image using the color image and the depth image transmitted from the 3D image generating apparatus 100. In this case, the 3D image reproducing apparatus 200 may reproduce an image suitable for a 2D display apparatus or a 3D display apparatus, and may reproduce a viewpoint selected from multiviews. For example, an apparatus for recognizing a viewer's position and playing a stereoscopic image of a viewpoint corresponding to the viewer's position provides a viewer with stereoscopic images of various viewpoints by using the transmitted viewpoint image.

The multi-view depth image generating apparatus 300 is a device for generating a multi-view depth image, and generates a multi-view depth image required for reproducing the 3D image of the multi-view in the 3D image reproducing apparatus 200. In order for the 3D image reproducing apparatus 200 to reproduce a 3D image of a free view, a greater number of viewpoints than the views obtained through the camera in the 3D image generating apparatus 100 are required. To this end, a view synthesis technique for generating a virtual intermediate viewpoint image existing between viewpoints using a multiview image is used. In this case, a depth map is required to synthesize the intermediate viewpoint image. The depth image is an image including 3D distance information of objects existing in the image, and each pixel included in the depth image includes depth information of the corresponding pixel. The multi-view depth image generating apparatus 300 may be formed on at least one of the 3D image generating apparatus 100 and the 3D image reproducing apparatus 200. That is, the depth image for the intermediate view image may be generated in advance in the 3D image generating apparatus 100 and transferred to the 3D image reproducing apparatus 200, and the multiview image transmitted from the 3D image reproducing apparatus 200. Can be generated using.

Next, a method of generating a multiview depth image using a multiview image will be described.

2A and 2B are diagrams illustrating a method of generating a depth image according to an exemplary embodiment of the present invention, respectively.

The multiview depth image generating apparatus 300 generates a depth image using a plurality of color images, that is, a multiview image. In this case, the multi-view depth image generating apparatus 300 may generate a depth image by using a left image and a right image, or by using a left image, a center image, and a right image. In this case, the depth image using the left image, the center image, and the right image can be used for the left and center images and the right and center images at the same time. Next, the depth information will be described using two color images, that is, a left image and a right image.

First, referring to FIG. 2A, the multi-view depth image generating apparatus 300 searches for a right image to find pixels corresponding to all pixels of a reference image, for example, a left image. The multi-view depth image generating apparatus 300 searches for pixels of the right image based on each pixel of the left image and finds a matching position between the left image and the right image. In this case, the matching position or matching pixel is a position or pixel of the same object in a different image. The multi-view depth image generating apparatus 300 uses a matching function to determine whether the pixels (x, y) of the left image and the pixels (x-d, y) of the right image are matching pixels. The matching function represents an error value when the pixels of the left image and the right image are compared. The smaller the error value is, the more likely it is to be a matching position. The multi-view depth image generating apparatus 300 determines the disparity d having the minimum matching function as the depth information of the pixels x and y. In this case, the variation (d) represents the distance between the corresponding pixels in the color image photographed by each color camera, and the larger the variation is, the farther the depth is.

The matching function [E (x, y, d)] can be expressed as Equation 1 to Equation 6.

In Equations 1 to 6, x and y are position coordinates of two-dimensional pixels of the pixel, and d is depth information of the pixel of the left image and the pixel searched from the right image as reference. The E _data (x, y, d) function includes the E _sim (x, y, d) function, and additionally the E _temp (x, y, d) function and the E _view (x, y) function to increase the accuracy of the matching function. , d) may further include a function.

E _sim (x, y, d) is a function that calculates the difference between the pixel values of the left and right images. The E _sim (x, y, d) function can find a matching position by comparing the difference of pixel values as shown in FIG. 2A.

The E _temp (x, y, d) function calculates the difference between depth information of the current image and the previous image including the pixels (x, y) and reflects the temporal depth difference to the matching function.

The E _view (x, y, d) function is a function for obtaining the difference between the depth information of the pixels (x, y) and the depth information (D _{i '} ) of the adjacent view and reflects the depth difference between the _views to the matching function.

Next, referring to FIG. 2B, the matching function [E (x, y, d)] further includes an E _smooth (x, y, d) function to minimize the error. The E _smooth (x, y, d) function is a pixel adjacent to pixel (x, y), for example, pixel (x1, x1), pixel (x2, x2), pixel (x3, x3), pixel (x4). A function that calculates the difference between the depth information (D (x1, x1), D (x2, x2), D (x3, x3), and D (x4, x4)) of, x4). Reflect on. In this case, the E _smooth (x, y, d) function may additionally use depth information of the corresponding pixel of the adjacent view in addition to the neighboring pixels of the same view.

Next, a method of generating a multiview depth image by the multiview depth image generating apparatus 300 based on a matching function will be described.

3 is a flowchart illustrating a method of generating a multiview depth image according to an embodiment of the present invention.

Referring to FIG. 3, the multi-view depth image generating apparatus 300 generates a depth image corresponding to a left view based on at least two color images positioned on the left side among the plurality of color images (S310). To this end, the multi-view depth image generating apparatus 300 obtains depth information d in which the matching function [E (x, y, d)] of Equation 1 is minimum, and includes depth information d that is minimum. Create a depth image. In particular, the multi-view depth image generating apparatus 300 includes the variation information between color images and the depth information of adjacent pixels [D (x1, x1), D (x2, x2), D (x3, x3), and D (x4, x4). A multi-view depth image is generated based on the depth determination information including).

The multi-view depth image generating apparatus 300 generates a depth image corresponding to a right viewpoint based on at least two color images positioned on the right side among the plurality of color images in the same manner as in operation S310 (S320).

The multi-view depth image generating apparatus 300 calculates depth information, that is, variation information, of the depth image based on at least two color images positioned at the center among the plurality of color images (S330). At this time point the depth image generating unit 300 again by using the E _data (x, y, d) function of Equation _{2 E data (x, y,} d) is obtained for the depth information (d) the function is minimized.

The multiview depth image generating apparatus 300 may further include a matching function E (x, y, d) including an E _smooth (x, y, d) function of Equation 6 based on the depth information d obtained in operation S330. d)] to determine the depth information d to be the minimum (S340). In this case, the multi-view depth image generating apparatus 300 additionally uses the pixels of the depth image obtained in steps S310 and S320 together with the adjacent pixels at the same time as the neighboring pixels. That is, when the depth image generating apparatus 300 generates the depth image corresponding to the center viewpoint, an error is generated by using pixels corresponding to the pixels (x, y) that are different from each other, that is, the pixels corresponding to the left and right views. Minimize. In this case, the multi-view depth image generating apparatus 300 may search for the corresponding pixel by projecting the depth image corresponding to the left view and the depth image corresponding to the left view to the center view. . Corresponding nodes thus connected serve as additional nodes in minimizing the error of the matching function of the center image.

Thereafter, the multi-view depth image generating apparatus 300 determines the depth information d having the minimum matching function E (x, y, d) as a final depth value and includes a final depth value. To generate (S350).

Next, the multiview depth image generating apparatus 300 changes the depth information of the pixels of the left and right views corresponding to each pixel of the center view to the same value as the depth information of each pixel of the center view to increase the correlation between the views. (S360). That is, the multiview depth image generating apparatus 300 corrects the corresponding pixels between viewpoints to have the same depth information. When the distance between the multiview cameras is constant, objects that exist at the same time point have the same depth information, but may have different depth information for each time point due to an error of the depth image search method. However, if the pixels corresponding to the viewpoints are corrected with the same depth information, the correlation between the viewpoints may increase.

4 is a flowchart illustrating a method of generating a multiview depth image according to an embodiment of the present invention.

Referring to FIG. 4, the multi-view depth image generating apparatus 300 generates a multi-view depth image using a plurality of color images, and the multi-view depth image generating apparatus 300 generates the multi-view images 10 and 11. , Depth information is calculated using the images 10, 11, and 12 located on the left side of the frames 12, 13, and 14 (S410). At this time point the depth image generating unit 300 again calculates the _data E (x, y, d) the depth information is a function which is minimized by using the _data E (x, y, d) a function of the expression (2).

The multi-view depth image generating apparatus 300 calculates depth information using the images 11, 12, and 13 positioned in the center (S420). At this time point the depth image generating unit 300 again calculates the _data E (x, y, d) the depth information is a function which is minimized by using the _data E (x, y, d) a function of the expression (2).

The multi-view depth image generating apparatus 300 calculates depth information using the images 12, 13, and 14 positioned on the right side (S430). At this time point the depth image generating unit 300 again calculates the _data E (x, y, d) the depth information is a function which is minimized by using the _data E (x, y, d) a function of the expression (2).

The multi-view depth image generating apparatus 300 minimizes an error based on depth information calculated using the images 10, 11, and 12 (S440). At this time, the multi-view depth image generating apparatus 300 considers an E _data (x, y, d) function of Equation 2 and an E _smooth (x, y, d) function of Equation 6 simultaneously. , y, d)] to obtain the depth information. In this case, the E _smooth (x, y, d) function may use four adjacent pixels at the same time as the pixel (x, y). As a result, an initial depth image 21 corresponding to the left view is generated.

Likewise, the multiview depth image generating apparatus 300 minimizes an error based on depth information calculated using the images 12, 13, and 14 (S450). Through this, the initial depth image 23 corresponding to the right view is generated.

The multi-view depth image generating apparatus 300 projects the initial depth images 21 and 23 to the center viewpoint, respectively, in order to obtain a pixel of the depth image corresponding to the center viewpoint and a corresponding pixel of the initial depth images 21 and 23 ( S460, S470).

The multi-view depth image generating apparatus 300 minimizes an error based on the depth information calculated using the images 11, 12, and 13 and the depth information of the initial depth images 21 and 23 (S480). At this time, the multi-view depth image generating apparatus 300 considers an E _data (x, y, d) function of Equation 2 and an E _smooth (x, y, d) function of Equation 6 simultaneously. , y, d)] to obtain the depth information. In particular, the E _smooth (x, y, d) function may use four pixels adjacent to the pixels (x, y) and pixels corresponding to the pixels (x, y) of the initial depth images 21 and 23, respectively. Can be. Through this, the depth image 32 corresponding to the center viewpoint is generated. Also, the multiview depth image generating apparatus 300 sets depth information of the pixel corresponding to the pixel (x, y) of the center view in the initial depth images 21 and 23 to the same value as the pixel (x, y) of the center view. In this case, the depth image 31 corresponding to the left view and the depth image 33 corresponding to the right view are generated.

5 is a block diagram illustrating an apparatus for generating a multiview depth image according to an embodiment of the present invention.

Referring to FIG. 5, the apparatus 100 for generating a multiview depth image includes an image storage unit 310 storing a plurality of color images, a left depth image generator 330 generating a depth image corresponding to a left view, and a right side. The right depth image generator 350 generates a depth image corresponding to a viewpoint, the depth image generator 370 generates a depth image corresponding to a center viewpoint, and a correlation controller 390 between viewpoints.

The image storage unit 310 stores a plurality of color images. If the multiview depth image generating apparatus 300 is formed in the 3D image generating apparatus 100, the image storage unit 310 stores a plurality of color images acquired from a color camera such as a multiview camera and again. When the point depth image generating apparatus 300 is formed in the 3D image reproducing apparatus 200, a plurality of color images received from the 3D image generating apparatus 100 are stored.

As described above, the left depth image generator 330 generates the depth image based on the depth determination information including the shift information between the at least two color images located on the left side and the depth information of the adjacent pixel.

As described above, the right depth image generator 350 generates the depth image based on the depth determination information including the shift information between the at least two color images positioned on the right side and the depth information of the adjacent pixel.

As described above, the center depth image generator 370 also obtains corresponding pixels corresponding to the pixels of the depth image of the center viewpoint from the left and right depth images projected to the center viewpoint, and based on the depth information of each of the corresponding pixels. The final depth value of the depth image is determined to generate a center depth image.

The inter-view correlation controller 390 changes the depth information of the corresponding pixel between the depth images corresponding to the left, right, and middle viewpoints to make the depth information of the objects existing at the same viewpoints the same.

When multi-view depth images are generated, image quality of composite images using multi-view depth images generated by increasing the correlation between viewpoints by minimizing the error of the matching function while simultaneously considering the matching errors of adjacent viewpoints in the process of searching depth information. Can increase.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concepts of the present invention defined in the following claims are also provided. It belongs to the scope of rights.

Claims (25)

A method of generating a multiview depth image using a plurality of color images in a 3D image generating apparatus,
Generating a first depth image corresponding to the first view based on at least two color images for a first view among the plurality of color images;
Generating a second depth image corresponding to the second viewpoint based on at least two color images for a second viewpoint among the plurality of color images; and
Based on at least two color images, a first depth image, and a second depth image corresponding to the third viewpoint between the first viewpoint and the second viewpoint among the plurality of color images, the third viewpoint corresponds to the third viewpoint. Generating a third depth image
Multi-view depth image generation method comprising a. In claim 1,
The generating of the third depth image may include
Obtaining corresponding pixels corresponding to the pixels of the third depth image in the first depth image and the second depth image, and
Generating a third depth image by determining a final depth value of the third depth image based on depth information of each of the corresponding pixels;
Multi-view depth image generation method comprising a. In claim 2,
Obtaining the corresponding pixels, respectively,
And multiplying the first depth image and the second depth image to the third viewpoint to obtain each. In claim 2,
Correcting the first depth image and the second depth image by changing depth information of the corresponding pixel to the final depth value;
Multi-view depth image generation method further comprising. 5. The method of claim 4,
Transmitting the corrected first depth image, the corrected second depth image, and the third depth image to a 3D image reproducing apparatus.
Multi-view depth image generation method further comprising. In claim 1,
The generating of the third depth image may include
Searching for a corresponding pixel between at least two color images for the third viewpoint;
Determining a matching pixel among the searched pixels using the first depth image and the second depth image, and
Obtaining a final depth value corresponding to the position difference of the matching pixels and generating the third depth image including the final depth value;
Multi-view depth image generation method comprising a. The method of claim 6,
Searching for the corresponding pixel
Selecting one color image from among at least two color images for the third viewpoint as a reference image, and
Searching for a pixel corresponding to each pixel of the reference image in another color image for the third viewpoint
Multi-view depth image generation method comprising a. In claim 7,
Determining the matching pixel
Obtaining a first corresponding pixel and a second corresponding pixel corresponding to each pixel in the first depth image and the second depth image, and
Determining a pixel corresponding to each pixel based on a depth difference between a depth pixel of each neighboring pixel including the first corresponding pixel and the second corresponding pixel and a neighboring pixel that is a difference between depth information of each pixel. Including;
And depth information of each pixel is information calculated based on variation information of each pixel and the searched pixel. 9. The method of claim 8,
Obtaining a difference between pixel values of each pixel and the searched pixel;
Determining a pixel corresponding to each pixel
Determining the searched pixel when the sum of the difference between the pixel value and the depth difference between the neighboring pixel is the minimum as a pixel corresponding to each pixel.
Multi-view depth image generation method comprising a. 9. The method of claim 8,
Obtaining a difference between pixel values of each pixel and the searched pixel, and
Obtaining a temporal depth difference which is a difference between depth information of each pixel and depth information corresponding to each pixel in a previous image;
More,
Determining a pixel corresponding to each pixel
Determining the searched pixel as the pixel corresponding to each pixel when the sum of the pixel value, the temporal depth difference, and the depth difference with the neighboring pixel becomes minimum
Multi-view depth image generation method comprising a. In paragraph 8
Obtaining a difference between pixel values of each pixel and the searched pixel;
Obtaining a temporal depth difference which is a difference between depth information of each pixel and depth information corresponding to each pixel in a previous image, and
Obtaining a depth difference between views, which is a difference between depth information of each pixel and depth information of an adjacent view;
More,
Determining a pixel corresponding to each pixel
Determining the searched pixel as the pixel corresponding to each pixel when the sum of the pixel value, the temporal depth difference, the depth difference between the viewpoints, and the depth difference with the neighboring pixel becomes minimum.
Multi-view depth image generation method comprising a. In claim 1,
The generating of the first depth image may include
Generating the first depth image based on depth determination information including shift information between at least two color images for the first viewpoint. The method of claim 12,
The depth determination information further includes depth information of adjacent pixels. A method of generating a multiview depth image in a 3D image reproducing apparatus,
Receiving a plurality of color images from the 3D image generating apparatus;
Generating a first depth image corresponding to the first view based on at least two color images for a first view among the plurality of color images;
Generating a second depth image corresponding to the second viewpoint based on at least two color images for a second viewpoint among the plurality of color images; and
Based on at least two color images, a first depth image, and a second depth image corresponding to the third viewpoint between the first viewpoint and the second viewpoint among the plurality of color images, the third viewpoint corresponds to the third viewpoint. Generating a third depth image
Multi-view depth image generation method comprising a. The method of claim 14,
The generating of the first depth image may include
Generating the first depth image based on depth determination information including shift information between at least two color images for the first viewpoint. 16. The method of claim 15,
The depth determination information further includes depth information of each adjacent pixel. The method of claim 14,
The generating of the third depth image may include
Projecting the first depth image and the second depth image to the third viewpoint; and
Corresponding pixels corresponding to the pixels of the third depth image are obtained from the projected first depth image and the second depth image, and the final depth value of the third depth image is determined based on depth information of each of the corresponding pixels. Generating the third depth image
Multi-view depth image generation method comprising a. The method of claim 17,
Correcting the first depth image and the second depth image by changing the final depth value to depth information of the corresponding pixel;
Multi-view depth image generation method further comprising. The method of claim 14,
The generating of the third depth image may include
Obtaining depth information corresponding to each pixel of the third depth image based on depth determination information obtained by using at least two color images, the first depth image, and the second depth image for the third viewpoint; Generating the third depth image by mapping depth information to the respective pixels. The method of claim 19,
The depth determination information is
The variation information of each pixel and depth information of each pixel adjacent to each pixel,
The neighboring pixel includes a first corresponding pixel and a second corresponding pixel corresponding to each pixel in the first depth image and the second depth image, and a neighboring pixel of each pixel. Image storage unit for storing a plurality of color images,
A first depth image generator configured to generate a first depth image corresponding to the first viewpoint based on at least two color images for a first viewpoint among the plurality of color images;
A second depth image generator configured to generate a second depth image corresponding to the second viewpoint based on at least two color images for a second viewpoint among the plurality of color images; and
Based on at least two color images, a first depth image, and a second depth image corresponding to the third viewpoint between the first viewpoint and the second viewpoint among the plurality of color images, the third viewpoint corresponds to the third viewpoint. Third depth image generator for generating a third depth image
Multi-view depth image generating device comprising a. 22. The method of claim 21,
The first depth image generator
Generating the first depth image based on the depth determination information including the shift information between at least two color images for the first viewpoint and the depth information of an adjacent pixel,
The second depth image generator
And generating the second depth image based on the depth determination information including the shift information between at least two color images for the second view and the depth information of adjacent pixels. 22. The method of claim 21,
The third depth image generator
The corresponding pixels corresponding to the pixels of the third depth image are respectively obtained from the first depth image and the second depth image projected to the third viewpoint, and the final depth of the third depth image is based on depth information of each of the corresponding pixels. A multi-view depth image generating apparatus generating the third depth image by determining a depth value. 22. The method of claim 21,
A correlation controller between points of time in which depth information of the first depth image and the second depth image is corrected based on depth information of the third depth image;
Multi-view depth image generating device further comprising. 25. The method of claim 24,
The correlation control unit between the time points
And a depth information of the pixel corresponding to the third depth image in the first depth image and the second depth image to be identical to the depth information of the third depth image.

Images