KR20100008677A - Device and method for estimating death map, method for making intermediate view and encoding multi-view using the same - Google Patents

Abstract

PURPOSE: A depth map estimation apparatus and method, an intermediate image production method using the same, and a multi-point video encoding method are provided to obtain a depth map strong against external affects. CONSTITUTION: A 3D warping unit(30) performs 3D warping of each segment set through a segment setup unit(20) into the view point of a reference image adjacent to a subject image. An initial depth value search unit(40) estimates the depth value of the subject image based on the reference image, and searches one depth value for each segment. A depth value refining part(50) obtains the final depth value by refining the initial depth value output from the initial depth value search unit.

Description

Device and method for estimating death map, method for making intermediate view and encoding multi-view using the same

The present invention relates to an apparatus and method for estimating a depth map, a method for generating an intermediate image using the same, and a method for encoding a multiview video. More specifically, the region is divided into segments based on the similarity of the pixels, an initial depth map of the segment unit is obtained by using a three-dimensional warping technique and a self-adaptive function in which an extended gradient map is reflected, and then the reliability diffusion method is performed by the segment unit. By refining the initial depth map, we can reduce the error, reduce the complexity, and obtain the depth map that is robust to external influences. A depth map estimating apparatus and method for obtaining encoding efficiency, an intermediate image generation method using the same, and a method for encoding a multiview video are provided.

As digital technology is highly developed and broadcasting media are diversified due to convergence of broadcasting and communication, broadcasting-related additional services using the characteristics of digital technology are newly introduced. Currently, the direction of TV development is going to be high quality and large screen, but since the TV screen itself is two-dimensional, the three-dimensional feeling cannot be felt through the current screen.

Three-dimensional video processing technology is a core technology in the next-generation information and communication service field, and it is a cutting-edge technology with competition for technology development with the development of the information industry society. Such three-dimensional video processing technology is an essential element to provide high quality video services in multimedia applications. Today, the application fields such as broadcasting, medical, education, military, games, virtual reality, as well as information and communication fields are diversified. have. In addition, three-dimensional video processing technology has become a core foundation technology of next-generation realistic three-dimensional multimedia, which is commonly required in many fields, and researches on it are being conducted actively in advanced countries.

In general, three-dimensional video can be defined from two perspectives. First, a 3D video may be defined as a video configured to apply a depth information to an image so that a user may feel a part of the image protruding from the screen. Secondly, the 3D video may be defined as a video configured to provide a user with various viewpoints so that the user can feel a reality in the image. The 3D video may be classified into a binocular, a polycular, an integral photography (IP), a multiview (omni, a panorama), a hologram, and the like according to an acquisition method, a depth impression, a display method, and the like. The three-dimensional video is represented by image-based representation and mesh-based representation.

Recently, Death Image-Based Rendering (DIBR) has been in the spotlight as a method of expressing such three-dimensional video. Depth-based rendering refers to a method of creating scenes at different viewpoints using reference images having information such as depth or difference angle for each pixel. This depth image based rendering not only renders difficult and complex shapes of the 3D model, but also enables the application of signal processing methods such as general image filtering, and has the advantage of generating high quality 3D video. have. The depth image based rendering uses a depth image and a texture image acquired through a depth camera and a multi-view camera.

The depth image is an image representing a distance between an object located in a three-dimensional space and a camera photographing the object in black and white units. Such depth images are widely used in 3D reconstruction technology or 3D warping technology through depth information and camera parameters. In addition, depth images are applied to free-view TVs and 3D TVs. A free view TV refers to a TV that enables a user to watch an image at an arbitrary point of time according to a user's selection without viewing the image only at a predetermined point in time. 3D TV realizes realistic image by adding depth image to existing 2D TV, and research and development is actively performed recently.

In order to smoothly change the viewpoint in such a free-view TV and 3D TV, an improved intermediate image should be generated, and it is important to estimate an accurate depth map. In order to estimate the depth map, a stereo matching algorithm is generally used. However, in the conventional stereo matching algorithm, many errors are generated around pixels having a discontinuity point of depth, and this error causes a problem that an object boundary overlaps or becomes unclear when generating an intermediate image. In addition, since the conventional stereo matching algorithm searches only the horizontal direction in the surrounding image to obtain the shift value, only the image obtained by the parallel camera configuration or the image undergoing the rectification process can be input. Therefore, according to this method, there is a problem in estimating a depth map for a multi-view image having various camera configurations such as a circular camera configuration as well as a parallel camera configuration. Moreover, according to the conventional stereo matching algorithm, it is suitable for stereo images by searching for the variance value in pixel units. However, in the case of multi-view images, which have a relatively large amount of data compared to the stereo image, there are many errors when searching for variance values in pixel units. In addition to the inclusion, there is a problem that the complexity is increased.

The present invention has been devised to solve the above problems, and in particular, it is possible to obtain a depth map that reduces errors, reduces complexity, and is robust against external influences, and generates intermediate images using such depth maps and uses them for encoding multiview video. Accordingly, an object of the present invention is to provide a depth map estimation apparatus and method for obtaining smooth viewpoint switching and improved encoding efficiency, an intermediate image generation method, and a multi-view video encoding method.

In the present invention, while estimating the depth value in units of segments using a three-dimensional warping and a trust propagation method, the following are proposed.

① Matching function weighted to MGRAD

Since the multi-view image is captured by multiple cameras, there is a color difference between the viewpoints. In order to consider the color difference between these viewpoints in the matching function, a weight ω that adjusts the ratio of MGRAD is added.

C (x, y, d) = C _MAD (x, y, d) + ω x C _MGRAD (x, y, d)

In this case, the weight ω is determined due to the difference between the average luminance values of the two time points to be compared.

DC _C is the mean luminance value at the midpoint, and DC _ref is the average luminance value at the time of comparison.

By setting the matching function as described above, the MGRAD using the gradient map is added to the existing MAD function, but the weight to determine the specific gravity of the MGRAD is automatically set.

② Matching function considering temporal correlation

Existing depth map estimation methods search depth maps independently for each frame of the image, and thus the temporal consistency of the depth maps is inferior, and even if the refinement process is performed, depth error still occurs.

In the present invention, in order to improve the temporal correlation and the reliability of the depth value in the process of searching the initial depth map, the weighting function C _temp (x, y) that considers the depth value searched in the previous frame to the self-adaptive function. , add d).

C (x, y, d) = C _MAD (x, y, d) + ω x C _MGRAD (x, y, d) + C _temp (x, y, d)

here,

λ is the slope of the weight function, D _prev (x, y) is the depth value of the previous frame

As such, the weight is adjusted in consideration of the depth value of the corresponding pixel between adjacent frames.

According to the present invention, a segment is formed between similar pixels to segment an area of a target image, and then 3D warping is applied to search for a depth value, thereby reducing the error of the depth value generated in the process of converting from a disparity map to a depth map and reducing complexity. By using the self-adaptive function that adds the extended gradient map to the depth value, the depth map robust to external influences such as color mismatch between cameras can be obtained.

In addition, according to the present invention, by applying a matching function to both the left image and the right image in the depth value search process for one segment, a more accurate depth value can be obtained by solving the problem of the occlusion area that may occur when only one image is used. It can be effective.

In addition, according to the present invention, by resolving the initial depth map using a segment-based reliability diffusion method, an error that may occur due to color similarity of pixels existing in the background is removed, and a depth acquired by the existing depth map estimation technique. Compared to the map, it is possible to obtain a relatively clear object boundary.

In addition, according to the present invention, by improving the accuracy of the depth map, an intermediate image with improved image quality can be obtained, thereby enabling smoother viewpoint switching in 3D TV, free view TV, and the like.

In addition, according to the present invention, the encoding efficiency can be improved by providing a reference image having a higher spatial correlation when encoding a multiview video through an improved intermediate image.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. First, in adding reference numerals to the components of each drawing, it should be noted that the same reference numerals are assigned to the same components as much as possible, even if shown on different drawings. In addition, in describing the present invention, when it is determined that the detailed description of the related well-known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, the following will describe a preferred embodiment of the present invention, but the technical idea of the present invention is not limited thereto and may be variously modified and modified by those skilled in the art.

First, a depth map estimation apparatus and a depth map estimation method according to a preferred embodiment of the present invention will be described.

1 is a block diagram of a depth map estimation apparatus according to a preferred embodiment of the present invention, Figure 6 is a flowchart of a depth map estimation method according to a preferred embodiment of the present invention.

Referring to FIG. 1, a depth map estimation apparatus according to an exemplary embodiment of the present invention may include a multiview image storage unit 10, a segment setting unit 20, a three-dimensional warping unit 30, and an initial depth value search unit ( 40 and a depth value refiner 50.

Also, in the depth map estimation method according to the preferred embodiment of the present invention, referring to FIG. 5, a multiview image stored through a multiview video camera (S10) is stored in step 10. In the target image for which the depth map is to be estimated, a segment is set between pixels whose luminance difference between adjacent pixels is less than or equal to a threshold value (S20). 3D warping to a viewpoint of an adjacent reference image to obtain a warped target image (S30), estimating a depth value of the warped target image based on a reference image, but one depth value for each segment In step (S40) and refine the depth value calculated in step 40, but using a confidence propagation (Belief Propagation) method in units of segments set in step 20 It comprises a (S50). Since the depth map estimation method will be fully described in the description of the depth map estimation apparatus according to the preferred embodiment of the present invention, only the depth map estimation apparatus will be described below.

The multiview image storage unit 10 stores a multiview image input through a multiview video camera. The multi-view image storage unit 10 stores at least as many images as the number of video cameras in the view direction, and stores consecutive images in the time direction for each of these views.

The segment setting unit 20 performs luminance or chrominance between adjacent pixels in an image (hereinafter, referred to as a “target image”) to estimate a depth map among multiview images stored in the multiview image storage unit 10. (chrominance) serves to set segments between pixels whose difference is less than or equal to a threshold. That is, the segment setting unit 20 divides the image into segments by grouping pixels having similar luminance or color information in the target image. At this time, it is assumed that the depth value changes slightly inside the segment, and the discontinuity point of the depth value is generated at the boundary of the segment. In order to satisfy this assumption, the segment is preferably subdivided as small as possible, and for this purpose, it is desirable to make the threshold small. As the threshold increases, the range of similar luminance increases, which may increase the size of the segment, thereby increasing the probability of including the discontinuity point of the depth value in the segment. For example, the threshold can be set to zero. An example of an image in which similar pixels are divided into segments by the segment setting unit 20 is illustrated in FIG. 2.

The 3D warping unit 30 performs a 3D warping on each segment set by the segment setting unit 20 to the viewpoint of the reference image adjacent to the target image to obtain a “warped target image”. Perform. The conventional stereo matching algorithm searches only the horizontal direction from the surrounding images to find the variation, so only the images obtained from the parallel camera configuration or the processed images can be input as inputs. There is a limit in estimating the depth map for a multiview image with. In order to solve this problem, the 3D warping unit 30 generates a warped target image by projecting a specific segment of the target image at the time to obtain the depth map to the surrounding image through 3D warping. Therefore, the three-dimensional warping unit 30 can estimate the depth map independently of the camera configuration so as not to be bound by the camera configuration as mentioned above. 3 illustrates a 3D warping process performed by the 3D warping unit 30 and a depth value search method using the warped target image and the reference image generated accordingly. In FIG. 3, the solid yellow area on the left is a specific segment of the target image for which the depth map is to be obtained, the solid yellow area on the right is a corresponding segment of the reference image, and the hollow yellow area on the right is a warped target image. Corresponds to a specific segment.

The initial depth value searcher 40 estimates a depth value of the warped target image based on the reference image, and searches for one depth value for each segment. Since multi-view images are relatively large and have a large amount of data compared to stereo matching test images, when searching for the variance value by pixel as in the conventional stereo matching algorithm, not only the error but also the complexity increases. Thus, the initial depth value searcher 40 searches for the depth value in segments instead of pixels.

In order to search the depth value of the target image segment with respect to the reference image segment, a matching function is required. Commonly used matching functions include Squared Intensity Difference (SD) and Absolute Intensity Difference (AD). However, there is a drawback that these general matching functions are sensitive to color mismatch between cameras. Therefore, the initial depth value searcher 40 searches for the initial depth value by using a Self Adaptation Function in which a function using a gradient map is added to the AD function. The self-adaptive function is given by Equation 1 below.

Where ω is a weight greater than 0 and less than 1, x and y are positions of pixels in a segment of the target image, d is displacement, and C _MAD is an average of luminance difference values of segments according to each measurement displacement (Mean Absolute Difference). , C _MGRAD means the average of the gradient difference of the segment according to each measured displacement.

In addition, C _MAD is a conventional AD function is given by Equation 2 below.

In addition, C _MGRAD is a function of the gradient map considering four directions is given by Equation 3 below.

Here, M is the number of pixels in the segment, S _k is the corresponding segment, I ₁ (x, y) is the luminance value of the pixel at the (x, y) position in the target image, I ₂ (x ', y' ) Is the luminance value of the pixel at the position (x ', y') in the reference image, ∇ _x , ∇ _y , ∇ _-x , ∇ _-y are + x direction, + y direction, -x direction,- It means the slope map in the y direction.

The self-adaptive function according to the preferred embodiment of the present invention uses the gradient maps in the + x direction and the + y direction so as not to be influenced by external factors such as color mismatch between cameras. The rigidity of the gradient map is enhanced by using the gradient map for all four directions extending in the x direction and the -y direction.

In addition, the self-adaptation function according to the preferred embodiment of the present invention may further be considered.

① Matching function weighted to MGRAD

Since the multi-view image is captured by multiple cameras, there is a color difference between the viewpoints. In order to consider the color difference between these viewpoints in the matching function, a weight ω to adjust the ratio of MGRAD is added.

C (x, y, d) = C _MAD (x, y, d) + ω x C _MGRAD (x, y, d)

In this case, the weight ω is determined due to the difference between the average luminance values of the two time points to be compared.

DC _C is the mean luminance value at the midpoint, and DC _ref is the average luminance value at the time of comparison.

By setting the matching function as described above, the MGRAD using the gradient map is added to the existing MAD function, but the weight to determine the specific gravity of the MGRAD is automatically set.

② Matching function considering temporal correlation

Existing depth map estimation methods search depth maps independently for each frame of the image, and thus the temporal consistency of the depth maps is inferior, and even if the refinement process is performed, depth error still occurs.

In the present invention, in order to improve the temporal correlation and the reliability of the depth value in the process of searching the initial depth map, the weighting function C _temp (x, y) that considers the depth value searched in the previous frame to the self-adaptive function. , add d).

C (x, y, d) = C _MAD (x, y, d) + ω x C _MGRAD (x, y, d) + C _temp (x, y, d)

here,

λ is the slope of the weight function, D _prev (x, y) is the depth value of the previous frame

As such, the weight is adjusted in consideration of the depth value of the corresponding pixel between adjacent frames.

In addition, the initial depth value searcher 40 may use both the left image and the right image as the peripheral reference image. That is, the initial depth value searcher 40 may obtain a more accurate depth value by applying a matching function to both the left image and the right image during the depth value search process for one segment. When searching for a depth value using only one image, as in the conventional stereo matching algorithm, an occlusion may occur in which a specific part is covered. However, when the object existing in the target image is covered by another object in the left image, the initial depth search unit 40 may search for the correct depth value by preventing the occurrence of the occlusion area by using the right image instead of the left image. To help.

The depth value refiner 50 performs a role of obtaining a final depth value by reducing an error by refining the initial depth value calculated by the initial depth value searcher 40. The initial depth value obtained through the initial depth value searcher 40 may have an error in some cases. For example, when searching for a depth value of a background inside an image, the color difference between pixels existing in the background is generally not so large. Situations that can be perceived as values can occur. As a method for solving this problem, there are refinement methods such as graph cut and dynamic programming, but there is a problem in that the performance is slightly reduced. The depth refiner 50 reduces an error (especially, an error in the background) of the initial depth map by using a trust propagation method in units of segments.

4A is a conceptual diagram illustrating a method of spreading confidence in units of pixels. The pixel-by-pixel confidence diffusion method of FIG. 4A has recently been evaluated as having superior performance compared to other refinement methods, and sends a message to adjacent pixels on the top, bottom, left, and right sides to consider the depth value of neighboring pixels (PF Felzenszwalb, DP Huttenlocher, "Efficient Belief Propagation for Early Vision"). That is, when sending a message from the current pixel to the surrounding pixel (bold arrow), the message from the neighboring pixel to the current pixel (dashed arrow) is used. The pixel-based confidence diffusion method is an energy function that considers data cost, which is the cost of assigning a label to a specific pixel, and discontinuity cost, which is the cost of assigning a label to two neighboring pixels. Is introduced and the message update is repeated using the Grid Graph.

4B is a conceptual diagram illustrating a method of spreading confidence in segments.

The confidence diffusion method of the segment unit performed by the depth value refiner 50 is based on the above-described confidence diffusion method of the pixel unit, but there is a difference that a unit for sending a message is a segment instead of a pixel. When the depth value refiner 50 sends a message from the current segment to the surrounding segment (bold arrow), the depth refiner 50 uses the message (dashed arrow) from the surrounding segment to the current segment. That is, since the initial depth value is estimated in the segment unit through the initial depth value search unit 40, in order to refine the initial depth value, a confidence diffusion method in the segment unit is used. The depth value refiner 50 reduces the error by applying the reliability diffusion method in the unit of segment to refine the initial depth value.

An example of an initial depth map is shown in FIG. 5A, and an example of a final depth map is shown in FIG. 5B. Referring to FIGS. 5A and 5B, it can be seen that the boundary of the object is relatively clear compared to the depth map obtained by the conventional depth map estimation technique. Compared to the initial depth map, the final depth map has an error (especially through a refinement process). , The background part) is significantly reduced.

Next, an intermediate image generating method according to a preferred embodiment of the present invention will be described.

In the intermediate image generating method according to the preferred embodiment of the present invention, an intermediate view image is generated by a depth image-based rendering (DIBR) technique. The DIBR technique is a technique of rendering an image at an arbitrary time point using a depth image composed of a texture image and distance information corresponding to each pixel of the texture image. After scene and modeling the color and depth image in a three-dimensional mesh (mesh) using a Cartesian coordinate system, a virtual camera is used to render the image at any point in time. In this case, the depth image used may be directly obtained by using a depth camera, but it is preferable to use the depth image generated by the above-described method. Here, the intermediate image generation method is not particularly limited, and the intermediate image is improved by generating the intermediate image using a general DIBR technique through the multiview image obtained from the multiview video camera and the depth map obtained by the method. You can get a video.

In addition, various preprocessing methods have been proposed to generate a multiview image from a depth image and a texture image. For example, Zhang applies an asymmetric Gaussian filter to the entire depth image, then applies three-dimensional warping and hole-filling to remove the non-occluded regions, thereby reducing the geometric folds caused by the non-occluded and symmetric Gaussian filters. Improved the quality of intermediate images.

In addition, a hierarchical natural-textured mesh stream (HNTMS) using a three-dimensional mesh structure has been proposed in addition to the DIBR technique. Render the images sequentially. In the intermediate image generating method according to the preferred embodiment of the present invention, these techniques may be selectively applied.

Next, an encoding method according to a preferred embodiment of the present invention will be described.

7 is a flowchart of an encoding method according to a preferred embodiment of the present invention.

In the encoding method according to the preferred embodiment of the present invention, referring to FIG. 6, a target image input step (S100), an intermediate image is added to a reference image list (S200), a motion vector is calculated (S300), Computing the difference of the target image with respect to the reference image (S400), DCT processing step (S500), quantization step (S600), and entropy encoding step (S700).

In step 100, a target image to be encoded of a multiview video is received. Video feeds received through transmission lines from multiple video cameras are temporarily stored by the capture buffer after capturing them.

In step 200, an intermediate image is generated and added to the reference image list. In this case, the generation of the intermediate image is preferably performed by the intermediate image synthesis method according to the above-described preferred embodiment of the present invention. This is because the encoding efficiency of the multiview video is better when there is a reference picture similar to the target picture to be encoded and when there is a reference picture with a higher quality.

The reason for using an intermediate image for encoding a multiview video is as follows. For example, when the B screen is encoded, if the screens of the left and right views are already encoded, the images of the intermediate views may be generated with reference to these images. The generated intermediate image is a highly correlated image because the viewpoint is the same as the target image to be encoded. Therefore, when the intermediate image is used as a reference image of the encoding process, encoding efficiency can be improved due to a high correlation with the target image.

In operation 300, a motion vector of the target image is calculated based on the reference image of the reference image list. In this case, the target image is an image input in step 100, and the reference image is an image stored in the reference image list including the intermediate image generated from the depth map. The motion of the target image is estimated using the reference image, and a motion vector of a corresponding block of the target image is calculated.

In operation 400, a difference value of the target image with respect to the reference image that is motion compensated by the motion vector calculated in operation 300 is obtained. In operation 400, a prediction image is formed by performing motion compensation on a reference image using a motion vector, and a difference matrix between the target image and the prediction image is calculated.

Step 500 is a step of obtaining discrete cosine transform (DCT) coefficients by discrete cosine transforming the difference matrix.

Step 600 quantizes the DCT coefficients obtained through step 500.

In step 700, entropy coding is performed on a quantized DCT coefficient by using a method such as Context Adaptive Variable Length Codes (CAVLC) or Context Adaptive Binary Arithmetic Coding (CABAC). Entropy coded feeds are sent to the external network via a buffer or the like.

The above description is merely illustrative of the technical idea of the present invention, and various modifications, changes, and substitutions may be made by those skilled in the art without departing from the essential characteristics of the present invention. will be. Accordingly, the embodiments disclosed in the present invention and the accompanying drawings are not intended to limit the technical spirit of the present invention but to describe the present invention, and the scope of the technical idea of the present invention is not limited by the embodiments and the accompanying drawings. . The scope of protection of the present invention should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

The present invention calculates the initial depth value in the unit of segment, refines it by the segment by applying the reliability diffusion method, calculates the final depth map, and obtains the intermediate image of the improved image quality by using the stereoscopic TV, free viewpoint TV, surveillance It can be widely used for camera images.

1 is a block diagram of a depth map estimation apparatus according to an embodiment of the present invention;

FIG. 2 is a diagram illustrating an example of region division through a segment setting unit of FIG. 1; FIG.

3 is a conceptual diagram of three-dimensional warping performed in the three-dimensional warping unit of FIG.

4A is a conceptual diagram of a conventional pixel-by-pixel reliability diffusion method;

4B is a conceptual diagram of a method of spreading reliability in units of segments performed by the depth refiner of FIG. 1;

5A and 5B show examples of an initial depth map and a final depth map, respectively;

6 is a flowchart of a depth map estimation method according to a preferred embodiment of the present invention;

7 is a flowchart of an encoding method according to a preferred embodiment of the present invention.

<Explanation of symbols for the main parts of the drawings>

10-Multiview image storage unit 20-Segment setting unit

30-3D warping unit 40-Initial depth search unit

50-Depth Refiner

Claims (2)

(a) dividing the region into segments based on the similarity of the pixels; (b) obtaining an initial depth map in units of segments by using a three-dimensional warping technique and a self-adaptive function in which an extended gradient map is reflected; And (c) purifying the initial depth map by performing a confidence diffusion method in segments; Depth map estimation method comprising a. A divider dividing the region into segments based on the similarity of the pixels; An acquisition unit obtaining an initial depth map in units of segments by using a three-dimensional warping technique and a self-adaptive function in which an extended gradient map is reflected; And Refiner that refines initial depth map by performing confidence diffusion method in segment units Depth map estimation device comprising a.

Images