KR20150078273A - Device and method for generating stereo depth images - Google Patents

Abstract

Disclosed is a method for generating a high speed stereo depth image. According to the method, a stereo image is downsampled to reduce complexity of a depth image generation process, and a low resolution left viewpoint depth image is obtained by applying a stereo matching technology. Then, through viewpoint change and a stratified hole filling method, a low resolution right viewpoint depth image is generated. The obtained low resolution stereo depth image may be restored to an original resolution through an image up-sampling method.

Description

TECHNICAL FIELD [0001] The present invention relates to a stereo depth image generating apparatus and a stereo depth image generating method.

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus and a method for generating stereo depth images or stereo depth maps, and more particularly, to an apparatus and method for generating a high-speed stereo depth image using video up-sampling .

Depth-Image-Based Rendering (DIBR) has attracted much attention as a core technology of 3D video systems. DIBR technology is a technique to generate arbitrary virtual viewpoint images from color images and corresponding depth images. With DIBR technology, the user can select and view the desired point of view.

Among various methods of acquiring three-dimensional images, a passive sensor method is widely used in which a plurality of cameras are arranged in order to acquire images at various points of view and acquire depth images using binocular parallax. This method can provide a wide viewing angle to a user from a muti-view color image, but has a complicated disadvantage of a stereo matching technique used to acquire a depth image. Therefore, it is essential to search the stereo depth image accurately and quickly to enable DIBR technology from acquired 3D images.

Accordingly, it is an object of the present invention to provide an apparatus and method for generating a stereo depth image that can reduce the complexity in the process of generating a stereo depth image.

According to another aspect of the present invention, there is provided an apparatus for generating a stereo depth image, including: an image acquiring unit acquiring a stereo image including a left view image and a right view image; A stereo matching unit for acquiring a depth image of the left or right viewpoint by performing a down-sampling operation on the viewpoint-converted depth image, A hierarchical hole filling unit filling a hole region in the obtained layered depth image with a depth value of a peripheral region corresponding to the hole region in a depth image of an upper layer, And an up-sampling unit for up-sampling the depth image to generate a stereo depth image.

According to another aspect of the present invention, there is provided a stereo depth image generation method including: obtaining a stereo image including a left view image and a right view image; performing a stereo matching process on the stereo image, A step of converting the viewpoint of the generated depth image, a step of converting the viewpoint-converted depth image into a hierarchical depth image by downsampling the viewpoint-converted depth image, Filling a depth image of the upper layer with reference to a depth value of a peripheral region corresponding to the hole region, and generating a stereo depth image by upsampling the hole filling depth image obtained through the hole filling process .

According to the present invention, a stereo depth image can be obtained at a high speed by performing stereo matching only once using a stereo image converted to a low resolution and generating a stereo depth image using viewpoint conversion, hierarchical hole filling, have.

1 is a block diagram illustrating a configuration of a stereo depth image generating apparatus according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating a viewpoint conversion process performed by the viewpoint conversion unit shown in FIG. 1. Referring to FIG.
FIG. 3 is a view illustrating a hierarchical hole filling process performed in the hierarchical hole filling portion shown in FIG. 1. FIG.
4 is a flowchart illustrating a method of generating a stereo depth image according to an exemplary embodiment of the present invention.

The existing stereo depth image generation method performs a stereo matching technique with high complexity for each of the right and left viewpoints. Therefore, it takes considerable time to acquire the depth image.

Accordingly, the present invention provides a method of performing a stereo matching method only once using a low-resolution converted stereo image, and generating a stereo depth image using viewpoint conversion, hierarchical hole filling, and up-sampling techniques. By doing so, the stereo depth image can be acquired at high speed, and the corresponding technology can be utilized in various applications such as a 3D video system using a stereo depth image.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. .

Depth image

The production of three-dimensional images basically starts with shooting or producing stereoscopic images. In general, images taken from two cameras placed at a distance of about 6.5 cm, which is the average distance between two eyes of a person, have disparity, which is parallax information. When the image is viewed through a binocular monitor, Unlike ordinary monitors that provide images at the same time in both eyes, viewers can feel the stereoscopic effect. However, multi-view images that are photographed at various viewpoints of one scene are required for producing a more realistic 3D image that is robust to the viewing environment. Multi-view images can deliver a more stable stereoscopic effect than a binocular image and provide a relatively wide viewing range. Also, it is possible to create a free-view stereoscopic image which can generate a smooth transition between viewpoints by generating depth-of-view depth information for a multi-viewpoint image and then generating an image of a virtual viewpoint between the viewpoint and the viewpoint using the information. Therefore, it is important to acquire scene depth information in order to produce a high-quality three-dimensional image.

The depth information of such a scene represents a distance from an object to be photographed by the camera, and is generally represented by a depth image which is a monochrome image (or a gray image) having 256 steps.

This depth image can be represented as another type of displacement image. When a pixel within a viewpoint based on two viewpoints corresponds to a certain pixel located on a parallel line of an adjacent viewpoint, the distance between the pixels is Is defined as an image having a pixel value.

There are passive, active, and mixed methods to obtain depth information of a scene. Stereo matching is a typical method of passive method, and depth information is derived from two images. The active method measures the physical distance using a device such as a depth camera and outputs the image to acquire depth information. The above two methods have a complementary relationship with respect to complexity, performance, and cost. The hybrid method is a method that can generate high quality results while reducing the complexity by simultaneously using two mutually complementary methods. In the following embodiment, a passive method, that is, a method of generating a depth image using stereo matching is exemplified.

Camera parameters ( camera parameter )

In order to generate a multi-view image using at least two cameras, the positional correlation between the cameras must be precisely defined. Especially, when the correlation between pixels is defined by each point using depth information, accurate parameters of each camera are used. The method of estimating the camera parameters of the multi-view camera is called camera calibration. Camera parameters consist of intrinsic parameters indicating camera internal structure and extrinsic parameters indicating camera direction and position. The camera internal variable is a 3x3 matrix composed of variables such as focal length and position of the optical axis. The camera external variables consist of a 3x3 directional matrix representing the direction pointed by the camera and a 3x1 transition vector pointing to the camera's three-dimensional position. Unlike camera calibration of a single-view camera, camera calibration of a multi-view camera requires a more precise estimation process since the camera-to-camera position relationship must be clear, and the Camera Calibration Toolbox (http: //www.vision.caltech.edu/bougue, Camera Calibration Toolbox for MATLAB).

Image (color image) preprocessing process

For the images taken after camera calibration, the quality of the image can be improved through several preprocessing steps and the efficiency in the subsequent processing can be increased. First, for color images, it is necessary to correct the geometric error and the color difference generated in the stereo image. The geometric errors are caused by differences in camera position, error in placement, and internal characteristics, and are represented by mismatches of the vertical coordinate values of corresponding points between the viewpoints. Also, since the two image planes are not parallel but distorted, the basic condition for obtaining the displacement value is not established. Therefore, this error is minimized through image rectification. Binocular image alignment converts two image planes to parallel and horizontal motion only. Also, by matching the camera variables, the aligned binocular images have characteristics of the images taken from the same camera with only horizontal intervals.

Hole Fill ( hole filling ) process

In the following embodiments, a viewpoint transformation process is performed on the generated depth image. For example, when the depth image of the left viewpoint is transformed into the depth image of the right viewpoint, a hole that is hidden at the left viewpoint, hole region or a disocclusion region. Here, a hole region or a non-occlusion region (disocclusion) may be defined as at least one pixel region in which a depth value is empty in a viewpoint-transformed depth image when a depth image is transformed into a viewpoint, The filling process of regions is called hole filling process. The hole filling process may be a process of filling the depth value of the neighboring pixel region adjacent to the pixel region where the depth value is empty by using the similarity with the neighboring pixel values.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention and the manner of achieving them will become apparent with reference to the embodiments described in detail below with reference to the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the invention to those skilled in the art. And the present invention is defined by the description of the claims. It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In the present specification, the singular form includes plural forms unless otherwise specified in the specification. It is noted that " comprises, " or "comprising," as used herein, means the presence or absence of one or more other components, steps, operations, and / Do not exclude the addition.

1 is a block diagram of an internal configuration of a stereo depth image generating apparatus according to an embodiment of the present invention.

Referring to FIG. 1, the apparatus 100 for generating a stereo depth image according to an exemplary embodiment of the present invention performs a stereo matching process on a low-resolution converted stereo image, And an up-sampling process to generate a stereo depth image at a high speed.

To this end, the stereo depth image generation apparatus 100 includes an image acquisition unit 110, an image input unit 112, a preprocessor 114, a down-sampling unit 116, a stereo matching unit 118, a viewpoint transformation unit 120 A hierarchical hole filling unit 122, an up-sampling unit 124, and a video output unit 126. [0033] FIG.

The image acquiring unit 110 acquires a stereo image composed of a left view image CL and a right view image CR by capturing an arbitrary object at a left or right viewpoint, Camera. When the image acquisition unit 110 is composed of two cameras, one camera captures the arbitrary object at the right viewpoint to generate a right viewpoint image CR, and the other camera captures the arbitrary object at the left viewpoint And generates a left view image CL. Here, in CR and CL, C is an abbreviation of Color, and subscripts R and L denote abbreviations of 'Right' and 'Left', respectively.

The image input unit 112 receives the left view image CL and the right view image CR and transmits the left view image CL and the right view image CR to the preprocessor 114. The image input unit 112 may be a frame buffer.

The preprocessing unit 114 obtains a camera parameter (CP) using a pattern image acquired from a manually aligned stereo camera, and obtains the left view image (CP) using the obtained camera parameter CL) and the right view image (CR) into a parallel camera array image.

The down-sampling unit 116 is a unit for converting a stereo image to a low resolution in order to reduce the amount of calculation in the stereo matching unit 118. The down-sampling unit 116 includes a first down-sampling unit 116-1 and a second down- And a down-sampling unit 116-2. The first down-sampling unit 116-1 down-samples the rearranged right view image CR 'by the preprocessor 114 to output a downsampled right view image CR' ', The second down-sampling unit 116-2 down-samples the rearranged left image CL 'by the preprocessing unit 114 and outputs the downsampled left image CL' '.

The stereo matching unit 118 performs a stereo matching process using the binocular parallax between the right view image CR '' and the left view image CL '', which is downsampled by the down-sampling unit 116, And generates a depth image for one of the viewpoints and the right viewpoint. In this embodiment, it is assumed that the left viewpoint depth image is generated, and therefore, the stereo matching process for generating the right viewpoint depth image is not performed.

The viewpoint conversion unit 120 is configured to convert the viewpoint of the left viewpoint depth image generated by the stereo matching unit 118 to right viewpoint. In the present embodiment, the pre- Since the view image (CR) is arranged in a parallel arrangement, the depth value is moved simply in the horizontal direction. For example, when the horizontal direction is the X axis direction and the vertical direction is the Y axis direction, the movement of the depth value occurs in the X axis direction and does not occur in the Y axis direction, as shown in FIG.

The hierarchical hole filling unit 122 is configured to fill the discocclusion hidden at the left point of view and the discocclusion revealed at the right point in the viewpoint conversion process by the viewpoint conversion unit 120 with adjacent depth values, The greater the difference in value is, the larger the boundary area becomes. The hierarchical hole filling method is used in this embodiment in order to effectively fill it with a fast time. This hierarchical hole filling method will be described in detail below with reference to FIG.

The up-sampling unit 124 reconstructs the left-point depth image DL down-sampled by the down-sampling unit 116 described above to the original resolution, and performs a viewpoint conversion process and a hierarchical hole filling process A first up-sampling unit 124-1 for restoring the downsampled left-point depth image DL to the original resolution, a second downsampling unit 124-1 for performing a view- And a second up-sampling unit 124-2 for reconstructing the image DL. This up-sampling process can be restored by referring to the peripheral value of the left viewpoint depth image DL. Alternatively, the up-sampling process can be performed by referring to the boundary information in the color images CR 'and CL' of the original resolution that have undergone the preprocessing process of the preprocessing unit 114 described above. By using the boundary information in the color image (CR ', CL') of the original resolution, the boundary can be restored more accurately when the depth image is restored. In this embodiment, the up-sampling process using the boundary information in the color image CR ', CL' having the original resolution is applied. For this purpose, the first up-sampling unit 124-1 performs a down- Sampled (or restored) image by using the left-view image CL 'of the original resolution, which has been subjected to the preprocessing process, in addition to the depth image DL of the original resolution, The second up-sampling unit 124-2 receives the right-view image CR 'of the original resolution that has undergone the preprocessing process and outputs the right-view image CR' (Or restored) depth image DL 'using the view image CR'.

The image output unit 126 receives the left and right depth images DL 'and DR' up-sampled by the up-sampling unit 124 and outputs it as a stereo depth image DI at a high speed.

FIG. 3 is a view for explaining a hierarchical hole filling process performed in the hierarchical hole filling unit shown in FIG. 1. FIG.

As described above, in the hole filling process, the discocclusion increases as the boundary region having a large difference in depth values is filled, and it is necessary to fill it in a short time effectively. For this purpose, a hierarchical hole filling method is used. In this hierarchical hole filling process, the resolution of the viewpoint-converted depth image is lowered step by step (or hierarchically) It means a method of filling a hole existing in a high-resolution depth image.

In the embodiment of FIG. 3, in the process (1), the color image (34) of the level 1 whose resolution is lowered through the first downsampling process for the color image (30) and the depth image (32) And the depth image 36 are generated.

In step 2, a process of generating a color image 38 and a depth image 40 of a level 2 in which the resolution of the level 1 color image 34 and the depth image 36 is lowered through a second downsampling process .

In the process (3), a process of filling a hole of the depth image 40 of level 2 is performed. At this time, the hole filling depth image 42 generated by the process (3) is a reference for generating a hole filling depth image at the level 1 It is used as an image.

In the process (4), the process of converting the viewpoint of the depth image (36) of level 1 is performed.

In step 5, referring to the hole filling depth image 42 of the level 2, a hole filling process is performed for the depth image 36 of the level 1, (46) is generated.

In step (6), a viewpoint conversion process is performed on the depth image (32) of level 0.

In step 7, referring to the hole fill depth image 46 of level 1, a hole filling process is performed on the depth image 32 of level 0 that has been time-shifted by the process 6, A process of generating an image 50 is performed.

Meanwhile, the color images 30, 34, and 38 are used in the hole filling process for each level of ③, ⑤ and ⑦. The hole in the foreground must fill the hole with the depth value of the adjacent pixel in the foreground, Holes present in the background should fill the holes with depth values of adjacent pixels present in the background. Since it is difficult to distinguish whether the hole is foreground or background, it is necessary to use color images together. Therefore, in the hole filling process for each level of (3), (5), and (7), the hole filling process is performed for each layer by referring to the downsampled color image for each layer (step or level).

As described above, in the hole filling process for each layer according to the embodiment of the present invention, the hole area is sporadically filled at the upper level of the low resolution, and the hole filling depth image at the higher level, And performs a hole filling process to refill the area. Repeatedly performing this process up to the level of the original resolution makes it possible to fill the hole for all areas of the depth image.

4 is a flowchart illustrating a method of generating a stereo depth image according to an exemplary embodiment of the present invention.

Referring to FIG. 4, in step S410, a stereo image including a left view image and a right view image obtained by capturing an arbitrary object using an image acquiring unit such as a stereo camera is performed. The acquired stereo image is converted to a lower resolution through a downsampling process.

In step S420, a stereo matching process is performed on the downsampled stereo image to acquire a depth image of a left or right viewpoint. At this time, The depth image of the right viewpoint is obtained.

In step S430, a viewpoint transformation process of transforming the viewpoint of the depth image is performed. For example, when the depth image of the left viewpoint is acquired in the process of acquiring the depth image, When the depth image of the right viewpoint is obtained in the process of acquiring the depth image, a process of converting the depth image of the obtained right viewpoint into the depth image of the left viewpoint is performed .

In step S440, a process of acquiring a layered depth image by downsampling the viewpoint-converted depth image is performed. The process of acquiring the layered depth image is performed by the viewpoint transformation unit 120 or the layer hole filling And a block for performing downsampling may be additionally provided in the viewpoint converting unit 120 or the layer hole filling unit 122 although not shown in the drawing.

In step S450, a hole filling process is performed to fill the hole area in the obtained depth image with reference to the depth value of the peripheral area corresponding to the hole area in the depth image of the upper layer. For example, although not particularly limited, the depth image may include a depth image of a first hierarchical level having a first resolution, a depth image of a second hierarchical level having a second resolution lower than the first resolution, The depth information of the third layer having the depth value of the peripheral region of the hole region in the depth image of the third layer, Acquiring a hole filling depth image of a second hierarchical level filling the hole area in the depth image of the second hierarchy with reference to the obtained hole filling depth image of the third hierarchy; And acquiring a hole filling depth image of the first hierarchy filling the hole in the depth image of the first hierarchy with reference to the hole filling depth image of the second hierarchy And the first-layer hole-filling depth image is obtained as the final hole filling depth image.

In S460, up-sampling is performed to restore the resolution of the final acquired hole-depth image to the resolution originally obtained by the down-sampling process performed in S410, and the up- And is obtained as a final stereo depth image.

As described above, in the present invention, a stereo image is down-sampled, a stereo matching technique is applied to obtain a low-resolution depth image, and a low-resolution hole filling depth image is generated through a viewpoint conversion and a hierarchical hole filling method. The low-resolution hole filling depth image thus obtained is restored to its original resolution through an up-sampling process, and is generated as a stereo depth image. By doing so, the complexity of the stereo depth image generation process can be reduced, and the stereo depth image can be acquired at high speed.

In view of the above, those skilled in the art can understand and implement the above-described embodiments. In particular, all or some of the blocks in the method may be implemented in the associated hardware that is commanded by the program. The program may be stored in a computer-readable storage medium. When executing the program, one block or a combination thereof may be included in the method embodiment described above.

Claims (11)

A step of acquiring a stereo image including a left view image and a right view image;
Performing a stereo matching process on the stereo image to acquire a depth image of a left viewpoint or a right viewpoint;
A viewpoint transformation step of transforming the viewpoint of the generated depth image;
The depth-converted depth image is downsampled for each layer to obtain a layered depth image, and the acquired hole region in the layered depth image is referred to as a depth value of a surrounding region corresponding to the hole region in the depth image of the upper layer Filling process; And
A process of generating a stereo depth image by upsampling the hole filling depth image obtained through the hole filling process
And generating a stereo depth image.
The method of claim 1, wherein the obtaining of the depth image comprises:
And performing a stereo matching process once to acquire a depth image of the left or right viewpoint.
The method according to claim 1,
The depth image of the left viewpoint is converted into the depth image of the right viewpoint when the depth image of the left viewpoint is acquired in the process of acquiring the depth image,
Wherein when the depth image of the right viewpoint is acquired in the process of acquiring the depth image, the depth image of the obtained right viewpoint is converted into viewpoint depth image of the left viewpoint.
2. The method of claim 1, wherein in the hole filling step, the depth image includes a first layer depth image having a first resolution, a second layer depth image having a second resolution lower than the first resolution, Layered depth image having a third resolution lower than the resolution of the third layer,
In the hole filling process,
Acquiring a hole filling depth image of a third hierarchical level filling the hole area with a depth value of a peripheral area of the hole area in the depth image of the third hierarchical level;
Acquiring a hole filling depth image of a second layer filling the hole area in the depth image of the second hierarchy with reference to the obtained hole filling depth image of the third hierarchy; And
Acquiring a hole filling depth image of the first hierarchy filling the hole in the depth image of the first hierarchy with reference to the obtained hole filling depth image of the second hierarchy
And generating a stereo depth image.
The method of claim 4, wherein the step of generating the stereo depth image comprises:
And generating the stereo depth image by upsampling the first-layer hall-fill depth image.
6. The method according to any one of claims 1 to 5,
Further comprising the step of downsampling the acquired stereo image,
The step of acquiring the depth image comprises:
And performing a stereo matching process on the downsampled stereo image.
An image acquiring unit acquiring a stereo image including a left view image and a right view image;
A stereo matching unit for performing a stereo matching process on the stereo image to acquire a left or right depth image;
A point-of-view conversion unit for converting point-in-time of the generated depth image;
The depth-converted depth image is downsampled for each layer to obtain a layered depth image, and the acquired hole region in the layered depth image is referred to as a depth value of a surrounding region corresponding to the hole region in the depth image of the upper layer A layered hole filling part for filling the hole; And
An up-sampling unit for up-sampling the hole filling depth image obtained through the hole filling process to generate a stereo depth image;
And a stereo depth image generating unit.
8. The stereoscopic image display device according to claim 7,
And the stereo depth image is obtained by performing the stereo matching process once to obtain the layered depth image of the left or right viewpoint.
8. The apparatus according to claim 7,
Wherein when the depth image of the left viewpoint is acquired by the stereo matching unit, the acquired depth image of the left viewpoint is converted into a depth image of the right viewpoint,
Wherein when the depth image of the right viewpoint is acquired by the stereo matching unit, the depth image of the obtained right viewpoint is converted into viewpoint depth image of the left viewpoint.
2. The semiconductor device according to claim 1,
The depth image is divided into a first layer depth image having a first resolution and a second layer depth image having a second resolution lower than the first resolution and a third layer having a third resolution lower than the second resolution, When layered into a depth image,
A hole filling depth image of a third hierarchical level filling the hole region with a depth value of a peripheral region of the hole region in the depth image of the third hierarchical level is obtained, A depth-first image of the second layer filling the hole region in the depth image of the first layer, a hole-filling depth image of the second layer filling the hole region in the depth image of the second layer, And acquiring a hole depth depth image of the layer.
11. The apparatus of claim 10, wherein the up-
Wherein the stereo depth image generator generates the stereo depth image by upsampling the first-layer hole depth image.

Images