KR20120057542A - Occlusion layer extension - Google Patents

Abstract

PURPOSE: An occlusion layer extension method is provided to encode at least one occlusion layer of a layered depth image/video frame which has a wider width than the width of the layered depth image/video. CONSTITUTION: At least one occlusion layer of a layered depth image has a wider width than the width of a foreground layer of the layered depth image. The width of the occlusion layer is in proportion to the maximum disparity value in lateral boundary areas of a main depth map in the foreground layer. The lateral boundary areas are made of a fixed number of outmost columns of the main depth map.

Description

Occlusion layer extension {OCCLUSION LAYER EXTENSION}

The present invention relates to the art of encoding visible data in a layer depth format.

Layered depth image (LDI) is a method of encoding information to render three-dimensional images. Similarly, layered depth video (LDV) is a method of encoding information to render three-dimensional videos.

LDI / LDV uses a foreground layer and at least one background layer to convey information. The background layer is also called the occlusion layer. The foreground layer includes a main color image / video frame with an associated main depth map. At least one background layer comprises a background color image / video frame with an associated background depth map. Normally, the occlusion layer is sparse in that the occlusion layer only contains image depth covered by the foreground objects in the main layer and corresponding depth information of the image content occluded by the foreground objects. ).

The way to generate LDI or LDV is to capture the same scene using two or more cameras from different viewpoints. Images / videos captured by two cameras are then warped, i.e. shifted, to fuse to produce a main image / video depicting the same scene from a central viewpoint positioned between different viewpoints. )do.

Moreover, a main depth map associated with the main image / video frame can be generated using the two captured images / video frames. The main depth map assigns a uniform scaled value in depth value, disparity value, or disparity for each pixel of the main image / video frame, where the assigned disparity value is from the main image plane. It is inversely proportional to the distance to the object to which each pixel belongs.

[Summary of invention]

According to the prior art, the foreground layer and the background layer have the same horizontal width. We have found that this same size does not allow conveying all the information provided in images / videos captured by at least two cameras.

Therefore, the inventors have identified a layered depth image / video frame that allows at least one occlusion layer of the layered depth image / video frame to have a horizontal width greater than the foreground layer of the layered depth image / video frame. We propose a data structure for which the horizontal width of the occlusion layer is proportional to the maximum disparity value contained in the lateral boundary areas of the main depth map contained in the foreground layer, and the transverse boundary areas are It consists of a predetermined number of outermost columns of the depth map.

We further propose a storage medium that carries at least one encoded layered depth image / video frame, wherein at least one occlusion layer of the layered depth image / video frame is a layered depth image / video frame. The horizontal width of the occlusion layer is proportional to the maximum disparity value included in the transverse boundary regions of the main depth map contained within the foreground layer, and the transverse boundary regions are the main depth map. Consists of a predetermined number of outermost columns.

Then we propose a method for layered depth image / video frame encoding, the method comprising at least one of a layered depth image / video frame having a horizontal width that is greater than the foreground layer of the layered depth image / video frame. Encoding an occlusion layer of the occlusion layer, wherein the horizontal width of the occlusion layer is proportional to a maximum disparity value included in the transverse boundary regions of the main depth map included in the foreground layer, and the transverse boundary regions It consists of a predetermined number of outermost columns of the main depth map.

Similarly, an apparatus for layered depth image / video frame encoding is proposed, wherein the apparatus is capable of at least one error of a layered depth image / video frame having a greater horizontal width than the foreground layer of the layered depth image / video frame. It is adapted to encode the occlusion layer, where the horizontal width of the occlusion layer is proportional to the maximum disparity value included in the transverse boundary regions of the main depth map contained in the foreground layer, and the transverse boundary regions are It consists of a predetermined number of outermost columns.

An additional horizontal width may be used to convey the portion of information provided in the images / videos captured by the at least two cameras but not included in the foreground layer.

Further advantageous features of the embodiments are specified in the dependent claims.

Exemplary embodiments of the invention are illustrated in the drawings and described in more detail in the following description. Exemplary embodiments will be described for the purpose of clarity only, and do not limit the scope or spirit of the invention as defined in the disclosure, claims.
In the drawings,
1 depicts an example depth map;
2 depicts an exemplary multi camera system;
3 depicts exemplary stereoscopic shooting; And
4 depicts an example occlusion layer extension.
Exemplary Embodiments of the Invention
The invention may be realized on any electronic device, including a correspondingly adapted processing device. For example, the present invention can be realized in a mobile phone, a personal computer, a digital still camera system, or a digital video camera system.
1 depicts an exemplary depth map Mdm. The depth map Mdm is composed of depth values, disparity values or scaled values homogeneous with the disparity. The values are arranged in columns C [0], ..., C [n] and rows R [0], ..., R [m]. The depth map has vertical boundaries (vbl, vbr), also called transverse boundaries or lateral edges, and horizontal boundaries (hbt, hbb), also called upper and lower boundaries or upper and lower edges. The neighboring region Nkl of the width k of the left vertical boundaries vbl comprises the columns C [0], C [1], ..., C [k-1] and the right vertical boundaries vbr The neighboring region Nkr of width k of) comprises columns C [n-k + 1], C [n-k + 2], ..., C [n]. There is no restriction on the width of the neighbors, ie a single neighbor can cover the entire depth map Mdm, i.e. when k = n, or the width k1 of the left vertical boundaries vbl. The neighbor of and the neighbor of the width k2 of the right vertical boundaries vbr can cover all frames if k1 + k2 = n + 1. The width of the neighbors can also be limited to only one-pixel rows.
In LDI / LDV, such an example depth map Mdm is associated with an example image. There is a value in the example depth map for each pixel in the example image. The set of maps and images is called a layer. If the layer is a foreground layer, also called the main layer, the image is called the foreground image and the image is fully populated with pixels. The associated depth map is hereinafter referred to as the main depth map Mdm.
In an exemplary embodiment, the main depth map Mdm and the associated foreground image CV result from processing two views LV and RV. As shown in FIG. 2, the two views LV and RV have two parallel optical axes OA1 and OA2, a focal length f and an inter-camera baseline distance 2 * b. Are captured by two cameras CAM1 and CAM2. Also let z_conv indicate the depth of the convergence plane that can be located at infinity distance if no post-processing shifting is applied to the rectified views. Two cameras CAM1 and CAM2 are located at the two different viewpoints mentioned above. Two views (LV, RV) are preprocessed to describe the aforementioned scenes from two different viewpoints and also to equalize colors and correct for geometric distortions. Thus, the inherent and external parameters of the cameras are integrated. In two camera settings, the foreground image CV appears to be captured by the virtual camera CAMv, thus positioned between two cameras CAM1, CAM2 having the distance b between cameras to each camera. In odd camera settings, the foreground image CV is calculated by the correction of the pictures taken by the central camera.
Under these conditions, the disparity d of the object located at depth z is given by:
d = h-f * b / z (1)
Where h emulates the sensor shift required to adjust the position of the viewing plane. As mentioned earlier, if no treatment is applied, the plane of look at is at infinity and h is equal to zero. As exemplarily depicted in FIG. 3, z_conv is located at a finite distance:
h = f * b / z_conv (2)
If the main depth map Mdm contains a disparity d and a homogeneous scaled value D, the relationship of the two is
D = 255 * (d_max-d) / (d_max-d_min) (3)
Can be.
In the case of scaled values included in the main depth map, the parameters d_max and d_min are transmitted as metadata or the corresponding depth values z_near and z_far are transmitted, according to equation (1)
z_near = f * b / (h-d_max) (4)
ego
z_far = f * b / (h-d_min) (5)
to be.
Exemplary embodiments are selected only to illustrate the bones of the present invention. The present invention provides a multi-camera system with cameras with non-parallel optical axes, for example by converting images captured by cameras into corresponding virtual images captured virtually by cameras with virtual parallel optical axes. Can be applied. Moreover, the present invention can be adapted to uncorrected views and / or more than two cameras. The invention is also not related to how the foreground layer image or the main depth map is determined.
An example embodiment includes determining the closest object in the neighboring regions Nkl, Nkr of the lateral edges vbl, vbr of the main depth map Mdm, which is the smallest disparity min (d )] Corresponds to Since the disparity is negative for objects located in front of the gaze plane, this corresponds to determining the largest absolute value among the negative disparities in the neighboring regions of the lateral edges.
If the main depth map Mdm contains scaled values homogeneous with disparity, | min (d) | is the maximum scaled value in the main depth map Mdm using the parameters transmitted as metadata [ max (D)]. When d_max and d_min are transmitted this is done according to the following equation:
| min (d) | d_max-max (D) * (d_max-d_min) / 255 | (6)
When z_near and z_far are transmitted, | min (d) | may be determined using equations (4), (5) and (6).
If z_conv is not determined, | (min (d)-h) | is determined.
The largest determined absolute value among the negative disparities in the neighboring regions Nkr, Nkl of both transverse edges vbl, vbr is the additional width, which is the occlusion layer image EOV by this additional width. And / or the occlusion layer depth map must be expanded on both sides to allow all the information provided by the two views to be conveyed, although not included in the foreground image.
The widths of neighboring regions may be chosen differently. For example, neighboring regions may consist of only the outermost columns C [0], C [n]. Or, for robustness, neighboring regions consist of eight columns C [0], ..., C [7] and C [n-7], ..., C [n] on each side. Can be. Or, for exhaustiveness, neighboring regions are selected such that neighboring regions cover the entire main depth map such that the largest absolute value of all negative disparities included in the main depth map is determined.
In the latter case, a reduced value may be used instead of the largest absolute value determined. The reduced value compensates for the largest absolute value among the negative disparities by the distance of the column with the largest absolute value from the nearest transverse edge of each. That is, if the largest absolute value of the negative disparities is | min (d) | and is found in column j of the main depth map of width n, the occlusion layer is (min | (d) |-min (j; n + 1-j)). Thus, the width of the occlusion layer depth map and / or the occlusion layer image EOV is n + 2 * (| min (d) |-min (j; n + 1-j)). By way of example, as shown in FIG. 4, the occlusion layer image EOV is populated only with information that is coarse, ie, not within the foreground image. This information can be copied or warped by being projected onto the central view.
In the case of LDV, occlusion extension may be determined independently for each frame. Alternatively, groups of frames or the entire video can be analyzed for the largest absolute value of negative disparities in neighboring regions of the lateral edges of the respective frames, and the largest absolute value determined is then determined for each of the frames. It is used to extend the occlusion layer of the group or the entire video.
The analysis of the largest absolute value among the negative disparities in the neighboring regions of the lateral edges may be performed at the decoder side in the same way as it is done at the encoder side for corrective decoding of the occlusion layer. Or, side information about the extension is provided. The former is more efficient in terms of encoding, and the latter requires less computation on the decoder side.

Claims (15)

Data structure for layered depth images,
At least one occlusion layer of the layered depth image has a horizontal width that is greater than the foreground layer of the layered depth image, wherein the horizontal width of the occlusion layer is contained within the foreground layer. Proportional to the maximum disparity value included in the transverse boundary regions of the depth map, wherein the transverse boundary regions consist of a predetermined number of outermost columns of the main depth map.
Layered Depth Image Data Structure. A storage medium carrying at least one encoded layered depth image,
At least one occlusion layer of the layered depth image has a horizontal width that is greater than the foreground layer of the layered depth image, wherein the horizontal width of the occlusion layer is a transverse side of the main depth map contained within the foreground layer. Proportional to a maximum disparity value included in boundary regions, wherein the lateral boundary regions are composed of a predetermined number of outermost columns of the main depth map.
Storage media that delivers layered depth images. A method for layered depth image encoding,
Encoding at least one occlusion layer of the layered depth image with a horizontal width greater than a foreground layer of the layered depth image
Including,
The horizontal width of the occlusion layer is proportional to the maximum disparity value included in the transverse boundary regions of the main depth map included in the foreground layer, wherein the transverse boundary regions are the outermost of a predetermined number of the main depth map. Composed of columns
Layered Depth Image Encoding Method. A method for layered depth image decoding,
Decoding at least one occlusion layer of the layered depth image to a horizontal width that is greater than a foreground layer of the layered depth image
Including,
The horizontal width of the occlusion layer is proportional to the maximum disparity value included in the transverse boundary regions of the main depth map included in the foreground layer, wherein the transverse boundary regions are the outermost of a predetermined number of the main depth map. Composed of columns
Layered Depth Image Decoding Method. A device for layered depth image encoding,
Is adapted to encode at least one occlusion layer of the layered depth image to a horizontal width that is greater than the foreground layer of the layered depth image, wherein the horizontal width of the occlusion layer is included in the foreground layer. Proportional to the maximum disparity value included in the transverse boundary regions of the depth map, wherein the transverse boundary regions consist of a predetermined number of outermost columns of the main depth map.
Layered depth image encoding device. An apparatus for layered depth image decoding,
Adapted to decode at least one occlusion layer of the layered depth image to a horizontal width that is greater than the foreground layer of the layered depth image, wherein a horizontal width of the occlusion layer is included in the foreground layer. Proportional to the maximum disparity value included in the transverse boundary regions of the depth map, wherein the transverse boundary regions consist of a predetermined number of outermost columns of the main depth map.
Layered depth image decoding device. The data structure of claim 1, the storage medium of claim 2, the method of claim 3 or 4, or the device of claim 5, wherein
And the transverse boundary regions consist of all columns of the main depth map. The data structure of claim 1, the storage medium of claim 2, the method of claim 3 or 4, or the device of claim 5, wherein
A data structure, wherein the horizontal axis of the occlusion layer is further proportional to the minimum values of the distances from pixels to the matrix of the main depth map containing the maximum disparity value at the lateral boundaries of the foreground depth map; Storage medium, method or device. The data structure of claim 1, the storage medium of claim 2, the method of claim 3 or 4, or the device of claim 5, wherein
And the layered depth image is included in a sequence of layered depth images of the same occlusion layer widths. The data structure of claim 1, the storage medium of claim 2, the method of claim 3 or 4, or the device of claim 5, wherein
And a background image contained within said occlusion layer has a larger horizontal width than a foreground image contained within said foreground layer. The data structure of claim 1, the storage medium of claim 2, the method of claim 3 or 4, or the device of claim 5, wherein
And a background depth map included in the occlusion layer has a larger horizontal width than a foreground depth map included in the foreground layer. The method of claim 2,
And an encoded value indicative of the amount of rows in which the horizontal widths differ is further conveyed by the storage medium. The method of claim 4, wherein
And decoding a value representing the amount of columns in which the horizontal widths vary. The method of claim 6,
And a layered depth image decoding apparatus further adapted to decode a value representing an amount of columns in which the horizontal widths vary. The data structure of claim 1, the storage medium of claim 2, the method of claim 3 or 4, or the device of claim 5, wherein
The data structure, storage medium, method or apparatus included in the sequence of layered depth images of occlusion layer widths in which the layered depth image is different.

Images