WO2015192371A1

WO2015192371A1 - Texture based depth coding method

Info

Publication number: WO2015192371A1
Application number: PCT/CN2014/080404
Authority: WO
Inventors: Kai Zhang; Jicheng An; Xianguo Zhang; Han HUANG
Original assignee: Mediatek Singapore Pte. Ltd.
Priority date: 2014-06-20
Filing date: 2014-06-20
Publication date: 2015-12-23

Abstract

This contribution presents a texture based depth coding method in 3D-HEVC. Based on at least two texture pictures from different views, a calibration process can be applied on a prediction depth block or a reconstruction depth block. The calibrated pixels are then output as new prediction block or reconstruction block for following coding process.

Description

TEXTURE BASED DEPTH CODING METHOD

FIELD OF INVENTION

The invention relates generally to Three-Dimensional (3D) video processing. In particular, the presented invention relates to depth coding method.

BACKGROUND OF THE INVENTION

In 3D video coding schemes such as 3D-HEVC, texture pictures in different views and their corresponding depth pictures are coded together as depicted in Fig. l . Coding tools are developed to explore the redundancy between different views, or between depth and texture.

A fundamental redundancy is between depth and texture pictures. With texture pictures at the same time from two different views, disparity information is attainable by analyzing the two texture pictures. By converting the disparity vectors for each pixel to depth values, a depth picture can be generated. In fact, this approach is widely used to get the original depth picture when no distance measurement equipment is available. Obviously, redundancy happens when a depth picture on View 1 is signaled after two texture pictures on View 0 and View 1 at the same time have already been signaled, since the depth information is behind the two texture pictures.

Disparity derived depth (DDD) coding is adopted in 3D-HEVC to explore this fundamental redundancy partially. A disparity vector (DVx, 0) can be derived from its corresponding depth value (dep) by a linear relationship as

DVx = wdep + b , (1) where w and b are two camera parameters. In 3D-HEVC, (1) is implemented in an integer form thus w and b can be transmitted from the encoder to the decoder as integers.

It can be seen from (1) that the conversion between a depth value dep and DVx is reversible. Thus, a depth value can also be derived from its corresponding disparity vector {DVx, DVy) as dep =—-DVx -—. (2) w w

When the collocated texture block of the current depth block is predicted by disparity- compensated prediction (DCP), a DDD candidate will be inserted into the merging candidate list right after the texture candidate. In a DDD candidate, all prediction samples in the current block are set as dep calculated by (2), which is implemented in an integer form. Fig. 2 shows the procedure to derive a depth value from its corresponding disparity vector. It should be noted that the DDD candidate is invalid on the base view since DCP is applied only on dependent views.

Although DDD reduce the fundamental redundancy to some extent, its utilization is restricted since it can only be applied when the collocated texture block of the current depth block is predicted by DCP. The fundamental redundancy has not been explored sufficiently by DDD.

SUMMARY OF THE INVENTION

In light of the previously described problems, several texture based depth coding methods are proposed to explore the fundamental redundancy more sufficiently.

Other aspects and features of the invention will become apparent to those with ordinary skill in the art upon review of the following descriptions of specific embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention can be more fully understood by reading the subsequent detailed description and examples with references made to the accompanying drawings, wherein:

Fig. 1 is a diagram illustrating an exemplary 3D video coding structure with three views;

Fig. 2 is a diagram illustrating a depth value derived from its corresponding disparity vector;

Fig. 3 is a diagram illustrating an exemplary depth compensation calibration architecture at decoder;

Fig. 4 is a diagram illustrating an exemplary depth reconstruction calibration architecture at decoder;

Fig. 5 is a diagram illustrating an exemplary calibration process architecture;

Fig.6 is a diagram illustrating an exemplary selection of candidate depth values from a 4x4 region. The depth pixel values at the black positions are chosen.

DETAILED DESCRIPTION

The following description is of the best-contemplated mode of carrying out the invention. This description is made for the purpose of illustrating the general principles of the invention and should not be taken in a limiting sense. The scope of the invention is best determined by reference to the appended claims.

Several texture based depth coding methods are proposed to explore the fundamental redundancy more sufficiently. In one embodiment, depth compensation calibration (DCC) is proposed. In DCC, the prediction of a depth block is calibrated before it is used to get the residues at encoder or get the reconstruction at decoder. Fig. 3 demonstrates an exemplary DCC architecture at decoder. The calibration process depends on the texture pictures from at least two different views.

In another embodiment, depth reconstruction calibration (DRC) is proposed. In DRC, the reconstruction of a depth block is calibrated before being output or being referred to by following blocks or pictures. Fig. 4 demonstrates an exemplary DRC architecture at decoder. The calibration process depends on the texture pictures from at least two different views.

The calibration process is designed to modify a depth block depending on the texture pictures from at least two different views. The input to the calibration process can be prediction pixels in a depth block in DCC; or the input can be reconstruction pixels in a depth block in DRC. The output of the calibration process can be modified prediction pixels in DCC; or the output can be modified reconstruction pixels in DRC.

In one embodiment of the calibration process, the input depth block is divided into several regions. The regions cover the whole input depth block and do not overlap with each other. The regions may possess the same size and shape. For a region RegionA in the input depth block, a depth value T is determined by analyzing texture pictures from at least two different views. All pixels in RegionA is set equal to be T.

Fig. 5 demonstrates an example how to determine the value T. T is chosen from several candidate depth values, denoted as Cando, Candi, Cand^. For each candidate, noted as Cand_/t, a corresponding disparity vector noted as DV^ can be obtained by converting the depth value into a disparity vector. For example, the equation (1) can be applied for the conversion. The corresponding region of RegionA in the texture picture at the same time and in the same view with the current depth picture is denoted as RegionB. The terminology 'corresponding' means RegionA and RegionB possess the same shape, same size and are at the same relative position in their pictures. Another region noted as RegionC covers RegionB totally. In other words, RegionC contains RegionB or RegionC is equal to RegionC. With DV^, a disparity aligned region for RegionC can be located in the texture picture at the same time in the base view. The sum of absolute difference (SAD) between RegionC and the disparity aligned region with DV_/t is calculated as SAD ^. Suppose the -Sth depth candidate can generate the minimum value of SAD, or in a formula way S=argmin^ (SAD k). Then the depth value Cand^is chosen as the value T.

In another embodiment, RegionA is an MxN block, where M and N are positive integers. In still another embodiment, RegionA is an MxM block, where is a positive integers. For example, M is 4 or 8.

In still another embodiment, RegionC is an MxN block, where M and N are positive integers.

In still another embodiment, RegionC is an MxM block, where M is positive integers. For example, M is 4 or 8.

In still another embodiment, RegionA is a single pixel.

In still another embodiment, RegionA is the whole input block.

In still another embodiment, RegionC is larger than RegionB.

In still another embodiment, RegionC is the same region as RegionB.

In still another embodiment, several depth values in RegionA are chosen as the candidate depth values. Fig. 6 demonstrates an example. RegionA is a 4x4 block. The depth pixel values at the black positions are chosen as candidate depth values.

In still another embodiment, if V is a candidate depth value, then V+offset and V-offset can be chosen as a candidate depth value, where offset is a positive integer.

In still another embodiment, the number of different candidate depth values used to calculate the SAD cannot exceed a maximum value M. If M different candidates have been checked, then determining process for value T is stopped. The candidate value producing the minimum SAD currently will be chosen to be T.

In still another embodiment, the determining process for value T is stopped if a candidate depth value produces a SAD lower than a threshold. The candidate value producing the minimum SAD currently will be chosen to be T.

In still another embodiment, the disparity vector converted from a candidate depth value can hold an integer-pixel precision or a sub-pixel precision. If it hold a sub-pixel precision, interpolation filtering is applied to get the disparity aligned region for RegionC. For example, the interpolation filter for luma or chroma used in the motion compensation (MC) process by HEVC can be applied. In another example a bi-linear interpolation filter can be applied.

The SAD calculation process can be applied on the luma component, it can also be applied on chroma components.

In still another embodiment, the SAD on luma (Y) and two chroma (Cb and Cr) components for the Ath candidate are SAD ^Y _FC

SAD^Cr _fe respectively, then the final SAD used in the comparison should be calculated as SAD=W^Y* SAD w^CB* SAD^cVw^CR* SAD^CR _FC ; w^Y, w^cb and w^Cr can be any integer or real numbers. For example, w^Y =1, w^cb =w^Cr=4.

In still another embodiment, the calibration process can be used or not adaptively. The encoder can send the information of whether to use the calibration process to the decoder explicitly. Or the decoder can derive whether to use the calibration process in the same way as the encoder explicitly.

In still another embodiment, the calibration process can be applied on coding tree unit (CTU), coding unit (CU), prediction unit (PU) or transform unit (TU).

In still another embodiment, the encoder can send the information of whether to use the calibration process to the decoder in video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), slice header (SH), CTU, CU, PU, TU.

In still another embodiment, the calibration process can only be applied for CU with some particular sizes. For example, it can only be applied to a CU with size larger than 8x8. In another example, it can only be applied to a CU with size smaller than 64x64.

In still another embodiment, the calibration process can only be applied for CU with some particular PU partition. For example, it can only be applied to a CU with 2Nx2N partition.

In still another embodiment, the calibration process can be applied for a PU coded as merge mode.

In still another embodiment, the calibration process can be applied for a PU coded as merge mode and a specific merging candidate is selected. For example, the specific merging candidate is after the texture candidate in the merging candidate list.

In still another embodiment, the specific merging candidate shares the same motion information as the texture candidate. And the prediction block input to the calibration process is obtained by this motion information.

In still another embodiment, the specific merging candidate shares the same motion information as the first candidate in the HEVC merging candidate list if the texture candidate is unavailable. And the prediction block input to the calibration process is obtained by this motion information.

The proposed method described above can be used in a video encoder as well as in a video decoder. Embodiments of methods according to the present invention as described above may be implemented in various hardware, software codes, or a combination of both. For example, an embodiment of the present invention can be a circuit integrated into a video compression chip or program codes integrated into video compression software to perform the processing described herein. An embodiment of the present invention may also be program codes to be executed on a Digital Signal Processor (DSP) to perform the processing described herein. The invention may also involve a number of functions to be performed by a computer processor, a digital signal processor, a microprocessor, or field programmable gate array (FPGA). These processors can be configured to perform particular tasks according to the invention, by executing machine-readable software code or firmware code that defines the particular methods embodied by the invention. The software code or firmware codes may be developed in different programming languages and different format or style. The software code may also be compiled for different target platform. However, different code formats, styles and languages of software codes and other means of configuring code to perform the tasks in accordance with the invention will not depart from the spirit and scope of the invention.

The invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples are to be considered in all respects only as illustrative and not restrictive. To the contrary, it is intended to cover various modifications and similar arrangements (as would be apparent to those skilled in the art). Therefore, the scope of the appended claims should be accorded the broadest interpretation so as to encompass all such modifications and similar arrangements.

Claims

1. A method of coding depth component, comprising,

receiving an input of a depth block;

calibrating pixel values of the depth block based on at least two texture pictures from different views; and

outputting the calibrated depth block.

2. The method as claimed in claim 1, wherein a prediction of a depth block is calibrated before it is used to get residues at encoder or get reconstruction at decoder.

3. The method as claimed in claim 1, wherein reconstruction of a depth block is calibrated before being output or being referred to by following blocks or pictures.

4. The method as claimed in claim 1, wherein the depth values generated from a disparity vectors is used in prediction or as a predictor of the depth coding.

5. The method as claimed in claim 1, wherein regions cover a whole input depth block and do not overlap with each other, the regions possess the same size and shape, for a region

RegionA in the input depth block, a depth value T is determined by analyzing texture pictures from at least two different views, and all pixels in RegionA is set equal to be T.

6. The method as claimed in claim 5, wherein T is chosen from several candidate depth values, denoted as Cando, Candi, ... , Cand^, for each candidate, noted as Cand^, a corresponding disparity vector noted as DV^ is obtained by converting the depth value into a disparity vector, and another region noted as RegionC covers RegionB which is the corresponding region of RegionA in the texture picture totally, with DV^, a disparity aligned region for RegionC is located in the texture picture at the same time in the base view, a sum of absolute difference (SAD) between RegionC and the disparity aligned region with DV^ is calculated as SAD _¾ wherein suppose the -Sth depth candidate generates a minimum value of SAD, or in a formula way S=argmin k (SAD k), then the depth value Cands is chosen as the value T.

7. The method as claimed in claim 5, wherein RegionA is an MxN block, where M and N are positive integers, particularly, RegionA is an MxM block, where is a positive integers, and RegionA is a single pixel if M is equal to 1.

8. The method as claimed in claim 6, wherein RegionC is an MxN block, where M and N are positive integers, particularly, RegionC is an MxM block, where is a positive integers, and RegionC is a single pixel if M is equal to 1.

9. The method as claimed in claim 6, wherein several depth values in RegionA are chosen as the candidate depth values; The pixel values at some specific positions can be chosen as the candidate depth values; The specific position includes but not limited to center, left above, right bottom, right above and left bottom; RegionA is a 4x4 block; The depth pixel values at the black positions are chosen as candidate depth values.

10. The method as claimed in claim 6, wherein if V is a candidate depth value, then V+offset and V-offset can be chosen as a candidate depth value, where offset is a positive integer.

11. The method as claimed in claim 6, wherein a number of different candidate depth values used to calculate the SAD never exceed a maximum value M, if M different candidates have been checked, then determining process for value T is stopped, and the candidate value producing the minimum SAD currently will be chosen to be T.

12. The method as claimed in claim 6, wherein the determining process for value T is stopped if a candidate depth value produces a SAD lower than a threshold; The candidate value producing the minimum SAD currently will be chosen to be T.

13. The method as claimed in claim 6, wherein the disparity vector converted from a candidate depth value can hold an integer-pixel precision or a sub-pixel precision; If it hold a sub-pixel precision, interpolation filtering is applied to get the disparity aligned region for RegionC.

14. The method as claimed in claim 6, wherein The SAD calculation process is applied on the luma component, chroma components, or both luma and chroma components.

15. The method as claimed in claim 6, wherein the SAD on luma (Y) and two chroma (Cb and Cr) components for the Ath candidate are SAD SAD^cb _¾ SAD^Cr _k respectively, then the final SAD used in the comparison is calculated as SAD = W^Y* SAD +

. _WY _wcb _{and w}cr _{ig any integer or real num}b_ers.

16. The method as claimed in claim 1, wherein the calibration process is used or not adaptively, the encoder sends information of whether to use the calibration process to the decoder explicitly, or the decoder derive whether to use the calibration process in the same way as the encoder explicitly.

17. The method as claimed in claim 1 and claim 16, wherein the calibration process is applied on coding tree unit (CTU), coding unit (CU), prediction unit (PU) or transform unit (TU) level.

18. The method as claimed in claim 1, wherein the encoder sends information of whether to use the calibration process to the decoder in video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), slice header (SH), CTU, CU, PU, or TU level.

19. The method as claimed in claim 1, wherein the calibration process is only applied for CU with a predetermined set of sizes.

20. The method as claimed in claim 1, wherein the calibration process is applied for CU with a predetermined set of PU partitions.

21. The method as claimed in claim 1, wherein the calibration process is applied for a PU coded as merge mode.

22. The method as claimed in claim 21, wherein the calibration process is applied for a PU coded as merge mode and a specific merging candidate is selected.

23. The method as claimed in claim 22, wherein the specific merging candidate shares same motion information as the texture candidate, and the prediction block input to the calibration process is obtained by this motion information.

24. The method as claimed in claim 23, wherein the specific merging candidate shares the same motion information as a first candidate in a merging candidate list if the texture candidate is unavailable, and the prediction block input to the calibration process is obtained by this motion information.