WO2015139205A1

WO2015139205A1 - An improved method for collocated picture in video coding

Info

Publication number: WO2015139205A1
Application number: PCT/CN2014/073613
Authority: WO
Inventors: Kai Zhang; Jicheng An; Xianguo Zhang; Han HUANG
Original assignee: Mediatek Singapore Pte. Ltd.
Priority date: 2014-03-18
Filing date: 2014-03-18
Publication date: 2015-09-24

Abstract

Methods are proposed to manage the collocated picture or even rebuild one used by TMVP in video coding. Some special methods are further proposed for MVC, SVC, and 3DVC.

Description

AN IMPROVED METHOD FOR COLLOCATED PICTURE

IN VIDEO CODING

TECHNICAL FIELD

[0001] The invention relates generally to video processing. In particular, the presented invention relates to the collocated picture in video coding.

BACKGROUND

[0002] In video coding, temporal motion vector prediction (TMVP) is utilized as a kind of motion prediction. In TMVP, the motion vector predictor (MVP) is derived from the motion information of a collocated block in a collocated picture as depicted in Fig. 1.

[0003] The collocated picture is a temporal reference picture for the current picture. The collocated picture can be derived implicitly at decoder as specified in H.264/AVC, or it can be signaled explicitly from the encoder to the decoder as specified in H.265/HEVC.

[0004] The collocated block and the current block are at the same relative position in the collocated picture and in the current picture respectively. The motion vector (MV) in the collocated block is named as collocated MV. TMVP is derived by scaling the collocated MV according to the time distances. Time distance (or named POC distance, POC difference) for a MV specifies the time distance between the picture the MV belonging to and the picture the MV pointing to. Herein POC represents picture order count.

[0005] The time distance for the collocated MV is available since the collocated MV has been determined when the collocated picture is encoded/ decoded. The time distance for the TMVP is determined by determining the reference picture of the TMVP. In HEVC, the reference for TMVP is signaled explicitly for a block coded with advanced motion vector prediction (AMVP) mode, while the reference for TMVP is determined implicitly for a block coded with merge mode.

[0006] Suppose the time distance of the collocated MV, cMV, is cD and the time distance of the TMVP, tMVP, is tD, then tMVP should be calculated as tMVP = (cMV * tD )/ cD. In practical implementation such as HEVC, some more sophisticated methods are used to guarantee the division can be done with limited bit-width. In the example of Fig. 1, the time distance of the collocated MV is 2 and the time distance of the TMVP is 1, thus tMVP should be equal to cMV/2.

[0007] A motion compression is applied to reduce the storage required by MVs. In H.265/HEVC, a 16: 1 compression is applied after a picture is encoded /decoded.

[0008] There are several extensions for video coding, such as Multi-view video coding (MVC), 3D video coding (3DVC) and scalable video coding (SVC). In MVC and 3DVC, an inter- view reference picture can be inserted into the reference list as a special reference picture for a picture on a dependent view. In SVC, a low-quality reference picture can be inserted into the reference list as a special reference picture for a picture on a high-quality layer. Still in SVC, an up-sampled low-resolution reference picture can be inserted into the reference list as a special reference picture for a picture on a high-resolution layer.

[0009] Since these special reference pictures are in the reference list, it is possible to treat the special reference picture as the collocated picture for the current picture when the special reference picture is available. However, there has been no study on this topic.

SUMMARY

[0010] In light of the previously described problems, methods are proposed to manage the collocated picture or even rebuild one in some cases.

[0011] Other aspects and features of the invention will become apparent to those with ordinary skill in the art upon review of the following descriptions of specific embodiments.

BRIEF DESCRIPTION OF DRAWINGS

[0012] The invention can be more fully understood by reading the subsequent detailed description and examples with references made to the accompanying drawings, wherein:

[0013] Fig. 1 is a diagram illustrating TMVP in video coding.

[0014] Fig. 2 is a diagram illustrating to use the interview reference picture as the collocated picture in MVC or 3DVC.

[0015] Fig. 3 is a diagram illustrating to use the corresponding texture picture as the collocated picture in MVC or 3DVC with depth.

[0016] Fig. 4 is a diagram illustrating to use the inter-quality-layer picture as the collocated picture in SVC.

[0017] Fig. 5 is a diagram illustrating a general paradigm for virtual collocated picture.

[0018] Fig. 6 is a diagram illustrating to use the inter-spatial-layer picture as the collocated picture in SVC.

[0019] Fig. 7 is a diagram illustrating an exemplary MV upsampling, which is 1 :4 upsampling.

[0020] Fig. 8 is a diagram illustrating to use the virtual collocated picture for MVC or 3DVC.

[0021] Fig. 9 is a diagram illustrating to set the MV in a block in the virtual collocated picture for 3DVC or MVC if the collocated block in the auxiliary picture possesses a DV pointing to the reference view, and the DV-calibrated block in the inter-view reference picture possesses a MV.

[0022] Fig. 10 is a diagram illustrating to set the MV in a block in the virtual collocated picture for 3DVC or MVC if the collocated block in the auxiliary picture does not possess a DV pointing to the reference view, but possesses a MV.

[0023] Fig. 11 is a diagram illustrating to set the MV in a block in the virtual collocated picture for 3DVC or MVC if the collocated block in the auxiliary picture possesses a DV pointing to the reference view, but the DV-calibrated block in the inter- view reference picture does not possess a MV, i.e., it is intra-coded.

[0024] Fig. 12 is a diagram illustrating to set a block as intra-coded in the virtual collocated picture for 3DVC or MVC if the collocated block in the auxiliary picture does not possess a MV or DV, i.e. it is intra-coded.

[0025] Fig. 13 is a diagram illustrating to set the MV in a block in the virtual collocated picture for 3DVC or MVC if the collocated block in the auxiliary picture does not possess a MV or DV, i.e. it is intra-coded.

[0026] Fig. 14 is a diagram illustrating an exemplary flowchart to process a block in the virtual collocated picture when there are two auxiliary pictures for 3DVC or MVC.

DETAILED DESCRIPTION

[0027] The following description is of the best-contemplated mode of carrying out the invention. This description is made for the purpose of illustrating the general principles of the invention and should not be taken in a limiting sense. The scope of the invention is best determined by reference to the appended claims.

[0028] It is proposed to treat the special reference pictures as the collocated picture used in TMVP for the current picture, or to construct a virtual collocated picture used in TMVP for the current picture. The term 'special reference pictures' includes but not limited to inter-layer reference pictures as specified in HEVC, such as inter-view reference pictures, inter-component reference pictures, inter-quality-layer reference pictures, and inter-spatial-layer reference pictures.

[0029] In one embodiment as depicted in Fig. 2, the inter-view reference picture is treated as the collocated picture used in TMVP in MVC, such as H.264 MVC extension or MV-HEVC, or 3DVC such as 3D-HEVC. The inter-view reference picture and the current picture should be captured at the same time. For example, they should possess the same POC.

[0030] In another embodiment as depicted in Fig. 3, the corresponding texture picture is treated as the collocated picture used in TMVP to code a depth picture in MVC, such as H.264 MVC extension or MV-HEVC, or 3DVC such as 3D-HEVC with depth coding. A texture picture is corresponding to a depth picture if they are in the same view and they are captured at the same time. For example, they possess the same POC and Viewld.

[0031] In still another embodiment as depicted in Fig. 4, the inter-quality-layer reference picture is treated as the collocated picture used in TMVP to code a picture in SVC, such as H.264 SVC extension or HEVC SVC extension. The inter-quality-layer reference picture and the current picture should be captured at the same time. For example, they should possess the same POC.

[0032] In still another embodiment as depicted in Fig. 5, a virtual collocated picture is proposed, which is treated as the collocated picture used in TMVP. A virtual collocated picture can be generated by a process which takes several previously encoded \ decoded pictures in the current layer or in a dependent layer as input, and output the MV information for each block in the virtual collocated picture as well as all other picture information for the virtual collocated picture. Herein the term 'layer' possesses the same meaning as defined in HEVC. Different layer can represent different views in MVC/3DVC, or different component (texture or depth) in MVC/3DVC with depth, or different quality layers in SVC, or different spatial layers in SVC. For example, the motion information in a block in the virtual collocated picture can be copied from the motion information in the collocated block in a specific reference picture in the virtual collocated picture generation process. Some changes or calibrations may happen during or after the copying process. In other words, the resulted motion information in the virtual collocated picture may be not exactly the same to that in the motion information in the reference picture which is copied from. Only information required by TMVP for the current picture, such as motion information for each block, needs to be generated by the virtual collocated picture process. It should be noted that the term 'virtual' can also be replaced by other words such as 'created', 'manufactured', 'made', 'pseudo' or any other similar words. Any picture, which is used as the collocated picture in TMVP for the current picture but its motion information is not exactly the same as the motion information or the compressed motion information of any reference picture of the current picture, should be regarded as a virtual collocated picture.

[0033] In still another embodiment, the virtual collocated picture generation process does not change the input pictures.

[0034] In still another embodiment, the virtual collocated picture generation process only takes reference pictures of the current picture as input.

[0035] In still another embodiment, the virtual collocated picture should be assigned with picture information, such as POC, picture height, picture width, reference picture set, reference picture lists, etc, as specified in H.264 or HEVC. The picture information should be signaled to the decoder from the encoder explicitly, or be derived by the encoder explicitly. The POC assigned to the virtual collocated picture will be used in the scaling process to get the TMVP.

[0036] In still another embodiment, the virtual collocated picture is assigned the same POC as the current picture.

[0037] In still another embodiment, the virtual collocated picture is assigned the same picture width and height as the current picture. [0038] In still another embodiment as depicted in Fig. 6, the inter-spatial-layer reference picture is used to generate the virtual collocated picture as the collocated picture used in TMVP to code a picture in SVC, such as H.264 SVC extension or HEVC SVC extension. The inter-quality-layer reference picture and the current picture should be captured at the same time. For example, they should possess the same POC. Since pictures possess different resolutions in different spatial layers, the MV information in the inter-spatial-layer reference picture should be upsampled or downsampled to generate the virtual collocated picture. The ratio of the upsampling or downsampling depends on the resolution ratio between the two spatial layers. Fig. 7 demonstrates an upsampling example when the ratio is 1 : 4. Other information such as POC and reference picture list of the virtual collocated picture is set to the same as that of the inter-quality-layer reference picture.

[0039] In still another embodiment, a MV can be calibrated when it is needed to be copied from one reference picture to the virtual collocated picture. For example, if MVs in depth coding only support integer pixel precision but MVs in texture coding support quarter pixel precision, the MV needed to be copied should be calibrated as MV'=(MV+2)»2 or MV'»2, and MV should be copied to the virtual collocated picture. In another example, if the width and height of the reference spatial layer are Wand H, and those of the current spatial layer are a* Wand b*H, the MV=(x,y) which is needed to be copied should be calibrated as MV=(a*x, b*y), and MV should be copied to the virtual collocated picture.

[0040] In still another embodiment, a MV can be scaled when it is needed to be copied from a reference picture to the virtual collocated picture. The MV should be scaled when it is needed to be copied from a first reference picture to the virtual collocated picture if the first reference picture and the virtual collocated picture possess different POC. For example, suppose the MV in the first reference picture needed to be copied is MV ^'_A, the resulted MV in the virtual collocated picture points is MV _B, POC of the reference picture is POC A, POC of the virtual collocated picture is POC B, POC of a second reference picture, to which MV A points, is POC C, and the third reference picture, to which MV B points, is POC D, then MV B should be calculated as MV B=( (POC D- POC B)* MV_A)I {POC C- POC A). Some more sophisticated methods are used to guarantee the division can be done with limited bit-width as specified in H.264 or HEVC.

[0041] In still another embodiment, the POC of a first reference picture, to which the MV in the virtual collocated picture points, can be signaled from the encoder to the decoder explicitly, or it can be derived by the decoder implicitly. And it can be different for different blocks in the virtual collocated picture. For example, if a MV noted as MV A is needed to be copied from a second reference picture to the virtual collocated picture, and the second reference picture and the virtual collocated picture possess the same POC, then the POC of the first reference picture should be set equal to a third reference picture, to which MV A points. In another example, if the second reference picture and the virtual collocated picture possess the different POCs, then the POC of the first reference picture should be set equal to the POC of the reference picture with reference index 0 in reference list 0 or reference list 1.

[0042] In still another embodiment, the reference picture, to which the MV in the virtual collocated picture points, must be in the reference picture lists of the virtual collocated picture.

[0043] In still another embodiment as depicted in Fig. 8, the inter-view reference picture and one or more auxiliary reference pictures are used to generate the virtual collocated picture as the collocated picture used in TMVP in MVC, such as H.264 MVC extension or MV-HEVC, or 3DVC such as 3D-HEVC. The inter-view reference picture and the current picture should be captured at the same time. For example, they should possess the same POC. The number of auxiliary reference pictures can be 1, 2, or more. Information such as POC, Viewld and reference picture list of the virtual collocated picture is set to the same as that of the inter-view reference picture.

[0044] In still another embodiment, auxiliary reference pictures should be reference pictures of the current picture.

[0045] In still another embodiment, auxiliary reference pictures should be in the current view.

[0046] In still another embodiment, auxiliary reference pictures should be signaled to the decoder from the encoder explicitly. The signaling information can be in video parameter set (VPS) or its extension, sequence parameter set (SPS) or its extension, picture parameter set (PPS) or its extension, slice header or its extension, coding tree unit (CTU) or its extension, coding unit (CU) or its extension.

[0047] In still another embodiment, auxiliary reference pictures should be derived by the decoder implicitly, without signaling.

[0048] In still another embodiment, virtual collocated picture generation process in MVC or 3DVC can work as follows with one interview reference picture and one auxiliary picture as input.

[0049] Case 1 : As depicted in Fig. 9, the MV information in a block in the virtual collocated picture is copied from that in the DV-calibrated block in the inter-view reference picture, if the collocated block in the auxiliary picture possesses a disparity vector (DV) pointing to the reference view, and the DV-calibrated block in the interview reference picture possesses a MV. A DV-calibrated block is obtained as follows. Suppose the collocated block is at the position (x,y), DV=(dx, dy), then the DV- calibrated block is at the position (x+dx, y+dy). Scaling may be needed in copying.

[0050] Case 2: As depicted in Fig. 10, the MV information in a block in the virtual collocated picture is copied from the collocated block in the auxiliary picture if the collocated block in the auxiliary picture does not possess a DV pointing to the reference view, but possesses a MV. Scaling may be needed in copying.

[0051] Case 3 : As depicted in Fig. 11, the MV information in a block in the virtual collocated picture is copied from the collocated block in the auxiliary picture if the collocated block in the auxiliary picture possesses a DV pointing to the reference view, but the DV-calibrated block in the inter-view reference picture does not possess a MV, i.e., it is intra-coded. Scaling may be needed in copying.

[0052] Case 4: As depicted in Fig. 12, a block in the virtual collocated picture is set as intra-coded if the collocated block in the auxiliary picture does not possess a MV or DV, i.e. it is intra-coded.

[0053] Case 5: Or as depicted in Fig. 13, the MV information in a block in the virtual collocated picture is copied from the collocated block in the inter-view reference picture if the collocated block in the auxiliary picture does not possess a MV or DV, i.e. it is intra-coded, and the collocated block in the inter-view reference picture possesses a MV. Scaling may be needed in copying.

[0054] Case 6: A block in the virtual collocated picture is set as intra-coded if the collocated block in the auxiliary picture does not possess a MV or DV, i.e. it is intra- coded and the collocated block in the interview reference picture does not possess a MV or DV, i.e. it is intra-coded.

[0055] In still another embodiment, if the collocated block in the auxiliary picture does not possess a DV, a default DV is used to locate the DV-calibrated block in the inter-view reference picture. The MV information in a block in the virtual collocated picture is copied from that in the DV-calibrated block in the inter-view reference picture if the DV-calibrated block in the inter-view reference picture possesses a MV. Scaling may be needed in copying. The default DV can be a fixed one, or it can be the founded DV in a previous collocated block in the auxiliary picture.

[0056] In still another embodiment, the virtual collocated picture generation process in MVC or 3DVC can work as follows with two interview reference pictures and two auxiliary pictures as input, depicted as an exemplary flowchart in Fig. 14. In Fig. 14, AuxPicO and AuxPicl represent the two auxiliary pictures and col-block represents the collocated block.

[0057] If the DV-calibrated block located by the motion information from AuxPicO possesses a MV as stated in easel, then copy the motion information in the DV- calibrated block in the inter- view reference block to the current block in the virtual collocated picture. Scaling may be needed in copying.

[0058] Otherwise, if the DV-calibrated block located by the motion information from AuxPicl possesses a MV as stated in easel, then copy the motion information in the DV-calibrated block in the inter-view reference block to the current block in the virtual collocated picture. Scaling may be needed in copying.

[0059] Otherwise, motion information in the collocated block in AuxPicO or AuxPicl can be copied to the current block in the virtual collocated picture, if the information is available. Scaling may be needed in copying.

[0060] If the collocated blocks in AuxPicO and AuxPicl are both intra-coded, then the current block in the virtual collocated picture is set as intra-coded.

[0061] The methods described above can be used in a video encoder as well as in a video decoder. Embodiments of disparity vector derivation methods according to the present invention as described above may be implemented in various hardware, software codes, or a combination of both. For example, an embodiment of the present invention can be a circuit integrated into a video compression chip or program codes integrated into video compression software to perform the processing described herein. An embodiment of the present invention may also be program codes to be executed on a Digital Signal Processor (DSP) to perform the processing described herein. The invention may also involve a number of functions to be performed by a computer processor, a digital signal processor, a microprocessor, or field programmable gate array (FPGA). These processors can be configured to perform particular tasks according to the invention, by executing machine-readable software code or firmware code that defines the particular methods embodied by the invention. The software code or firmware codes may be developed in different programming languages and different format or style. The software code may also be compiled for different target platform. However, different code formats, styles and languages of software codes and other means of configuring code to perform the tasks in accordance with the invention will not depart from the spirit and scope of the invention.

[0062] The invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples are to be considered in all respects only as illustrative and not restrictive. To the contrary, it is intended to cover various modifications and similar arrangements (as would be apparent to those skilled in the art). Therefore, the scope of the appended claims should be accorded the broadest interpretation so as to encompass all such modifications and similar arrangements.

Claims

1. A method of generalized a collocated picture in video coding, comprising: treating special reference pictures as a collocated picture used in temporal motion vector prediction (TMVP) for a current picture; and constructing a virtual collocated picture used in TMVP for the current picture.

2. The method as claimed in claim 1, wherein an inter- view reference picture is treated as the collocated picture used in TMVP for multi-view video coding.

3. The method as claimed in claim 1, wherein a corresponding texture picture is treated as the collocated picture used in TMVP to code a depth picture for multi-view video coding.

4. The method as claimed in claim 1, wherein an inter-quality-layer reference picture is treated as the collocated picture used in TMVP to code a picture in scalable video coding.

5. The method as claimed in claim 1, wherein a virtual collocated picture is treated as the collocated picture used in TMVP, wherein the virtual collocated picture is generated by a process which takes several previously encoded or decoded pictures in a current layer or in a dependent layer as input, and output motion vector (MV) information for each block in the virtual collocated picture as well as all other picture information for the virtual collocated picture.

6. The method as claimed in claim 5, wherein the virtual collocated picture (VCP) generation process has one or some of the following properties: VCP does not change input pictures; VCP only takes reference pictures of the current picture as input; VCP is assigned same POC as the current picture; VCP is assigned same picture width and height as the current picture; and only information required by TMVP for the current picture, such as motion information for each block, needs to be generated by the virtual collocated picture process.

7. The method as claimed in claim 5, wherein a MV is scaled when it is needed to be copied from a reference picture to a virtual collocated picture; the MV is scaled when it is needed to be copied from a first reference picture to the virtual collocated picture if a first reference picture and the virtual collocated picture possess different picture order count (POC).

8. The method as claimed in claim 5, wherein POC of a first reference picture, to which the MV in the virtual collocated picture points, is signaled from an encoder to a decoder explicitly, or is derived by the decoder implicitly; wherein it can be different for different blocks in the virtual collocated picture.

9. The method as claimed in claim 5, wherein the reference picture, to which the MV in the virtual collocated picture points, is in the reference picture lists of the virtual collocated picture.

10. The method as claimed in claim 5, wherein an inter-spatial-layer reference picture is used to generate the virtual collocated picture as the collocated picture used in TMVP to code a picture in scalable video coding SVC.

11. The method as claimed in claim 10, wherein the MV information in the inter- spatial-layer reference picture is upsampled or downsampled to generate the virtual collocated picture, wherein a ratio of the upsampling or downsampling depends on a resolution ratio between two spatial layers.

12. The method as claimed in claim 5, wherein the inter- view reference picture and one or more auxiliary reference pictures are used to generate the virtual collocated picture as the collocated picture used in TMVP in multi-view video coding, or three- dimensional video coding; and the inter-view reference picture and the current picture is captured at the same time; information comprising POC, Viewld and reference picture list of the virtual collocated picture is set to the same as that of the inter-view reference picture.

13. The method as claimed in claim 12, wherein one or more following conditions is satisfied for auxiliary reference pictures: the auxiliary reference pictures are reference pictures of the current picture; the auxiliary reference pictures are in the current view; the auxiliary reference pictures are signaled to a decoder from an encoder explicitly; signaling information is in video parameter set (VPS) or its extension, sequence parameter set (SPS) or its extension, picture parameter set (PPS) or its extension, slice header or its extension, coding tree unit (CTU) or its extension, coding unit (CU) or its extension; the auxiliary reference pictures are derived by the decoder implicitly, without signaling.

14. The method as claimed in claim 12, wherein the MV information in a block in the virtual collocated picture is copied from that in the DV-calibrated block in the inter-view reference picture, if the collocated block in the auxiliary picture possesses a disparity vector (DV) pointing to the reference view, and the DV-calibrated block in the inter-view reference picture possesses a MV; wherein scaling is adaptively applied in copying.

15. The method as claimed in claim 12, wherein the MV information in a block in the virtual collocated picture is copied from the collocated block in the auxiliary picture if the collocated block in the auxiliary picture does not possess a DV pointing to the reference view, but possesses a MV; scaling is adaptively applied in copying.

16. The method as claimed in claim 12, wherein the MV information in a block in the virtual collocated picture is copied from the collocated block in the auxiliary picture if the collocated block in the auxiliary picture possesses a DV pointing to the reference view, but the DV-calibrated block in the inter-view reference picture does not possess a MV; wherein scaling is adaptively applied in copying.

17. The method as claimed in claim 12, wherein a block in the virtual collocated picture is set as intra-coded if the collocated block in the auxiliary picture does not possess a MV or DV, as is intra-coded.

18. The method as claimed in claim 12, wherein the MV information in a block in the virtual collocated picture is copied from the collocated block in the inter- view reference picture if the collocated block in the auxiliary picture does not possess a MV or DV, and the collocated block in the inter-view reference picture possesses a MV, wherein scaling is adaptively applied in copying.

19. The method as claimed in claim 12, wherein a block in the virtual collocated picture is set as intra-coded if the collocated block in the auxiliary picture does not possess a MV or DV, and the collocated block in the interview reference picture does not possess a MV or DV.

20. The method as claimed in claim 12, wherein if the collocated block in the auxiliary picture does not possess a DV, a default DV is used to locate the DV- calibrated block in the inter-view reference picture, the MV information in a block in the virtual collocated picture is copied from that in the DV-calibrated block in the inter-view reference picture if the DV-calibrated block in the inter-view reference picture possesses a MV, wherein scaling is adaptively applied in copying; and the default DV is a fixed one, or is founded DV in a previous collocated block in the auxiliary picture.

21. The method as claimed in claim 14, wherein the MV information in a block in the virtual collocated picture is copied from that in the DV-calibrated block in the inter-view reference picture, if such MV information can be found with any auxiliary reference pictures.

22. The method as claimed in claims 14-19, wherein the processes as stated are conducted by one or more auxiliary reference pictures sequentially if MV information in the DV-calibrated block cannot be found with any auxiliary reference pictures.