KR102450491B1 - System and method for combined inter and intra prediction - Google Patents

Abstract

This disclosure relates to a method for video coding. The method includes obtaining a first reference picture and a second reference picture associated with a current prediction block, obtaining a first prediction L0 based on a first motion vector MV0 from the current prediction block to a reference block in the first reference picture obtaining a second prediction L1 based on a second motion vector MV1 from the current prediction block to the reference block in the second reference picture, determining whether a bidirectional optical flow (BDOF) operation is applied, and and calculating a bidirectional prediction of the current prediction block based on the first prediction L0 and the second prediction L1, and the first gradient value and the second gradient value.

Description

SYSTEM AND METHOD FOR COMBINED INTER AND INTRA PREDICTION

This application is based on and claims priority to Provisional Application No. 62/790,421 filed on January 9, 2019, the contents of which are incorporated herein by reference in their entirety.

This application relates to video coding and compression. More specifically, the present application relates to a Combined Inter and Intra Prediction (CIIP) method and apparatus for video coding.

Various video coding techniques may be used to compress video data. Video coding is performed according to one or more video coding standards. For example, video coding standards include versatile video coding (VVC), joint exploration test model (JEM), high-efficiency video coding (H.265/HEVC), and advanced video coding (H.264). /AVC), moving picture experts group (MPEG) coding, and the like. Video coding generally utilizes a prediction method (eg, inter-prediction, intra-prediction, etc.) that uses redundancy present in a video image or sequence. An important goal of video coding technology is to prevent or minimize video quality degradation while compressing video data into a format that uses a lower bit rate.

An example of the present disclosure provides a method for improving the efficiency of syntax signaling in a merge-related mode.

및 제1 수직 그래디언트 값

그리고 상기 제2 예측 L1과 연관된 제2 수평 그래디언트 값

및 제2 수직 그래디언트 값

을 계산함 -; 및 상기 제1 예측 L0 및 상기 제2 예측 L1, 제1 그래디언트 값

및

그리고 제2 그래디언트 값

및

에 기반하여 상기 현재 예측 블록의 이중 예측(bi-prediction)을 계산하는 단계를 포함한다. According to a first aspect of the present disclosure, a video coding method includes: obtaining a first reference picture and a second reference picture associated with a current predictive block, wherein the first reference picture in a display order is before a current picture, and the second reference picture is a reference picture is after the current picture; obtaining a first prediction L0 based on a first motion vector MV0 from the current prediction block to a reference block in the first reference picture; obtaining a second prediction L1 based on a second motion vector MV1 from the current prediction block to a reference block in the second reference picture; determining whether a bidirectional optical flow (BDOF) operation is applied, wherein the BDOF is a first horizontal gradient value for a prediction sample associated with the first prediction L0

and a first vertical gradient value

and a second horizontal gradient value associated with the second prediction L1.

and a second vertical gradient value.

Calculates -; and the first prediction L0 and the second prediction L1, a first gradient value

and

and the second gradient value

and

and calculating a bi-prediction of the current prediction block based on .

According to a second aspect of the present disclosure, a video coding method includes: obtaining a reference picture from a reference picture list associated with a current prediction block; generating an inter prediction based on a first motion vector from the current picture to a first reference picture; obtaining an intra prediction mode associated with the current prediction block; generating an intra prediction of the current prediction block based on the intra prediction; generating a final prediction of the current prediction block by averaging the inter prediction and the intra prediction; and determining whether the current prediction block is treated as an inter mode or an intra mode for intra mode prediction based on most probable mode (MPM).

및 제1 수직 그래디언트 값

그리고 상기 제2 예측 L1과 연관된 제2 수평 그래디언트 값

및 제2 수직 그래디언트 값

을 계산함 -; 및 상기 현재 예측 블록의 이중 예측을 계산하는 작동을 포함하는 작동들을 수행하게 한다. According to a third aspect of the present disclosure, there is provided a computer-readable non-transitory storage medium having instructions stored thereon. the instructions, when executed by the one or more processors, cause the computing device to obtain a first reference picture and a second reference picture associated with a current prediction block, wherein in display order the first reference picture is before the current picture, and the second reference picture is after the current picture; obtaining a first prediction L0 based on a first motion vector MV0 from the current prediction block to a reference block in the first reference picture; obtaining a second prediction L1 based on a second motion vector MV1 from the current prediction block to a reference block in the second reference picture; determining whether a bidirectional optical flow (BDOF) operation is applied, wherein the BDOF is a first horizontal gradient value for a prediction sample associated with the first prediction L0

and a first vertical gradient value

and a second horizontal gradient value associated with the second prediction L1.

and a second vertical gradient value.

Calculates -; and calculating a double prediction of the current prediction block.

According to a fourth aspect of the present disclosure, there is provided a computer-readable non-transitory storage medium having instructions stored thereon. The instructions, when executed by the one or more processors, cause the computing device to: obtain a reference picture from a reference picture list associated with a current prediction block; generating an inter prediction based on the first motion vector from the current picture to the first reference picture; obtaining an intra prediction mode associated with the current prediction block; generating an intra prediction of the current prediction block based on the intra prediction; generating a final prediction of the current prediction block by averaging the inter prediction and the intra prediction; and determining whether the current prediction block is treated as an inter mode or an intra mode for most probable mode (MPM)-based intra mode prediction.

It should be understood that both the foregoing general description and the following detailed description are illustrative only and not limiting of the present disclosure.

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate examples consistent with the disclosure, and together with the description serve to explain the principles of the disclosure.
1 is a block diagram of an encoder according to an example of the present disclosure.
2 is a block diagram of a decoder according to an example of the present disclosure.
3 is a flowchart illustrating a method of generating combined inter and intra prediction (CIIP) according to an example of the present disclosure.
4 is a flowchart illustrating a method of generating a CIIP according to an example of the present disclosure.
5A is a diagram illustrating a block partition in a multi-type tree structure according to an example of the present disclosure.
5B is a diagram illustrating a block partition in a multi-type tree structure according to an example of the present disclosure.
5C is a diagram illustrating a block partition in a multi-type tree structure according to an example of the present disclosure.
5D is a diagram illustrating a block partition in a multi-type tree structure according to an example of the present disclosure.
5E is a diagram illustrating a block partition in a multi-type tree structure according to an example of the present disclosure.
6A is a diagram illustrating combined inter and intra prediction (CIIP) according to an example of the present disclosure.
6B is a diagram illustrating combined inter and intra prediction (CIIP) according to an example of the present disclosure.
6C is a diagram illustrating combined inter and intra prediction (CIIP) according to an example of the present disclosure.
7A is a flowchart of an MPM candidate list generation process according to an example of the present disclosure.
7B is a flowchart of an MPM candidate list generation process according to an example of the present disclosure.
8 is a diagram illustrating a workflow of an existing CIIP design in VVC according to an example of the present disclosure.
9 is a diagram illustrating a workflow of the proposed CIIP method by removing BDOF according to an example of the present disclosure.
10 is a diagram illustrating a workflow of a single prediction-based CIIP for selecting a prediction list based on a POC distance according to an example of the present disclosure.
11A is a flowchart of a method when activating a CIIP block for generating an MPM candidate list according to an example of the present disclosure.
11B is a flowchart of a method for deactivating a CIIP block for generating an MPM candidate list according to an example of the present disclosure.
12 is a diagram illustrating a computing environment coupled with a user interface, according to an example of the present disclosure.

Reference will now be made in detail to examples of the present disclosure, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings in which the same numbers in different drawings indicate the same or similar elements, unless otherwise indicated. The implementations presented in the following description of examples of the present disclosure do not represent all implementations consistent with the present disclosure. Instead, these are merely examples of apparatus and methods consistent with aspects relevant to the present disclosure as recited in the appended claims.

The terminology used in the present disclosure is for the purpose of describing specific embodiments only and is not intended to limit the present disclosure. As used in this disclosure and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural as well, unless the context clearly indicates otherwise. It is intended Also, as used herein, the term “and/or” should be understood to mean and encompass any or all possible combinations of one or more associated listed items.

Herein, terms such as “first”, “second”, “third”, etc. may be used to describe various pieces of information, but it should be understood that the information should not be limited by these terms. These terms are only used to distinguish one category of information from another. For example, first information may be referred to as second information without departing from the scope of the present disclosure; Similarly, the second information may also be referred to as first information. As used herein, the term “if” may be understood to mean “when” or “upon” or “in response to judgment” depending on the context.

The first version of the HEVC standard was completed in October 2013, which provides about 50% bit rate savings or equivalent perceptual quality compared to the previous generation video coding standard H.264/MPEG AVC. Although the HEVC standard provides significant coding improvements over previous versions, there is evidence that greater coding efficiencies can be achieved with additional coding tools compared to HEVC. Based on this, VCEG and MPEG started the search for a new coding technology for future video coding standardization. A Joint Video Exploration Team (JVET) was formed in October 2015 by the ITU-T VECG and ISO/IEC MPEG to start significant research into advanced technologies that could significantly improve coding efficiency. One reference software, called the joint exploration model (JEM), was maintained in JVET by integrating several additional coding tools on top of the HEVC test model (HM).

In October 2017, ITU-T and ISO/IEC published a joint call for proposals (CfP) for video compression with capabilities beyond HEVC. In April 2018, 23 CfP responses were received and evaluated at the 10th JVET meeting, which showed a compression efficiency improvement of about 40% over HEVC. Based on these evaluation results, JVET started a new project to develop a next-generation video coding standard called Versatile Video Coding (VVC). In the same month, a reference software codebase called the VVC test model (VTM) was built to demonstrate a reference implementation of the VVC standard.

Like HEVC, VVC is built on top of a block-based hybrid video coding framework. 1 (discussed below) provides a block diagram of a typical block-based hybrid video encoding system. The input video signal is processed for each block (referred to as a coding unit (CU)). In VTM-1.0, a CU can be up to 128x128 pixels. However, unlike HEVC, which partitions a block based only on quad-trees, in VVC, one coding tree unit (CTU) is split into CUs to split a quad. It adapts to various local properties based on /binary/ternary trees. In addition, the concept of multiple partition unit type (multiple partition unit type) has been removed from HEVC, that is, the separation of CU, prediction unit (PU) and transform unit (TU) is no longer a VVC does not exist in; Instead, each CU is always used as a basic unit for prediction and transformation without additional partitions. In a multi-type tree structure, one CTU is first partitioned into a quadtree structure. Then, each quadtree leaf node can be further partitioned into binary and ternary tree structures. There are five splitting types, quaternary partitioning, horizontal binary partitioning, vertical binary, as shown in FIGS. 5A, 5B, 5C, 5D and 5E (described below). There are partitioning, horizontal ternary partitioning, and vertical ternary partitioning respectively.

In FIG. 1 (described below), spatial prediction and/or temporal prediction may be performed. Spatial prediction (or "intra prediction") uses pixels from already coded samples of a neighboring block (called a reference sample) in the same video picture/slice, Predict the current video block. Spatial prediction reduces the spatial redundancy inherent in the video signal. Temporal prediction (also called "inter prediction" or "motion compensated prediction") uses reconstructed pixels from an already coded video picture to predict a current video block. Temporal prediction reduces the temporal redundancy inherent in the video signal. The temporal prediction signal for a given CU is generally signaled by one or more motion vectors (MVs) indicating the amount and direction of motion between the current CU and its temporal reference. Also, when multiple reference pictures are supported, one reference picture index is additionally transmitted, which is used to identify from which reference picture in the reference picture store the temporal prediction signal comes. After spatial and/or temporal prediction, the mode decision block at the encoder selects the best prediction mode, for example based on a rate-distortion optimization method. A prediction block is subtracted from the current video block; The prediction residual is de-correlated and quantized using a transform.

The quantized residual coefficients are inverse quantized and inverse transformed to form a reconstructed residual, which is then added back to the prediction block to form the reconstructed signal of the CU. In-loop filtering, such as deblocking filter, sample adaptive offset (SAO), and adaptive in-loop filter (ALF), is a reference picture store It can be further applied to the reconstructed CU before being stored in the CU and used to code future video blocks. To form the output video bitstream, the coding mode (inter or intra), prediction mode information, motion information and quantized residual coefficients are all sent to an entropy coding unit to be further compressed and packed to form the bitstream. Packed.

2 (discussed below) provides a general block diagram of a block-based video decoder. The video bitstream is first entropy decoded in an entropy decoding unit. The coding mode and prediction information are transmitted to a spatial prediction unit (when intra-coded) or a temporal prediction unit (when inter-coded) to form a predictive block. The residual transform coefficients are transmitted to an inverse quantization unit and an inverse transform unit to reconstruct the residual block. Then, the prediction block and the residual block are added together. The reconstructed block may be further subjected to in-loop filtering before being stored in the reference picture store. The reconstructed video in the reference picture store is transmitted to drive the display device as well as used to predict future video blocks.

1 shows a typical encoder 100 . The encoder 100 includes a video input 110 , motion compensation 112 , motion estimation 114 , intra/inter mode determination 116 , a block predictor 140 , an adder 128 , Transform 130 , quantization 132 , prediction related info 142 , intra prediction 118 , picture buffer 120 , inverse quantization 134 , inverse transform 136 , adder 126 . , a memory 124 , an in-loop filter 122 , entropy coding 138 and a bitstream 144 .

2 shows a typical decoder 200 block diagram. Decoder 200 includes bitstream 210, entropy decoding 212, inverse quantization 214, inverse transform 216, adder 218, intra/inter mode selection 220, intra prediction 222, memory 230 , in-loop filter 228 , motion compensation 224 , picture buffer 226 , prediction related information 234 and video output 232 .

3 illustrates an example method 300 for generating combined inter and intra prediction (CIIP) in accordance with this disclosure.

In step 310, a first reference picture and a second reference picture associated with the current prediction block are obtained, wherein in display order the first reference picture is before the current picture and the second reference picture is after the current picture.

In step 312, a first prediction L0 is obtained based on the first motion vector MV0 from the current prediction block to the reference block in the first reference picture.

In step 314, a second prediction L1 is obtained based on the second motion vector MV1 from the current prediction block to the reference block of the second reference picture.

및 제1 수직 그래디언트 값

그리고 제2 예측 L1과 연관된 제2 수평 그래디언트 값

및 제2 수직 그래디언트 값

을 계산한다. In step 316, it is determined whether a bidirectional optical flow (BDOF) operation is applied, where the BDOF is combined with first horizontal and vertical gradient values for the prediction samples associated with the first prediction L0 and the second prediction L1 Calculate the associated second horizontal and vertical gradient values. For example, the BDOF is a first horizontal gradient value for a prediction sample associated with a first prediction L0.

and a first vertical gradient value.

and a second horizontal gradient value associated with the second prediction L1.

and a second vertical gradient value.

to calculate

및

그리고 제2 그래디언트 값

및

이다. In step 318, a bi-prediction of the current prediction block is calculated based on the first prediction L0 and the second prediction L1, the first gradient value and the second gradient value. For example, the first gradient value

and

and the second gradient value

and

to be.

4 illustrates an exemplary method 400 for generating a CIIP in accordance with the present disclosure. For example, this method includes uni-prediction-based inter prediction and MPM-based intra prediction for CIIP generation.

In step 410, a reference picture is obtained from a reference picture list associated with the current prediction block.

In step 412, an inter prediction is generated based on the first motion vector from the current picture to the first reference picture.

In step 414, an intra prediction mode associated with the current prediction block is obtained.

In step 416, an intra prediction of the current prediction block is generated based on the intra prediction.

In step 418, a final prediction of the current prediction block is generated by averaging the inter prediction and the intra prediction.

In step 420 , it is determined whether the current prediction block is treated as an inter mode or an intra mode for most probable mode (MPM)-based intra mode prediction.

5A shows a diagram illustrating a block quaternary partition in a multi-type tree structure according to an example of the present disclosure.

5B shows a diagram illustrating a block vertical binary partition in a multi-type tree structure according to an example of the present disclosure.

5C shows a diagram illustrating a block horizontal binary partition in a multi-type tree structure according to an example of the present disclosure.

5D shows a diagram illustrating a block vertical ternary partition in a multi-type tree structure according to an example of the present disclosure.

5E shows a diagram illustrating a block horizontal ternary partition in a multi-type tree structure according to an example of the present disclosure.

Combined Inter and Intra Prediction

1 and 2 , inter and intra prediction methods are used in a hybrid video coding scheme, where each PU selects only inter prediction or intra prediction to utilize correlation in the temporal or spatial domain. can be used, but not both. However, as pointed out in the previous literature, the residual signal generated by the inter-predicted block and the intra-predicted block may exhibit very different characteristics. Therefore, when the two types of prediction can be efficiently combined, one more accurate prediction can be expected to increase the coding efficiency by reducing the energy of the prediction residual. Additionally, motion of a moving object in nature video content can be complex. For example, there may be an area that contains both old content (eg, objects included in a previously coded picture) and new content (eg, objects excluded from previously coded pictures). . In this scenario, neither inter prediction nor intra prediction can provide one accurate prediction for the current block.

In order to further improve prediction efficiency, the VVC standard adopts CIIP (Combined Inter and Intra Prediction), which combines intra prediction and inter prediction of one CU coded in merge mode. Specifically, for each merge CU, one additional flag is signaled to indicate whether CIIP is enabled for the current CU. For a luma component, CIIP supports four frequently used intra modes, including a planar mode, a DC mode, a horizontal mode, and a vertical mode. In the case of a chroma component, DM (ie, chroma reuses the same intra mode of the luma component) is always applied without additional signaling. Additionally, in the existing CIIP design, a weighted average is applied to combine the inter prediction sample and the intra prediction sample of one CIIP CU. Specifically, when selecting the planar mode or DC mode, the same weight (ie, 0.5) is applied. Otherwise (ie, horizontal mode or vertical mode is applied), the current CU is first split vertically (for horizontal mode) or vertically (for vertical mode) into four equal-size areas.

Four sets of weights, demoted by ( w_intra _i , w_inter _i ) are applied to combine inter-prediction samples and intra-prediction samples in different regions, where i = 0 and i = 3 are used for intra prediction Shows the regions closest to and farthest from the reconstructed neighboring sample. In the current CIIP design, the values of the weight set are ( w_intra ₀ , w_inter ₀ ) = (0.75, 0.25) , ( w_intra ₁ , w_inter ₁ ) = (0.625, 0.375) , ( w_intra ₂ , w_inter ₂ ) = (0.375, 0.625) ) and ( w_intra ₃ , w_inter ₃ ) = (0.25, 0.75) . 6A, 6B, and 6C (described below) provide examples for explaining the CIIP mode.

Additionally, in the current VVC working specification, the intra mode of one CIIP CU may be used as a predictor for predicting the intra mode of a neighboring CIIP CU through a most probable mode (MPM) mechanism. Specifically, for each CIIP CU, when a neighboring block is also a CIIP CU, the intra mode of these neighboring blocks is first rounded to the nearest mode within the planar mode, DC mode, horizontal mode and vertical mode, and then is added to the MPM candidate list of the current CU. However, when constructing the MPM list for each intra CU, if one of the neighboring blocks is coded in CIIP mode, it is considered unavailable, that is, the intra mode of one CIIP CU is the intra mode of its neighboring intra CU. It is not allowed to predict the mode. 7A and 7B (described below) compare the MPM list generation process of an intra CU and a CIIP CU.

Bi-Directional Optical Flow

Conventional double prediction in video coding is a simple combination of two temporal prediction blocks obtained from an already reconstructed reference picture. However, due to a limitation of block-based motion compensation, a small observable motion remains between samples of two prediction blocks, so that the efficiency of motion compensation prediction may be reduced. To solve this problem, a bi-directional optical flow (BDOF) is applied to the VVC to lower the effect of this motion on all samples within one block.

은 BDOF가 서브 블록 주변의 하나의 6×6 윈도(window) Ω 내에 적용된 후 L0 예측 샘플과 L1 예측 샘플 사이의 차이를 최소화하는 것에 의해 계산된다. 구체적으로,

의 값은 다음:Specifically, as shown in FIGS. 6A, 6B, and 6C (described below), BDOF is a sample-by-sample motion adjustment performed on top of block-based motion compensated prediction when dual prediction is used. -wise motion refinement). Motion adjustment of each 4×4 sub-block

is calculated by minimizing the difference between the L0 prediction sample and the L1 prediction sample after the BDOF is applied within one 6×6 window Ω around the sub-block. Specifically,

The values of are:

은 플로어 함수(floor function)이며; clip3(min, max, x)는 [min, max] 범위 내에서 주어진 값 x를 클리핑하는(clip) 함수이고; 심볼 >>는 비트의 오른쪽 시프트 연산(bitwise right shift operation)을 나타내며; 심볼 <<는 비트의 왼쪽 시프트 연산(bitwise left shit operation)을 나타내고;

는 불규칙한 로컬 모션으로 인한 전파 에러(error)를 방지하기 위한 모션 미세 조정 임계 값(the motion refinement threshold)이며, 이는

와 같으며, 여기서

는 입력 비디오의 비트 깊이(bit-depth)이다. (1)에서,

이다. is derived as, where

is the floor function; clip3(min, max, x) is a function that clips a given value x within the range [min, max]; The symbol >> indicates a bitwise right shift operation; The symbol << denotes a bitwise left shit operation;

is the motion refinement threshold to avoid propagation errors due to irregular local motion, which is

is equal to, where

is the bit-depth of the input video. In (1),

to be.

및

의 값은 다음:

and

The values of are:

It can be calculated as, where

는 리스트 k에서의 예측 신호의 좌표

에서의 샘플 값이며 - k=0, 1임 - 이는 중간 고정밀도(intermediate high precision)(즉, 16-비트)에서 생성되며;

및

는 그의 2개의 이웃 샘플 사이의 차이를 직접 계산하는 것에 의해 획득된 샘플의 수직 그래디언트 및 수평 그래디언트이며, 즉,ego,

is the coordinates of the prediction signal in list k

is the sample value in − k=0, 1 − it is generated at intermediate high precision (ie 16-bit);

and

is the vertical and horizontal gradients of a sample obtained by directly calculating the difference between its two neighboring samples, i.e.,

이다.

to be.

Based on the motion coordination derived in (1), the final double prediction sample of the CU is calculated by interpolating the L0/L1 prediction sample along the motion trajectory based on the optical flow model, as follows:

및

은 이중 예측에 대한 L0 예측 신호 및 L1 예측 신호를 조합하기 위해 적용되는 오른쪽 시프트 값과 오프셋 값이며, 이는 각각

및

와 같다. is indicated as, where

and

is the right shift value and the offset value applied to combine the L0 prediction signal and the L1 prediction signal for double prediction, which are respectively

and

same as

6A shows a diagram illustrating combined inter and intra prediction for a horizontal mode according to an example of this disclosure.

6B shows a diagram illustrating combined inter and intra prediction for vertical mode according to an example of this disclosure.

6C shows a diagram illustrating combined inter and intra prediction for planar mode and DC mode, according to an example of this disclosure.

7A shows a flowchart of an MPM candidate list generation process of an intra CUS according to an example of the present disclosure.

7B shows a flowchart of an MPM candidate list generation process of a CIIP CU according to an example of the present disclosure.

CIIP Improvement

Although CIIP can improve the efficiency of conventional motion compensated prediction, its design can still be further improved. Specifically, in the existing CIIP design of VVC, the following problems are identified in the present disclosure.

First, as discussed in the “Combined Inter and Intra Prediction” section, since CIIP combines samples of inter and intra prediction, each CIIP CU must use the reconstructed neighboring samples to generate the prediction signal. This means that the decoding of one CIIP CU depends on the entire reconstruction of the neighboring block. Due to this interdependence, for actual hardware implementation, CIIP must be performed at a reconstruction stage where neighboring reconstructed samples can be used for intra prediction. Since decoding of CUs in the reconstruction stage must be performed sequentially (i.e., one by one), to ensure sufficient throughput of real-time decoding, if the number of computational operations (e.g., multiplication, addition, and bit shift) involved in the CIIP process is too large, Can not be done.

As mentioned in the “Bidirectional Optical Flow” section, BDOF can improve the prediction quality when one inter-coded CU is predicted from two reference blocks in the forward temporal direction and the backward temporal direction. As shown in FIG. 8 (described below), in current VVC, BDOF is also included to generate inter prediction samples for CIIP mode. Given the additional complexity introduced by BDOF, this design can significantly lower the encoding/decoding throughput of the hardware codec when CIIP is enabled.

Second, in the current CIIP design, when one CIIP CU refers to one double-predicted merging candidate, both the motion compensation prediction signals of the lists L0 and L1 must be generated. When one or more MVs are not integer precision, an additional interpolation process must be invoked to interpolate the samples at fractional sample positions. This process not only increases computational complexity, but also increases memory bandwidth when more reference samples need to be accessed from external memory.

Third, as discussed in the “Combined Inter and Intra Prediction” section, in the current CIIP design, the intra mode of a CIIP CU and the intra mode of an intra CU are treated differently when constructing the MPM list of a neighboring block. Specifically, when one current CU is coded in CIIP mode, its neighboring CIIP CU is considered intra, that is, the intra mode of the neighboring CIIP CU may be added to the MPM candidate list. However, when the current CU is coded in intra mode, its neighboring CIIP CU is considered as inter, that is, the intra mode of the neighboring CIIP CU is excluded from the MPM candidate list. This non-integrated design may not be suitable for the final version of the VVC standard.

8 is a diagram illustrating a workflow of an existing CIIP design in VVC according to an example of the present disclosure.

Simplifying CIIP

In this disclosure, a method of simplifying an existing CIIP design to facilitate hardware codec implementation is provided. In general, the main aspects of the technology proposed in the present disclosure are summarized as follows.

First, in order to improve CIIP encoding/decoding throughput, we propose to exclude BDOF from inter prediction sample generation in CIIP mode.

Second, in order to reduce computational complexity and memory bandwidth consumption, when one CIIP CU is double-predicted (i.e., contains both L0 MV and L1 MV), the block is converted from double-prediction to single-prediction to generate inter-prediction samples. suggest

Third, when forming an MPM candidate of a neighboring block, two methods for harmonizing the intra mode of CIIP and CU are proposed.

CIIP without BDOF

As pointed out in the "problem description" section, BDOF is always activated to generate inter prediction samples for CIIP mode when the current CU is double predicted. Due to the additional complexity of BDOF, existing CIIP designs can significantly lower the encoding/decoding throughput, making real-time decoding difficult, especially in VVC decoders. On the other hand, in the case of CIIP CU, the final prediction sample is generated by averaging the inter prediction sample and the intra prediction sample. In other words, a prediction sample refined by BDOF is not directly used as a prediction signal for a CIIP CU. Thus, compared to the conventional double predicted CU, where the BDOF is directly applied to generate the prediction sample, the corresponding improvement obtained from the BDOF is less efficient for the CIIP CU. Therefore, based on the above considerations, it is proposed to disable BDOF when generating inter prediction samples of CIIP mode. 9 (discussed below) illustrates the corresponding workflow of the proposed CIIP process after removing the BDOF.

9 is a diagram illustrating a workflow of a CIIP method by BDOF removal proposed according to an example of the present disclosure.

CIIP based on a single prediction

As discussed above, when a merge candidate referenced by one CIIP CU is double predicted, both an L0 prediction signal and an L1 prediction signal are generated to predict a sample inside the CU. To reduce memory bandwidth and interpolation complexity, in an embodiment of the present disclosure, only inter prediction samples generated using single prediction are used (even if the current CU is double predicted) and combined with intra prediction samples in CIIP mode . Specifically, when the current CIIP CU is single predicted, the inter prediction sample will be directly combined with the intra prediction sample. Otherwise (ie, the current CU is double predicted), the inter prediction sample used by CIIP is generated based on single prediction from one prediction list (L0 or L1). Different methods may be applied to select the prediction list. In the first method, it is proposed to always select the first prediction (ie list L0) for any CIIP block predicted by two reference pictures.

In the second method, it is proposed to always select the second prediction (ie list L1) for a CIIP block predicted by two reference pictures. In the third method, one adaptive method is applied in which a prediction list associated with one reference picture having a smaller Picture Order Count (POC) distance from the current picture is selected. 10 (discussed below) illustrates the workflow of a single prediction based CIIP to select a prediction list based on POC distance.

Finally, in the last method, when the current CU is single predicted, we propose a method of only activating the CIIP mode. In addition, in order to reduce overhead, signaling of a CIIP enabling/disabling flag varies according to a prediction direction of a current CIIP CU. When the current CU is single predicted, the CIIP flag is signaled in the bitstream to indicate whether CIIP is activated or deactivated. Otherwise (ie, the current CU is double predicted), the signaling of the CIIP flag is skipped and always inferred to be false, ie, CIIP is always deactivated.

10 is a diagram illustrating a workflow of a single prediction based CIIP for selecting a prediction list based on a POC distance, according to an example of the present disclosure.

Harmonization of intra mode of CIIP and intra mode of intra CU for MPM candidate list configuration

As discussed above, the current CIIP design is not unified with respect to the intra mode of the CIIP CU and the method of forming the MPM candidate list of the neighboring block using the intra mode of the intra CU. Specifically, both the intra mode of the CIIP CU and the intra mode of the intra CU may predict the intra mode of a neighboring block coded in the CIIP mode. However, only the intra mode of the intra CU can predict the intra mode of the intra CU. To achieve one or more integrated designs, in this section, we propose two methods to reconcile the use of intra mode of CIIP and intra mode of intra CU for MPM list construction.

In the first method, it is proposed to treat the CIIP mode as an inter mode for configuring the MPM list. Specifically, when generating the MPM list of one CIIP CU or one intra CU, if the neighboring block is coded in the CIIP mode, the intra mode of the neighboring block is marked as unavailable. In this way, the MPM list cannot be constructed using the intra mode of the CIIP block. Conversely, in the second method, a method of treating the CIIP mode as an intra mode for building the MPM list is proposed. Specifically, in this method, the intra mode of a CIIP CU can predict the intra mode of both the neighboring CIIP block and the intra block. 11A and 11B (described below) illustrate the MPM candidate list generation process when the above two methods are applied.

Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the present disclosure disclosed herein. This application is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include departures from the scope of known or customary practice in this disclosure. The specification and examples are intended to be considered by way of example only, with the true scope and spirit of the disclosure being indicated by the following claims.

It will be understood that the present disclosure is not limited to the precise examples described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. It is intended that the scope of the present disclosure be limited only by the appended claims.

11A is a flowchart illustrating a method of activating a CIIP block for generating an MPM candidate list according to an example of the present disclosure.

11B is a flowchart illustrating a method of deactivating a CIIP block for generating an MPM candidate list according to an example of the present disclosure.

12 illustrates a computing environment 1210 coupled with a user interface 1260 . Computing environment 1210 may be part of a data processing server. Computing environment 1210 includes processor 1220 , memory 1240 , and I/O interface 1250 .

Processor 1220 generally controls the overall operation of computing environment 1210, such as operations related to display, data acquisition, data communication, and image processing. Processor 1220 may include one or more processors for executing instructions to perform all or some steps of the method described above. Moreover, the processor 1220 may include one or more circuitry that facilitates interaction between the processor 1220 and other components. The processor may be a central processing unit (CPU), a microprocessor, a single chip machine, a GPU, or the like.

Memory 1240 is configured to store various types of data to support operation of computing environment 1210 . Examples of such data include instructions for any application or method running on computing environment 1210 , video data, image data, and the like. The memory 1240 includes static random access memory (SRAM), electrical erasable programmable read-only memory (EEPROM), electrically erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), and read-only memory (ROM). ), magnetic memory, flash memory, magnetic or optical disks, or any type of volatile or non-volatile memory device, or a combination thereof.

The I/O interface 1250 provides an interface between the processor 1220 and peripheral interface modules such as a keyboard, a click wheel, a button, and the like. The buttons may include, but are not limited to, a home button, a scan start button, and a scan stop button. I/O interface 1250 may be coupled to an encoder and decoder.

In one embodiment, a computer-readable non-transitory storage medium comprising a plurality of programs 1242 included in the memory 1240 executable by the processor 1220 in the computing environment 1210 to perform the method described above. is also provided. For example, the computer-readable non-transitory storage medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, or the like.

The computer-readable non-transitory storage medium stores therein a plurality of programs that are executed by a computing device having one or more processors, wherein the plurality of programs, when executed by the one or more processors, cause the computing device to perform the motion prediction method described above. to do

In one embodiment, the computing environment 1210 includes one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices ( digital signal processing device (DSPD), programmable logic device (PLD), field-programmable gate array (FPGA), graphic processing unit (GPU), controller, microcontroller , a microprocessor or other electronic components.

Claims (20)

A video coding method comprising:
The video coding method comprises:
obtaining a reference picture from a reference picture list associated with a current coding block;
generating an inter prediction based on a motion vector from a current picture to the reference picture;
obtaining an intra prediction mode associated with the current coding block;
generating an intra prediction of the current coding block based on the intra prediction mode;
generating a final prediction of the current coding block by weighted averaging of the inter prediction and the intra prediction; and
Determining that the current coding block is treated as one unified mode when constructing a most probable mode (MPM) list of neighboring coding blocks - The unified mode is in all the coding blocks predicted in the same way as the current coding block pre-defined as inter mode for, or the integration mode is predefined as intra mode for all coding blocks predicted in the same way as the current coding block;
A video coding method comprising a. According to claim 1,
when the current coding block is predicted from one reference picture in a reference picture list L0, the reference picture list is L0. According to claim 1,
when the current coding block is predicted from one reference picture in a reference picture list L1, the reference picture list is L1. According to claim 1,
When the current coding block is predicted from one first reference picture in reference picture list L0 and one second reference picture in reference picture list L1, the reference picture list is L0. According to claim 1,
When the current coding block is predicted from one first reference picture in reference picture list L0 and one second reference picture in reference picture list L1, the reference picture list is L1. According to claim 1,
The reference picture list has a smaller POC ( picture order count) associated with one reference picture with a distance. According to claim 1,
The intra prediction mode of the current coding block is used for MPM-based intra mode prediction of the neighboring coding block. According to claim 1,
A video coding method, wherein a bidirectional optical flow (BDOF) operation is disabled for the current coding block. A video coding device comprising:
one or more processors and one or more storage coupled to the one or more processors;
The video coding device comprises:
obtaining a reference picture from a reference picture list associated with a current coding block;
generating an inter prediction based on a motion vector from a current picture to the reference picture;
obtaining an intra prediction mode associated with the current coding block;
generating an intra prediction of the current coding block based on the intra prediction mode;
generating a final prediction of the current coding block by weighted averaging the inter prediction and the intra prediction; and
An operation of determining that the current coding block is treated as one unified mode when constructing a most probable mode (MPM) list of neighboring coding blocks - The unified mode is applied to all coding blocks predicted in the same way as the current coding block pre-defined as inter mode for, or the integration mode is predefined as intra mode for all coding blocks predicted in the same way as the current coding block;
A video coding device for performing operations comprising: 10. The method of claim 9,
when the current coding block is predicted from one reference picture in a reference picture list L0, the reference picture list is L0. 10. The method of claim 9,
When the current coding block is predicted from one reference picture in a reference picture list L1, the reference picture list is L1. 10. The method of claim 9,
When the current coding block is predicted from one first reference picture in reference picture list L0 and one second reference picture in reference picture list L1, the reference picture list is L0. 10. The method of claim 9,
When the current coding block is predicted from one first reference picture in reference picture list L0 and one second reference picture in reference picture list L1, the reference picture list is L1. 10. The method of claim 9,
The reference picture list is a smaller picture for the current picture when the current coding block is predicted from one first reference picture in the reference picture list L0 and one second reference picture in the reference picture list L1. order count) distance associated with one reference picture. 10. The method of claim 9,
The intra prediction mode of the current coding block is used for MPM-based intra mode prediction of the neighboring coding block. 10. The method of claim 9,
wherein bidirectional optical flow (BDOF) operation is disabled for the current coding block. A computer-readable non-transitory storage medium storing a plurality of programs for execution by a computing device having one or more processors, comprising:
9. A computer-readable non-transitory storage medium, wherein the plurality of programs, when executed by the one or more processors, cause the computing device to perform the method according to any one of claims 1 to 8. A computer program stored in a computer-readable storage medium, comprising:
contains a command,
A computer program stored in a computer-readable storage medium, wherein said instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8. delete delete

Images