TWI705696B - Improvement on inter-layer prediction - Google Patents

Abstract

Devices, systems and methods for video processing are described. In a representative aspect, a video processing method includes determining, based on a component type of a current video block, whether an inter-layer prediction mode is applicable to a conversion between the current video block and a bitstream representation of the current video block, and performing the conversion by applying the inter-layer prediction mode due to the determining that the inter-layer prediction mode is applicable to the current video block, wherein the applying the inter-layer prediction includes dividing a portion of the current video block into at least one sub-block using more than one dividing patterns and generating a predictor for the current video block as a weighted average of predictors determined for each of the more than one dividing patterns.

Description

Interweaving prediction improvements

This patent document relates to a video encoding and decoding technology, device and system.

Despite advances in video compression, digital video accounts for the largest broadband usage on the Internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, it is expected that the broadband demand for digital video usage will continue to grow.

This document discloses a technique that can be used in video encoding and decoding embodiments to improve the performance of sub-block-based encoding, and in particular, when an affine motion encoding mode is used.

In one embodiment, a video processing method is provided, including: determining whether an interlaced prediction mode is suitable for conversion between the current video block and the bitstream representation of the current video block based on the component type of the current video block; and responding to It is determined that the interleaving prediction mode is suitable for the current video block, and the conversion is performed by applying the interleaving prediction mode, where applying interleaving prediction includes subdividing part of the current video block into at least one sub-block using more than one subdivision pattern, and generating as a pair of more than one subdivision Each of the patterns determines the weighted average predictor of the current video block.

In another embodiment, a video processing method is provided, including: based on the prediction direction of the current video block, determining whether the interleaving prediction mode is suitable for conversion between the current video block and the bitstream representation of the current video block; and responding In order to determine that the interlaced prediction mode is suitable for the current video block, conversion is performed by applying the interlaced prediction mode, where applying interlaced prediction includes subdividing part of the current video block into at least one sub-block using more than one subdivision pattern, and generating as a pair of more than one subdivision Each of the patterns determines the weighted average predictor of the current video block.

In another embodiment, a video processing method is provided, which includes: based on the low-delay mode of the current picture, determining whether the interlaced prediction mode is applicable between the current video block in the current picture and the bitstream representation of the current video block Conversion; and in response to determining that the interleaving prediction mode is suitable for the current video block, performing conversion by applying the interleaving prediction mode, and wherein applying the interleaving prediction includes subdividing part of the current video block into at least one sub-block using more than one subdivision pattern, and generating The predictor of the current video block as a weighted average of the predictors determined for each of more than one subdivision pattern.

In another embodiment, a video processing method is provided, including: determining whether an interlaced prediction mode is applicable between the current video block and the bitstream representation of the current video block based on using a current picture containing the current video block as a reference Converting; and in response to determining that the interleaving prediction mode is suitable for the current video block, performing conversion by applying the interleaving prediction mode, and wherein applying the interleaving prediction includes subdividing part of the current video block into at least one sub-block style using more than one subdivision pattern, and A predictor of the current video block as a weighted average of the determined predictors for each of more than one subdivision patterns is generated.

In another embodiment, a video processing method is provided, including: selectively performing one or more of the luminance component, the first chrominance component, and the second chrominance component from the video frame of the video based on the video condition Coding of components based on interleaving prediction, where performing interleaving prediction includes determining a prediction block from the current block of the video components: selecting a set of pixels of the components of the video frame to form a block; dividing the block into sub-blocks according to the first pattern Generate a first intermediate prediction block based on the first set of sub-blocks; divide the block into a second set of sub-blocks according to the second pattern, wherein at least one sub-block in the second set is not in the first set; A second intermediate prediction block is generated based on the second set of sub-blocks; and the prediction block is determined based on the first intermediate prediction block and the second intermediate prediction block.

In yet another embodiment, a video encoder device implementing the video encoding method described herein is disclosed.

In yet another representative aspect, the various technologies described in this article are implemented as non-transitory computers Computer program products stored on readable media. The computer program product contains program code for performing the methods described in this article.

In yet another representative aspect, a video decoder device may implement the method described herein.

The details of one or more implementation manners are proposed in the attached appendices, drawings, and description below. Other features will become apparent from the description and drawings and claims.

v ₁ , v ₀ , v _x , v _y , MVx, MVy: vector

A to E, 1300 to 1305, P, P _i: the video block

Wa, Wb: weight value

τ ₀ , τ ₁ : distance

Ref0, Ref1: reference frame

1800: Video processing equipment

1802: processor

1804: memory

1806: Video processing circuit

1900, 2000, 2100, 2200, 2300: method

1902 to 1904, 2002 to 2004, 2102 to 2104, 2202 to 2204, 2302 to 2312: steps

Figure 1 shows an example of sub-block-based prediction.

Figure 2 shows an example of a simplified affine motion model.

Figure 3 shows an example of the affine motion vector field (MVF) of each sub-block.

Figure 4 shows an example of motion vector prediction (MVP) in AF_INTER mode.

Figures 5A and 5B show examples of candidates for the AF_MERGE coding mode.

Figure 6 shows an exemplary process of motion prediction by the advanced temporal motion vector predictor (ATMVP) of the coding unit (CU).

Figure 7 shows an example of a CU with four sub-blocks (A-D) and its neighboring blocks (a-d).

Figure 8 shows an example of optical flow trajectories in video encoding.

Figures 9A and 9B show examples of bidirectional optical (BIO) encoding techniques without block extension. FIG. 9A shows an example of access locations outside of the block, and FIG. 9B shows an example of padding used in order to avoid additional memory access and calculation.

Figure 10 shows an example of bilateral matching.

Figure 11 shows an example of template matching.

Figure 12 shows an example of unilateral motion estimation (ME) in frame rate up conversion (FRUC).

Figure 13 shows an exemplary implementation of interleaving prediction.

Figures 14A to 14C show examples of partial interleaving prediction. The dotted line represents the first subdivision style; the solid line Indicates the second subdivision style; the thick line indicates the area where interlaced prediction is applied. Outside this area, no interleaving prediction is applied.

Figure 15 shows an example of weight values in sub-blocks. Exemplary weight values {Wa, Wb} are {3, 1}, {7, 1}, {5, 3}, {13, 3}, etc.

Figure 16 shows an example of interleaving prediction with two subdivision patterns according to the technology of the present disclosure.

FIG. 17A shows an exemplary subdivision pattern in which a block is subdivided into 4×4 sub-blocks according to the technology of the present disclosure.

FIG. 17B shows an exemplary subdivision pattern in which a block is subdivided into 8×8 sub-blocks according to the technology of the present disclosure.

FIG. 17C shows an exemplary subdivision pattern in which a block is subdivided into 4×8 sub-blocks according to the technology of the present disclosure.

FIG. 17D shows an exemplary subdivision pattern in which a block is subdivided into 8×4 sub-blocks according to the technology of the present disclosure.

FIG. 17E shows an exemplary subdivision pattern in which a block is subdivided into non-uniform sub-blocks according to the technology of the present disclosure.

FIG. 17F shows another exemplary subdivision pattern in which a block is subdivided into non-uniform sub-blocks according to the technology of the present disclosure.

FIG. 17G shows another exemplary subdivision pattern in which a block is subdivided into non-uniform sub-blocks according to the technology of the present disclosure.

Figure 18 is a block diagram of an example of a hardware platform used to implement the video processing method described in this document.

Figure 19 is a flowchart of an exemplary method of video processing described in this document.

Figure 20 is a flowchart of another exemplary method of video processing described in this document.

Figure 21 is a flowchart of another exemplary method of video processing described in this document.

Figure 22 is a flowchart of another exemplary method of video processing described in this document.

Figure 23 is a flowchart of another exemplary method of video processing described in this document.

Chapter titles are used in this document to improve readability, and the techniques and embodiments described in the chapters are not limited to only this chapter.

To improve the compression ratio of video, researchers continue to seek new technologies for encoding video.

1 Introduction

The present invention relates to video/image coding technology. Specifically, it relates to sub-block-based prediction in video/image coding. It can be applied to existing video coding standards such as HEVC, or a standard to be finalized (Versatile Video Coding). It can also be applied to future video/image coding standards or video/image codecs. The present invention can further improve P1805026601.

Brief discussion

Sub-block-based prediction was first introduced into the video coding standard through HEVC Appendix I (3D-HEVC). Through sub-block-based prediction, a block (such as a coding unit (CU) or a prediction unit (PU)) is subdivided into several non-overlapping sub-blocks. Different sub-blocks can be assigned different motion information, such as reference index or motion vector (MV), and motion compensation (MC) is performed on each sub-block individually. Figure 1 shows the concept of sub-block-based prediction.

In order to explore future video coding technologies beyond HEVC, VCEG and MPEG jointly established a Joint Video Exploration Team (JVET) in 2015. Since then, JVET has adopted many new methods and incorporated them into a reference software called Joint Exploration Model (JEM).

In JEM, sub-block-based prediction is used in several coding tools, such as affine prediction, optional temporal motion vector prediction (ATMVP), space-time motion vector prediction (STMVP), bidirectional optical flow (BIO), and frame Rate up conversion (FRUC).

2.1 Affine prediction

In HEVC, only the translational motion model is applied for motion compensation prediction (MCP). and In the real world, there are many kinds of movement, such as zoom in/out, rotation, perspective movement, and other irregular movements. In JEM, a simplified affine transform motion compensation prediction is applied. As shown in Figure 2, the affine motion field of the block is described by two control point motion vectors.

The motion vector field (MVF) of the block is described by the following equation:

Where ( v _0x , v _0y ) is the motion vector of the control point at the left top corner, and ( v _1x , v _1y ) is the motion vector of the control point at the right top corner.

To further simplify the motion compensation prediction, sub-block-based affine transformation prediction is applied. The sub-block size M × N is derived from equation (2), where MvPre is the accuracy of the motion vector score (1/16 in JEM), and ( v _2x , v _2y ) is the motion vector of the lower left control point, which is based on Formula (1) calculation.

After deriving from equation (2), if necessary, M and N should be adjusted downward so that they are divisors of w and h, respectively.

As shown in Figure 3, in order to derive the motion vector of each M×N sub-block, the motion vector of the center sample of each sub-block is calculated according to equation (1) and rounded to 1/16 fractional accuracy. Then, a motion compensation interpolation filter is applied to use the derived motion vector to generate a prediction for each sub-block.

After the MCP, the high-precision motion vector of each sub-block is rounded and saved with the same accuracy as the normal motion vector.

In JEM, there are two affine motion modes: AF_INTER mode and AF_MERGE mode. For CUs whose width and height are both greater than 8, AF_INTER mode can be applied. The CU-level affine flag is signaled in the bit stream to indicate whether to use the AF_INTER mode. In this mode, neighboring blocks are used to construct a candidate with motion vector pairs {(v ₀ ,v ₁ )|v ₀ ={v _A ,v _B ,v _c },v ₁ ={v _D ,v _E }} List. As shown in Figure 4, select v ₀ from the motion vectors of block A, block B, or block C. The motion vector from the neighboring block is scaled according to the reference list and according to the relationship between the referenced POC of the neighboring block, the referenced POC of the current CU, and the POC of the current CU. And the method of selecting v ₁ from adjacent blocks D and E is similar. If the number of the candidate list is less than 2, the list is filled with a pair of motion vectors composed by duplicating each AMVP candidate. When the candidate list is greater than 2, the candidates are first classified according to the consistency of adjacent motion vectors (the similarity of the two motion vectors in the candidate pair), and only the first two candidates are retained. The RD cost check is used to determine which motion vector pair candidate is selected as the control point motion vector prediction (CPMVP) of the current CU. And, an index indicating the position of the CPMVP in the candidate list is signaled in the bit stream. After determining the CPMVP of the current affine CU, apply affine motion estimation and find the control point motion vector (CPMV). Then signal the difference between CPMV and CPMVP in the bit stream.

When the CU is applied in the AF_MERGE mode, it obtains the first block encoded using the affine mode from the valid neighboring reconstructed block. As shown in Fig. 5A, the selection order for candidate blocks is from left, top, top right, bottom left to top left. As shown in Figure 5B, if the adjacent lower left block A is coded in affine mode, the motion vectors v ₂ , v ₃ and v _{4 of the} top left corner, top right corner, and bottom left corner of the CU containing block A are derived . And calculate the motion vector v _{0 of the} top left corner of the current CU according to v ₂ , v ₃ and v ₄ . Second, calculate the motion vector v ₁ of the upper right of the current CU.

After deriving the CPMV v ₀ and v ₁ of the current CU, the MVF of the current CU is generated according to the simplified affine motion model equation (1). In order to identify whether the current CU uses AF_MERGE mode coding, when there is at least one neighboring block coded in affine mode, an affine flag is signaled in the bit stream.

2.2 ATMVP

In the optional temporal motion vector prediction (ATMVP) method, the motion vector temporal motion vector prediction (TMVP) is modified by extracting multiple sets of motion information (including motion vectors and reference indices) from blocks smaller than the current CU. As shown in Figure 6, the sub-CU is a square N×N block (N is set to 4 by default).

ATMVP predicts the motion vector of the sub-CU in the CU in two steps. The first step is to use the so-called time The vector identifies the corresponding block in the reference picture. The reference picture is also called the motion source picture. The second step is to divide the current CU into sub-CUs, and obtain the motion vector and reference index of each sub-CU from the block corresponding to each sub-CU, as shown in Figure 6.

In the first step, the reference picture and the corresponding block are determined by the motion information of the spatial neighboring blocks of the current CU. In order to avoid copy scan processing of adjacent blocks, the first MERGE candidate in the MERGE candidate list of the current CU is used. The first available motion vector and its associated reference index are set as the index of the time vector and the motion source picture. In this way, compared with TMVP, the corresponding block can be identified more accurately in ATMVP, where the corresponding block (sometimes called a collocated block) is always located in the lower right corner or the center position relative to the current CU.

In the second step, by adding the time vector to the coordinates of the current CU, the corresponding block of the sub-CU is identified by the time vector in the motion source picture. For each sub-CU, the motion information of its corresponding block (the smallest motion grid covering the center sample) is used to derive the motion information of the sub-CU. After the motion information of the corresponding N×N block is identified, it is converted into the motion vector and reference index of the current sub-CU in the same way as the TMVP of HEVC, where motion scaling and other programs are applied. For example, the decoder checks whether the low-delay condition is satisfied (that is, the POC of all reference pictures of the current picture is less than the POC of the current picture), and may use the motion vector MVx (the motion vector corresponding to the reference picture list X) to predict each sub The motion vector MVy of the CU (X is equal to 0 or 1 and Y is equal to 1-X).

3.STMVP

In this method, the motion vector of the sub-CU is recursively derived in the raster scan order. Figure 7 illustrates this concept. Let us consider an 8×8 CU 700, which contains four 4×4 sub-CUs A, B, C, and D. The adjacent 4×4 blocks in the current frame are labeled a, b, c, and d.

The motion derivation of sub CU A starts by identifying its two spatial neighbors. The first neighbor is the N×N block (block c) above the sub CU A. If the block c is not available or internally coded, check the other N×N blocks above the sub-CU A (from left to right, starting at block c). The second neighbor is a block to the left of the sub CU A (block b). If block b is not available or is internally coded, check the other blocks on the left of sub-CU A (from top to bottom, starting from block b). The motion information obtained from neighboring blocks for each list is scaled to the first reference frame of a given list. Next, according to the same formula specified in HEVC as TMVP, the temporal motion vector prediction (TMVP) of sub-block A is derived. Extract the motion information of the collocated block at position D and perform corresponding scaling. Finally, after retrieving and scaling the motion information, average all available motion vectors (up to 3) for each reference list. The average motion vector is designated as the motion vector of the current sub-CU.

4.BIO

Bidirectional optical flow (BIO) is a sample direction motion refinement for bidirectional prediction based on block motion compensation. No signaling is used for motion refinement at the sample level.

/

,

/

分別為I ^(k)梯度的水平分量和垂直分量。假設光流是有效的，則運動向 量場(v _x ,v _y )由等式給出：

Let I ^{( k )} be the brightness value from the block motion compensation to the reference k (k=0, 1) , and

/

,

/

These are the horizontal and vertical components of the I ^{( k )} gradient. Assuming that the optical flow is valid, the motion vector field ( v _x , v _y ) is given by the equation:

/

,

/

。該多項式在t=0時的值是BIO預測：

Combine this optical flow equation with the Hermitian interpolation of each sample's trajectory to obtain a unique third-order polynomial, which matches the function value I ^{( k )} and its derivative at the same time at the end

/

,

/

. The value of the polynomial at t=0 is the BIO prediction:

Here, τ ₀ and τ ₁ represent the distance to the reference frame, as shown in Figure 8. The distances τ ₀ and τ _{1 are} calculated based on the POC of Ref0 and Ref1: τ ₀ =POC(current)-POC(Ref0), τ ₁ =POC(Ref1)-POC(current). If both predictions are from the same time direction (both from the past or both from the future), the signs are different (ie, τ ₀ , τ ₁ <0). In this case, BIO is only applied if the prediction is not from the same point in time (ie, τ ₀ ≠τ ₁ ). Both reference regions have non-zero motion (ie, MVx ₀ , MVy ₀ , MVx ₁ , MVy ₁ ≠ 0), and the block motion vector is proportional to the time distance (ie, MVx ₀ /MVx ₁ = MVy ₀ /MVy ₁ =-τ ₀ /τ ₁ ).

The motion vector field ( v _x , v _y ) is determined by minimizing the difference Δ between the values of point A and point B (the intersection point of the motion trajectory and the reference frame plane in Figs. 9A and 9B). For △, the model only uses the first linear term of the local Taylor expansion:

All the values in Equation 5 depend on the sample position ( i ′, j ′), which has been omitted so far. Assuming that the motion in the local surrounding area is consistent, then we minimize △ in a (2M+1)×(2M+1) square window Ω centered on the current prediction point (i, j), where M is equal to 2. :

For this optimization problem, JEM uses a simplified method, which is first minimized in the vertical direction, and then minimized in the horizontal direction. The results are as follows:

among them,

In order to avoid division by zero or small values, regularization is introduced in equations (7) and (8) Parameters r and m.

r = 500. 4 ^{d -8} (10)

m = 700. 4 ^{d -8} (11)

Here d is the bit depth of the video sample.

/

,

/

。在等式(9)中，以預測區塊邊界上當前預測點為中心的(2M+1)×(2M+1)的方形窗口Ω需要訪問區塊外的位置(如第9A圖所示)。在JEM中，塊外的值I ^(k),

/

.

/

設置為等於塊內最近的可用值。例如，這可以實現為填充，如第9B圖所示。 In order to make the memory access of BIO the same as conventional bidirectional predictive motion compensation, all predictions and gradient values I ^{( k ) are} calculated only in the current block position,

/

,

/

. In equation (9), the (2M+1)×(2M+1) square window Ω centered on the current prediction point on the boundary of the prediction block needs to be accessed outside the block (as shown in Figure 9A) . In JEM, the value I ^{( k )} outside the block,

/

.

/

Set equal to the nearest available value in the block. For example, this can be implemented as padding, as shown in Figure 9B.

Using BIO, the sports field of each sample can be refined. In order to reduce the computational complexity, BIO based on block design is used in JEM. The motion refinement is calculated based on 4×4 blocks. In block-based BIO, the _sn value in equation (9) of all samples in the 4×4 block is aggregated, and then the aggregated value of _sn is used for the derived BIO motion vector offset of the 4×4 block . More specifically, the following equation is used for block-based BIO derivation:

Among them, b _k represents the sample group of the k-th 4×4 block belonging to the prediction block. Replace s _{n in} equation (7) and equation (8) with ((s _n ,b _k )>>4) to derive the associated motion vector offset.

In some cases, BIO's MV regimen may be unreliable due to noise or irregular movement. Therefore, in BIO, the size of the MV group is fixed to a threshold thBIO. The threshold is determined based on whether the reference pictures of the current picture all come from one direction. If all reference pictures of the current picture are from one direction, the value of the threshold is set to 12×2 ^{14- d} , otherwise it is set to 12×2 ^{13- d} .

I/

x的情況下，首先使用BIOfilterS對信號進行垂直插值，該BIOfilterS對應於具有去縮放標度位移d-8的分數位置fracY，然後在水平方向上應用梯度濾波器BIOfilterG，該BIOfilterG對應於具有去縮放標度位移18-d的分數位置fracX。在垂直梯度

I/

y的情況下，使用BIOfilterG垂直地應用第一梯度濾波器，該BIOfilterG對應於具有去縮放標度位移d-8的分數位置fracY，然後，然後在水平方向上使用BIOfilterS執行信號替換，該BIOfilterS對應於具有去縮放標度位移18-d的分數位置fracX。用於梯度計算BIOfilterG和信號替換BIOfilterS的插值濾波器的長度更短(6-tap)，以保持合理的複雜度。表格示出了用於BIO中塊運動向量的不同分數位置的梯度計算的濾波器。表格示出了用於BIO中預測信號產生的插值濾波器。 The operation (2D separable FIR) consistent with the HEVC motion compensation process is used to calculate the gradient of BIO simultaneously through motion compensation interpolation. The input of this 2D separable FIR is the same reference frame as the motion compensation process, and the fractional position (fracX, fracY) according to the fractional part of the block motion vector. In horizontal gradient

I/

In the case of x , first use BIOfilterS to interpolate the signal vertically, which corresponds to the fractional position fracY with a de-scaling scale displacement d-8, and then apply the gradient filter BIOfilterG in the horizontal direction, which corresponds to the de-scaling The scale is displaced by 18-d fractional position fracX. In vertical gradient

I/

In the case of y , apply the first gradient filter vertically using BIOfilterG, which corresponds to the fractional position fracY with a de-scaling scale displacement d-8, and then perform signal replacement in the horizontal direction using BIOfilterS, which corresponds to At the fractional position fracX with a de-scaling scale shift of 18-d. The length of the interpolation filter used for gradient calculation BIOfilterG and signal replacement BIOfilterS is shorter (6-tap) to keep reasonable complexity. The table shows the filters used for gradient calculation of different fractional positions of block motion vectors in BIO. The table shows the interpolation filters used for the prediction signal generation in BIO.

In JEM, when two predictions come from different reference pictures, BIO is applied to all bidirectional prediction blocks. When LIC is enabled for CU, BIO is disabled.

In JEM, OBMC is applied to the block after normal MC processing. To reduce computational complexity, BIO is not applied during OBMC processing. This means that during OBMC processing, BIO is applied to the MC processing of a block only when its own MV is used, and when the MV of an adjacent block is used, BIO is not applied to the MC processing of the block.

2.5 FRUC

When the merge flag of the CU is true, the FRUC flag is signaled to the CU. When the FRUC flag is false, the merge index is signaled, and the regular Merge mode is used. When the FRUC flag is true, the additional FRUC mode flag is signaled to indicate which method (bilateral matching or template matching) will be used to derive the motion information of the block.

On the encoder side, the decision on whether to use the FRUC merge mode for the CU is based on the RD cost selection as done for the normal merge candidates. In other words, two matching modes (bilateral matching and template matching) of the CU are checked by using RD cost selection. The matching mode that results in the least cost is further compared with other CU modes. If the FRUC matching mode is the most effective mode, set the FRUC flag to true for the CU, and use the relevant matching mode.

The motion derivation process in FRUC merge mode has two steps. The CU-level motion search is performed first, and then the sub-CU-level motion refinement is performed. At the CU level, the initial motion vector is derived for the entire CU based on bilateral matching or template matching. First, a list of MV candidates is generated, and the candidate that leads to the smallest matching cost is selected as the starting point for further CU level refinement. Then, a partial search based on bilateral matching or template matching is performed around the starting point, and the MV that results in the smallest matching cost is taken as the MV of the entire CU. Subsequently, the motion information is further refined at the sub-CU level, with the derived CU motion vector as the starting point.

For example, the following derivation process is executed for W × H CU motion information derivation. In the first stage, the MV of the overall W × H CU is derived. In the second stage, the CU is further divided into M × M sub-CUs. For example, when calculating the value of M in (16), D is the predefined segmentation depth, which is set to 3 by default in JEM. Then export the MV of each sub-CU.

As shown in Figure 10, bilateral matching is used to derive the motion information of the current CU by finding the closest match between two blocks along the motion trajectory of the current CU in two different reference images. In continuous Under the assumption of the motion trajectory, the motion vectors MV0 and MV1 pointing to the two reference blocks should be proportional to the time distance between the current image and the two reference images—that is, TD0 and TD1. As a special case, when the current image is between two reference images in time and the time distance from the current image to the two reference images is the same, bilateral matching becomes a mirror-based bidirectional MV.

As shown in Figure 11, template matching is used to find the template in the current image (the top adjacent block and/or the left adjacent block of the current CU) and the block in the reference image (with the same size as the template) The closest match between the two is used to derive the motion information of the current CU. In addition to the aforementioned FRUC merge mode, template matching is also applicable to AMVP mode. In JEM, as in HEVC, AMVP has two candidates. Use template matching methods to derive new candidates. If the newly derived candidate matched by the template is different from the first existing AMVP candidate, it is inserted at the very beginning of the AMVP candidate list, and then the list size is set to 2 (which means removing the second existing AMVP candidate). When applied to AMVP mode, only CU level search is applied.

CU level MV candidate set

The CU-level MV candidate set consists of the following: i. If the current CU is in AMVP mode, it is the original AMVP candidate, ii. All merged candidates, iii. Interpolating several MVs in the MV domain (described later), iv. Top and The motion vector adjacent to the left.

When bilateral matching is used, each valid MV of the merge candidate is used as input to generate MV pairs assuming bilateral matching. For example, a valid MV of the merge candidate is (MVa, refa) in the reference list A. Then, find the reference image refb of the paired bilateral MV in the other reference list B, so that refa and refb are located on different sides of the current picture in time. If such refb in reference list B is not available, refb is determined to be a different reference from refa, and the time distance from refb to the current image is the minimum value in list B. After determining refb, based on the time distance between the current image and refa and refb Export MVb by scaling MVa.

Four MVs from the interpolation MV domain are also added to the CU level candidate list. More specifically, the interpolation MVs at the positions (0, 0), (W/2, 0), (0, H/2), and (W/2, H/2) of the current CU are added.

When FRUC is applied to AMVP mode, the original AMVP candidate is also added to the CU-level MV candidate set.

At the CU level, up to 15 MVs for AMVP CU and up to 13 MVs for merging CU are added to the candidate list.

Sub-CU-level MV candidate set The sub-CU-level MV candidate set consists of: i. MV determined from the CU level search, ii. adjacent MVs at the top, left, top left, and top right, and iii. from the reference image The scaled version of the parallel MV, iv. up to 4 ATMVP candidates, v. up to 4 STMVP candidates.

The zoomed MV from the reference image is derived as follows. Iterate through all reference images in the two lists. The MV at the juxtaposed position of the sub-CU in the reference image is scaled to the reference of the starting CU level MV.

The ATMVP and STMVP candidates are limited to the first four.

At the sub-CU level, up to 17 MVs are added to the candidate list.

Generation of interpolated MV domain

Before encoding the frame, an interpolated motion field is generated for the entire image based on a single-sided ME. Then, the motion domain can be used as a CU-level or sub-CU-level MV candidate later.

First, the motion domain of each reference image in the two reference lists is traversed at a 4×4 block level. For each 4×4 block, if the motion associated with the block passes through a 4×4 block in the current image (as shown in Figure 12) and the block has not been assigned any interpolation motion, the motion of the reference block is based on time Distance TD0 and TD1 (the same way as the MV scaling of TMVP in HEVC) scales to the current image, and assigns the scaled motion to the block in the current frame. If an unscaled MV is allocated to a 4×4 block, the motion of the block is marked as unusable in the interpolated motion domain.

Interpolation and matching costs

When the motion vector points to the fractional sample position, motion compensation interpolation is required. In order to reduce complexity, both bilateral matching and template matching use bilinear interpolation instead of conventional 8-tap HEVC interpolation.

The calculation of the matching cost is a bit different in different steps. When selecting candidates from the candidate set at the CU level, the matching cost is the sum of absolute differences (SAD) of bilateral matching or template matching. After the starting MV is determined, the matching cost C of the bilateral match searched at the sub-CU level is calculated as follows:

Where w is a weighting factor and is set to 4 based on experience, MV and MV ^s indicate the current MV and the starting MV respectively. SAD is still used as the matching cost for template matching in sub-CU level searches.

In FRUC mode, MV is derived by using only luminance samples. The derived motion will be used for the luma and chroma of MC inter prediction. After the MV is determined, the final MC is performed using an 8-tap interpolation filter for luma and a 4-tap interpolation filter for chroma.

MV refinement

MV refinement is a pattern-based MV search based on bilateral matching cost or template matching cost. In JEM, two search modes are supported--unrestricted center-biased diamond search (UCBDS) and adaptive cross search (UCBDS) for CU-level and sub-CU-level MV refinement. search). For CU-level and sub-CU-level MV refinement, the MV is directly searched with a quarter-luminance sample MV accuracy, and then the MV is refined with one-eighth luminance sample MV. The search range of MV refinement for the CU step and the sub-CU step is set equal to 8 luminance samples.

Selection of prediction direction in template matching FRUC merge mode

In the bilateral matching Merge mode, bi-directional prediction is always applied because the motion information of the CU is derived based on the closest match between two blocks along the motion trajectory of the current CU in two different reference images. There is no such limitation in template matching Merge mode. In the template matching Merge mode, the encoder can choose among unidirectional prediction from list 0, unidirectional prediction from list 1, or bidirectional prediction for the CU. The selection is based on the template matching cost, as follows: if costBi <= factor * min( cost 0, cost1 )

Use bidirectional prediction; otherwise, if cost 0<= cost1

Use the one-way prediction from list 0; otherwise, use the one-way prediction from list 1; where cost0 is the SAD matched by the list 0 template, cost1 is the SAD matched by the list 1 template, and costBi is the SAD matched by the two-way prediction template. The value of factor is equal to 1.25, which means that the selection process is biased towards bidirectional prediction.

Inter-frame prediction direction selection is only applied to the CU-level template matching process.

Interleaved prediction example

Through interleaving prediction, the block is subdivided into sub-blocks in more than one subdivision pattern. The subdivision style is defined as a way of subdividing a block into sub-blocks, including the size and position of the sub-block. For each subdivision pattern, the corresponding prediction block can be generated by deriving the motion information of each sub-block based on the subdivision pattern. Therefore, even for one prediction direction, multiple prediction blocks can be generated from multiple subdivision patterns. Alternatively, for each prediction direction, only the subdivision pattern may be applied.

Suppose the X refinement patterns, the X and the prediction block of the current block exists, the X prediction block is denoted as _{P 0, P 1, ...,} P X -1, which is based on the use of X subblock subdivided by a pattern of The forecast is generated. The final prediction of the current block, denoted as P , can be generated as

w _i (x,y)=(1≪N)，其中N是非負值。第13圖示出了使用兩個細分樣式的交織預測的示例。 Where (x, y) are the coordinates of pixels in the block, and w _i (x, y) is the weighting value of P _i. Without losing generalization, assuming

w _i ( x,y )= ( 1≪ N) , where N is a non-negative value. Figure 13 shows an example of interleaving prediction using two subdivision patterns.

3. Exemplary problems solved by the described embodiments

There are two potential shortcomings in the MV derivation process of affine merge, as shown in Figure 5.

First, the left vertex of the CU and the size of the CU must be stored by each 4×4 block belonging to the CU. This information does not need to be stored in HEVC.

Second, the decoder must access the MV of a 4×4 block that is not adjacent to the current CU. In HEVC, the decoder only needs to access the MV of the 4×4 block adjacent to the current CU.

4. Example of embodiment

We propose several methods to further improve sub-block-based prediction, including interleaving prediction and affine combined MV derivation process.

The following detailed invention should be considered as an example to explain the overall concept. These inventions should not be understood in a narrow manner. In addition, these inventions can be combined in any manner. Combinations between the present invention and other inventions are also applicable.

Use of interleaved prediction

1. In one embodiment, whether and how to apply interleaving prediction may depend on the color components.

a. For example, interleaving prediction is only applied to the luminance component, not to the chrominance component; b. For example, the subdivision pattern is different for different color components; c. For example, the weight value is different for different color components.

2. In one embodiment, whether and how to apply interlaced prediction may depend on whether the inter prediction direction and/or reference pictures are the same.

a. For example, interleaving prediction can only be used for unidirectional prediction, but not for bidirectional prediction.

b. For example, interleaving prediction can only be used for bidirectional prediction, but the two reference pictures of the two reference picture lists are the same.

c. In one example, interleaving prediction is disabled for the low delay P(LDP) case.

d. In one example, when predicting the current block from the current picture, interleaving prediction is also enabled.

Partial interleaving prediction

1. In one embodiment, interleaving prediction can be applied to parts of the entire block.

a. The second subdivision style can only cover part of the entire block. The samples outside this part are not affected by the interleaving prediction.

b. This part can exclude samples located at the block boundary, for example, the top/last n rows or the top/last m columns.

c. This part can exclude samples located at sub-blocks having a size different from the size of most of the sub-blocks in the second subdivision pattern within the block.

d. Figures 14A and 14B show some examples of partially interleaved affine prediction. The first subdivision style is the same as that in JEM, that is, the left vertex of the sub-block is at ( i × w , j × h ), and then from equation (1), (x, y)=( i × w + w /2, j × h + h /2) calculate the MV of this sub-block (the ( i, j )-th sub-block). For the two subdivision styles, the size of the sub-block is w × h . For example, w = h = 4 or w = h = 8.

i. In Figure 14A, the top left of the sub-block of the second subdivision pattern (the ( i, j )-th sub-block) is ( i × w + w /2, j × h ), and from equation (1) Calculate the MV of this sub-block with (x,y)=( i × w + w , j × h + h /2).

ii. In Figure 14B, the top left of the sub-block of the second subdivision pattern is ( i × w , j × h + h /2), and from equation (1), (x, y) = ( i × w + w /2, j × h + h ) calculate the MV of this sub-block.

iii. In Figure 14B, the top left of the sub-block of the second subdivision pattern is ( i × w + w /2, j × h + h /2), and from equation (1) to (x, y) =( i × w + w , j × h + h ) Calculate the MV of this sub-block.

Figures 14A-14C show examples of partial interleaving prediction. The dotted line indicates the first subdivision style; the solid line indicates the second subdivision style; the thick line indicates the area where interlaced prediction is to be applied. Outside this area, no interleaving prediction is applied.

Weight value in interleaving prediction

2. In one embodiment, there are two possible weight values Wa and Wb, satisfying Wa+Wb=2 ^N. Exemplary weight values {Wa, Wb} are {3, 1}, {7, 1}, {5, 3}, {13, 3}, etc.

a. If the weight value w1 associated with the predicted sample P1 generated by the first subdivision style and the weight value w2 associated with the predicted sample P2 generated by the second subdivision style are the same (both equal to Wa or Wb), then this sample The final prediction P is calculated as P=(P1+P2)>>1 or P=(P1+P2+1)>>1.

b. If the weight value w1 associated with the predicted sample P1 generated by the first subdivision style is different from the predicted sample P2 generated by the second subdivision style ({w1, w2}={Wa, Wb} or {w1, w2 }={Wb, Wa}), the final prediction P of this sample is calculated as P=(w1×P1+w2×P2+offset)>>N, where the offset can be 1<<(N-1) or 0.

c. It can be similarly extended to the case when there are more than 2 subdivision patterns.

3. In one embodiment, if the specimen A MV deriving a position closer than the sub-sample block B, the weight values of the samples A sub-block is greater than a weight value of the sub-sample B blocks. Exemplary weight values of 4×4 sub-blocks, 4×2 sub-blocks, 2×4 sub-blocks, or 2×2 sub-blocks are shown in Fig. 16.

Figure 15 shows an example of weight values in sub-blocks. Exemplary weight values {Wa, Wb} are {3, 1}, {7, 1}, {5, 3}, {13, 3}, etc.

Example of interleaving prediction

Figure 16 shows an example of interleaving prediction with two subdivision patterns according to the disclosed technology. The current block 1300 can be subdivided into multiple styles. For example, as shown in Figure 16, the current block is subdivided into pattern 0 (1301) and pattern 1 (1302). Two prediction blocks P ₀ (1303) and P ₁ (1304) are generated. By calculating the weighted sum of P ₀ (1303) and P ₁ (1304), the final prediction block P (1305) of the current block 1300 can be generated.

Generally speaking, given X subdivision patterns, X prediction blocks (represented as P ₀ , P ₁ ,..., P _{X -1} ) of the current block can be generated by sub-block-based prediction in X subdivision patterns . The final prediction of the current block (denoted as P) can be produced as:

Here, (x, y) are the coordinates of pixels in the block, and w _i (x, y) is the weight P _i is a weight coefficient. By way of example rather than restriction, the weight can be expressed as:

N is a non-negative value. Optionally, the displacement operation in equation (8) can also be expressed as:

The weight sum is a power of 2. By performing a shift operation instead of floating point division, the weighted sum P can be calculated more efficiently.

The subdivision style can have different sub-block shapes, sizes or positions. In some embodiments, the subdivision pattern may include irregular sub-block sizes. Figures 17A-17G show examples of several subdivision patterns of 16×16 blocks. In Figure 17A, the block is subdivided into 4×4 sub-blocks according to the disclosed technology. This style is also used in JEM. Figure 17B shows an example of a subdivision pattern in which a block is subdivided into 8×8 sub-blocks according to the disclosed technology. Figure 17C shows an example of a subdivision pattern in which a block is subdivided into 8×4 sub-blocks according to the disclosed technology. FIG. 17D shows an example of a subdivision pattern in which a block is subdivided into 4×8 sub-blocks according to the disclosed technology. In Figure 17E, a part of the block is subdivided into 4×4 sub-blocks according to the disclosed technology. The pixels on the block boundary are subdivided into smaller sub-blocks, the size of which is 2×4, 4×2, or 2×2. Some sub-blocks can be combined to form larger sub-blocks. Figure 17F shows an example of adjacent sub-blocks (such as 4×4 sub-blocks and 2×4 sub-blocks). These sub-blocks are combined to form a larger sub-block with a size of 6×4, 4×6, or 6×6. In Figure 14G, part of the block is subdivided into 8×8 sub-blocks. The pixels at the block boundary are subdivided into smaller sub-blocks such as 8×4, 4×8, or 4×4.

In sub-block-based prediction, the shape and size of the sub-block may be determined based on the shape and/or size of the coding block and/or coding block information. For example, in some embodiments, when the size of the current block is M×N When, the size of the sub-block is 4×N (or 8×N, etc.), that is, the sub-block has the same height as the current block. In some embodiments, when the size of the current block is M×N, the size of the sub-block is M×4 (or M×8, etc.), that is, the sub-block has the same width as the current block. In some embodiments, when the size of the current block is M×N (where M>N), the size of the sub-block is A×B, where A>B (for example, 8×4). Alternatively, the size of the sub-block is B×A (for example, 4×8).

In some embodiments, the size of the current block is M×N. When M×N<=T (or min(M,N)<=T, or max(M,N)<=T, etc.), the size of the sub-block is A×B; when M×N>T(or When min(M,N)>T, or max(M,N)>T, etc.), the size of the sub-block is C×D, where A<=C, B<=D. For example, if M×N<=256, the size of the sub-block may be 4×4. In some implementations, the size of the sub-block is 8×8.

In some embodiments, whether to apply interlaced prediction may be determined based on the inter prediction direction. For example, in some embodiments, interleaving prediction may be applied to bidirectional prediction, but not to unidirectional prediction. As another example, when multiple-hypothesis is applied, when there is more than one reference block, interleaving prediction can be applied to one prediction direction.

In some embodiments, how to apply interleaving prediction may be determined based on the direction of inter prediction. In some embodiments, a block that uses sub-block-based prediction but bidirectionally predicted is subdivided into sub-blocks with two different subdivision patterns for two different reference lists. For example, when predicting from the reference list 0 (L0), the bi-predicted block is subdivided into 4×8 sub-blocks, as shown in Figure 17D. When predicted from reference list 1 (L1), the same block is subdivided into 8×4 sub-blocks, as shown in Figure 17C. The final prediction P is calculated as

Here, P0 and P1 are predictions from L0 and L1, respectively. w0 and w1 are the weight values of L0 and L1, respectively. As shown in equation (16), the weight value can be determined as: w ⁰ ( x, y ) + w ¹ ( x, y )=1<<N (where N is a non-negative integer value). Because fewer sub-blocks are used for prediction in each direction (for example, 4×8 sub-blocks compared to 8×8 sub-blocks), less calculation is required compared to existing methods based on sub-blocks Broadband. By using larger sub-blocks, the prediction result is also less susceptible to noise interference.

In some embodiments, a block that is unidirectionally predicted with sub-block-based prediction is subdivided into sub-blocks using two or more different subdivision patterns for the same reference list. For example, the prediction of the list L (L=0 or 1) ^PL is calculated as

是用第i細分樣式 產生的預測，並且

是

的權重值。例如，當XL為2時，兩個細分樣 式被應用於列表L。在第一細分樣式中，塊被細分為4×8子塊，如第17D圖所示。在第二細分樣式中，塊被細分為8×4子塊，如第17D圖所示。 Here, XL is the number of subdivision patterns used for the list L.

Is the prediction generated with the i-th subdivision style, and

Yes

The weight value of. For example, when XL is 2, two subdivision styles are applied to list L. In the first subdivision pattern, the block is subdivided into 4×8 sub-blocks, as shown in Figure 17D. In the second subdivision pattern, the block is subdivided into 8×4 sub-blocks, as shown in Figure 17D.

In some embodiments, a bi-directionally predicted block with sub-block-based prediction is considered to be a combination of two uni-directionally predicted blocks from L0 and L1, respectively. The prediction from each list can be derived as described in the example above. The final prediction P can be calculated as

Here, the parameters a and b are two additional weights, which are applied to two intra prediction blocks. In this specific example, both a and b can be set to 1. Similar to the above example, because fewer sub-blocks are used for prediction in each direction (for example, 4×8 sub-blocks compared to 8×8 sub-blocks), broadband usage is better than existing sub-block-based The method is at the same level as the existing method based on sub-blocks. At the same time, the prediction result can be improved by using larger sub-blocks.

In some embodiments, a single non-uniform pattern may be used in each unidirectionally predicted block. For example, for each list L (e.g., L0 or L1), the block is subdivided into different styles (e.g., as shown in Figure 17E or Figure 17F). Using a smaller number of sub-blocks reduces the need for broadband. The non-uniformity of the sub-blocks also increases the robustness of the prediction results.

In some embodiments, for multi-hypothesis coded blocks, there may be (Or reference picture list) more than one prediction block generated by different subdivision patterns. Multiple prediction blocks can Used to generate the final prediction with additional weights applied. For example, the additional weight can be set to 1/M, where M is the total number of generated prediction blocks.

In some embodiments, the encoder can determine whether and how to apply interlaced prediction. Then the encoder can correspond to the determined information in sequence level, picture level, view level, slice level, coding tree unit (CTU) (also known as the largest coding unit (LCU)) level, CU level, PU level, The tree unit (TU) level, or area level (which may contain multiple CU/PU/TU/LCU) is sent to the decoder. Information can be signaled in the first block of sequence parameter set (SPS), view parameter set (VPS), picture parameter set (PPS), slice header (SH), CTU/LCU, CU, PU, TU, or region Notice.

In some implementations, interleaving prediction is applied to existing sub-block methods, such as affine prediction, ATMVP, STMVP, FRUC, or BIO. In this case, no additional signaling cost is required. In some implementations, new sub-block merging candidates generated by interleaving prediction may be inserted into the merge list, for example, interleaving prediction+ATMVP, interleaving prediction+STMVP, interleaving prediction+FRUC, etc.

In some embodiments, the subdivision pattern to be used by the current block may be derived based on information from spatial and/or temporal neighboring blocks. For example, instead of relying on the encoder to signal related information, both the encoder and the decoder can adopt a set of predetermined rules to be based on temporal adjacency (for example, the previously used subdivision pattern of the same block) or spatial phase. Neighborhood (for example, the subdivision style used by neighboring blocks) obtains the subdivision style.

In some embodiments, the weight value w may be fixed. For example, all subdivision styles can be weighted equally: w _i ( x,y )=1. In some embodiments, the weight value may be determined based on the location of the block and the subdivision style used. For example, w _i ( x , y) can be different for different (x, y). In some embodiments, the weight value may further depend on sub-block prediction based on coding technology (for example, affine or ATMVP) and/or other coding information (for example, skip or non-skip mode, and/or MV information) .

In some embodiments, the encoder can determine the weight value, and use the value in sequence level, picture level, slice level, CTU/LCU level, CU level, PU level, or region level (which may include multiple CU/PU /TU/LCU) is sent to the decoder. The weight value can be signaled in the sequence parameter set (SPS), picture parameter set (PPS), slice header (SH), CTU/LCU, CU, PU, or the first block of the region. In some embodiments, the weight value may be derived from the weight value of spatial and/or temporal neighboring blocks.

Note that the interleaving prediction technique disclosed herein can be applied to one, some or all of the coding techniques based on sub-block prediction. For example, the interleaving prediction technique can be applied to affine prediction, while other coding techniques based on sub-block prediction (for example, ATMVP, STMVP, FRUC or BIO) do not use interleaving prediction. As another example, all of affine, ATMVP, and STMVP apply the interleaving prediction technique disclosed herein.

Figure 18 is a block diagram of an exemplary video processing device 1800. The device 1800 can be used to implement one or more of the methods described herein. The device 1800 may be implemented as a smart phone, a tablet, a computer, an Internet of Things (IoT) receiver, and so on. The device 1800 may include one or more processors 1802, one or more memories 1804, and video processing hardware 1806. The processor(s) 1802 may be configured to implement one or more methods described in this document. The memory (multiple memories) 1804 can be used to store data and codes, which are used to implement the methods and techniques described herein. The video processing circuit 1806 can be used to implement some of the techniques described in this document with hardware circuits.

Figure 19 shows a flowchart of an exemplary method 1900 of video processing. The method 1900 includes, in step 1902, determining whether the interlaced prediction mode is suitable for the conversion between the current video block and the bitstream representation of the current video block based on the component type of the current video block. The method 1900 further includes, in step 1904, in response to determining that the interleaving prediction mode is applicable to the current video block, performing conversion by applying the interleaving prediction mode, wherein applying the interleaving prediction includes subdividing a part of the current video block into at least one subdivision pattern using more than one subdivision pattern. Block, and generate a predictor of the current video block as a weighted average of the determined predictor for each of more than one subdivision patterns.

Figure 20 shows a flowchart of an exemplary method 2000 of video processing. The method 2000 includes, in step 2002, determining whether the interlaced prediction mode is suitable for the conversion between the current video block and the bitstream representation of the current video block based on the prediction direction of the current video block. The method 2000 further includes, in step 2004, in response to determining that the interleaving prediction mode is suitable for the current video block, performing conversion by applying the interleaving prediction mode, and wherein applying the interleaving prediction includes subdividing part of the current video block into at least one subdivision pattern using more than one subdivision pattern. Sub-blocks, and generate the predictor of the current video block as a weighted average of the determined predictors for each of more than one subdivision patterns.

Figure 21 shows a flowchart of an exemplary method 2100 of video processing. The method 2100 includes, in step 2102, determining whether the interlaced prediction mode is suitable for the conversion between the current video block in the current picture and the bitstream representation of the current video block based on the low delay mode of the current picture. The method 2100 further includes, in step 2104, in response to determining that the interleaving prediction mode is applicable to the current video block, performing conversion by applying the interleaving prediction mode, and wherein applying the interleaving prediction includes subdividing part of the current video block into at least one subdivision pattern using more than one subdivision pattern. One sub-block, and generate the predictor of the current video block as a weighted average of the determined predictors for each of more than one subdivision patterns.

Figure 22 shows a flowchart of an exemplary method 2200 of video processing. The method 2200 includes, in step 2202, determining whether the interlaced prediction mode is suitable for conversion between the current video block and the bitstream representation of the current video block based on using the current picture containing the current video block as a reference. The method 2200 further includes, in step 2204, in response to determining that the interleaving prediction mode is suitable for the current video block, converting by applying the interleaving prediction mode, and wherein applying the interleaving prediction includes subdividing part of the current video block into at least one subdivision pattern using more than one subdivision pattern. Sub-block pattern, and generate the predictor of the current video block as a weighted average of the determined predictor for each of more than one sub-division pattern.

Another exemplary method of video processing is provided. The method includes selectively performing (2300) coding based on the interleaving prediction of one or more of the luminance component, the first chrominance component, and the second chrominance component of the video frame from the video frame based on the video condition. Perform inter-frame interleaving prediction, including, through Determine the prediction block from the current block of the components of the video as follows: select (2302) the set of pixels of the components of the video frame to form a block; divide the block (2304) into the first set of sub-blocks according to the first pattern; based on The first set of sub-blocks is generated (2306) the first intermediate prediction block; according to the second pattern, the block is divided (2308) into a second set of sub-blocks, wherein at least one sub-block in the second set is not in the first set Medium; Based on the second set of sub-blocks, generate (2310) a second intermediate prediction block; and, based on the first intermediate prediction block and the second intermediate prediction block, determine (2312) the prediction block.

The following uses a clause-based format to describe additional features and embodiments of the above methods/techniques.

1. A method of video processing, comprising: determining whether an interleaved prediction mode is suitable for conversion between the current video block and the bitstream representation of the current video block based on the component type of the current video block; and in response to determining that the interleaved prediction mode is suitable The current video block is converted by applying the interlaced prediction mode

Wherein applying interlaced prediction includes subdividing part of the current video block into at least one sub-block using more than one subdivision pattern, and generating a predictor of the current video block as a weighted average of the predictor determined for each of the more than one subdivision patterns .

2. The method according to clause 1, wherein the interleaving prediction is applied in response to the component type being equal to the luminance component.

3. The method of clause 1, wherein more than one subdivision pattern for the first color component of the video is different from another more than one subdivision pattern for the second color component of the video.

4. The method according to clause 1, wherein a weighted average is used, and the value of the weight depends on the component type.

5. A method of video processing, including: Based on the prediction direction of the current video block, determine whether the interleaving prediction mode is suitable for the current video block Conversion between the frequency block and the bitstream representation of the current video block; and In response to determining that the interleaving prediction mode is suitable for the current video block, conversion is performed by applying the interleaving prediction mode, and Wherein applying interlaced prediction includes subdividing part of the current video block into at least one sub-block using more than one subdivision pattern, and generating a predictor of the current video block as a weighted average of the predictor determined for each of the more than one subdivision patterns .

6. The method according to clause 5, wherein interlaced prediction is applied in response to the prediction direction being equal to one-way prediction.

7. The method of clause 5, wherein interlaced prediction is applied in response to the prediction direction being equal to two-way.

8. A method of video processing, including: Based on the low-latency mode of the current picture, determine whether the interlaced prediction mode is suitable for the conversion between the current video block in the current picture and the bitstream representation of the current video block; and In response to determining that the interleaving prediction mode is suitable for the current video block, conversion is performed by applying the interleaving prediction mode, and Wherein applying interlaced prediction includes subdividing part of the current video block into at least one sub-block using more than one subdivision pattern, and generating a predictor of the current video block as a weighted average of the predictor determined for each of the more than one subdivision patterns .

9. The method according to clause 8, wherein the interleaving prediction is disabled for the low delay mode of the current picture.

10. A method of video processing, including: Based on using the current picture containing the current video block as a reference, determine whether the interlaced prediction mode is suitable for the conversion between the current video block and the bitstream representation of the current video block; and In response to determining that the interleaving prediction mode is suitable for the current video block, by applying the interleaving prediction mode Method, and wherein applying interlaced prediction includes subdividing part of the current video block into at least one sub-block pattern using more than one subdivision pattern, and generating a weighted average of predictors determined for each of the more than one subdivision pattern The predictor of the current video block.

11. The method of clause 10, wherein when the current video block is predicted from the current picture, interlaced prediction is enabled.

12. The method according to clause 1, 5, 8 or 10, wherein part of the current video block includes less than all current video blocks.

13. The method according to any one of clauses 1-12, wherein the interlaced prediction mode is applied to part of the current video block.

14. The method according to clause 13, wherein at least one of the subdivision patterns only covers part of the current video block.

15. The method of clause 14, wherein samples located at the boundary of the current video block are partially excluded.

16. The method of clause 14, wherein samples located at sub-blocks having a size different from the size of most sub-blocks within the current video block are partially excluded.

17. The method according to clause 1, 5, 8, 10 or 12, wherein subdividing part of the current video block further comprises: dividing the current video block into a first set of sub-blocks according to the first pattern of the subdivision style; The first set of blocks generates a first intermediate prediction block; the current video block is divided into a second set of sub-blocks according to the second pattern of the subdivision style, wherein at least one sub-block in the second set is not in the first set; and The second set of sub-blocks produces a second intermediate prediction block.

18. The method according to clause 17, wherein the prediction sample generated by the first pattern is The associated weight value w1 is the same as the weight value w2 associated with the prediction sample generated by the second style, and the final prediction P is calculated as P=(P1+P2)>>1 or P=(P1+P2+1)> >1.

19. The method according to clause 17, wherein the weight value w1 associated with the prediction sample generated by the first style is different from the weight value w2 associated with the prediction sample generated by the second style, and the final prediction P is calculated It is P=(w1×P1+w2×P2+offset)>>N, where the offset is 1<<(N-1) or 0.

20. The method according to clause 17, wherein the first weight Wa of the first intermediate prediction block and the second weight Wb of the second intermediate prediction block satisfy the condition Wa+Wb=2 ^N , where N is an integer.

21. The method according to clause 17, wherein if the first sample is closer to the position where the motion vector of the sub-block is derived than the second sample, the weight value of the first sample in the sub-block is greater than the second sample in the sub-block The weight value of.

22. A method of video processing, comprising: selectively based on video conditions, performing interleaving prediction of one or more of the luminance component, the first chrominance component, and the second chrominance component of the video frame. Encoding, where performing interleaving prediction includes determining a prediction block from the following current block, which is the component of the video: selecting a set of pixels of the components of the video frame to form a block; dividing the block into a first set of sub-blocks according to a first pattern; based on The first set of sub-blocks generates a first intermediate prediction block; the block is divided into a second set of sub-blocks according to the second pattern, wherein at least one sub-block in the second set is not in the first set; a second sub-block-based The set generates a second intermediate prediction block; and the prediction block is determined based on the first intermediate prediction block and the second intermediate prediction block.

23. The method of clause 22, wherein interlaced prediction is used only for the luminance component to form the prediction block.

24. The method according to clause 22, wherein a different first or second style is used To split the different components of the video.

25. The method according to any one of clauses 22 to 24, wherein the video conditions include the direction of prediction, and wherein interleaving prediction is performed only for one of unidirectional prediction or bidirectional prediction, and not for unidirectional prediction and bidirectional prediction. The other of the predictions performs interlaced prediction.

26. The method of clause 22, wherein the video conditions include the use of a low-delay P coding mode, and wherein in the case of using the low-delay P-mode, the method includes suppressing interleaving prediction.

27. The method of clause 22, wherein the video conditions include using a current picture containing the current block as a reference for prediction.

28. The method according to any one of clauses 1-27, wherein interleaving-based predictive coding includes using only a first set of sub-blocks from part of the current block and a second set of sub-blocks.

29. The method of clause 28, wherein a smaller part of the current block excludes samples in the boundary area of the current block.

30. The method of clause 28, wherein using a part of the current block for interleaving-based predictive coding includes using a part of the current block for affine prediction.

31. The method of clause 22, wherein determining the prediction block comprises using a weighted average of the first intermediate prediction block and the second intermediate prediction block to determine the prediction block.

32. The method of additional clause 31, wherein the first weight Wa of the first intermediate prediction block and the second weight Wb of the second intermediate prediction block satisfy the condition Wa+Wb=2 ^N , where N is an integer.

33. The method according to clause 32, wherein Wa=3 and Wb=1.

34. The method according to clause 22, wherein when the first sample is closer to the position of deriving the motion vector of the sub-block than the second sample, the weight value of the first sample in the sub-block is greater than that of the first sample in the sub-block. The weight value of the second sample.

35. A device comprising a processor and a non-transitory memory with instructions thereon, wherein when the instructions are executed by the processor, the processor is caused to implement the method in one or more of clauses 1 to 34.

36. A computer program product stored on a non-transitory computer readable medium, the computer program product containing program code, and the program code is used to perform one or more of the methods described in clauses 1 to 34.

From the above, it can be understood that the specific embodiments of the technology of the present disclosure described herein are for illustrative purposes, but various modifications can be made without departing from the scope of the present invention. Accordingly, the technology of the present disclosure is not restricted except for the appended claims.

The disclosure and other embodiments, modules, and functional operations described in this document can be implemented by digital electronic circuits, or by computer software, firmware, or hardware, including the structures disclosed in this document and their structural equivalents, or by using them A combination of one or more of them. The disclosed and other embodiments can be implemented as one or more computer program products, that is, one or more computer program instruction modules encoded on a computer-readable medium, used to execute or control the operation of the data processing device. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a combination of substances that affect a machine-readable propagated signal, or a combination of one or more of them. The term "data processing device" covers all devices, equipment, and machines used to process data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, for example, constituting processor firmware, protocol stack, database management system, operating system, or one or more of them The combined code. Propagated signals are artificially generated signals, such as electrical, optical or electromagnetic signals generated by machines, which are generated to encode information for transmission to a suitable receiver device.

Computer programs (also called programs, software, software applications, scripts or codes) can be written in any form of programming language, including compiled or interpreted languages, and computer programs can be deployed in any form, including as stand-alone programs or as suitable for Modules, components, subroutines, or other units used in a computing environment. Computer programs do not necessarily correspond to documents in the file system. Programs can be stored in a part of a document that holds other programs or data (for example, one or more scripts stored in a markup language document), in a single document dedicated to the program in question, or in multiple coordination file Medium (for example, a document that stores one or more modules, subroutines, or code parts). Computer programs can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

The processes and logic flows described in this document can be executed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The process and logic flow can also be executed by a dedicated logic circuit, and the device can also be implemented as a dedicated logic circuit, such as FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit).

For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, the processor will receive commands and data from read-only memory or random access memory or both. The basic components of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include or be operatively coupled to one or more mass storage devices for storing data, such as magnetic disks or optical discs, to receive data from the one or more mass storage devices, or to transfer data To the one or more mass storage devices, or both receive and transmit data. However, the computer does not need to have such equipment. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM and flash memory devices; Disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory can be supplemented by or incorporated into a dedicated logic circuit.

Although this patent document contains many details, these details should not be construed as limitations on the scope of any invention or claimable, but as descriptions of features specific to specific embodiments of specific inventions. In this patent document, certain features described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. In addition, despite the above The features can be described as acting in certain combinations and even initially claimed as such, but in some cases, one or more features from the combination can be removed from the claimed combination, and the claimed combination can point to Sub-combination or variant of sub-combination.

Similarly, although operations are depicted in a specific order in the drawings, this should not be understood as requiring that such operations be performed in the specific order shown or in order, or that all the operations shown are performed to achieve the desired result . In addition, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

Only a few implementations and examples are described, and other implementations, enhancements and modifications can be made based on the content described and shown in this patent document.

The foregoing descriptions are only preferred embodiments of the present invention, and all equivalent changes and modifications made in accordance with the scope of the patent application of the present invention shall fall within the scope of the present invention.

2300: method

2302 to 2312: steps

Claims (36)

A video processing method includes: determining whether an interlaced prediction mode is suitable for conversion between the current video block and a bitstream representation of the current video block based on the component type of the current video block; and in response to determining the interlaced prediction mode The conversion is applied to the current video block by applying the interlaced prediction mode; wherein applying the interlaced prediction includes subdividing part of the current video block into at least one sub-block using more than one subdivision pattern, and generating as the A weighted average predictor of the current video block of each determined predictor in a subdivision pattern. The method of claim 1, wherein the interleaving prediction is applied in response to the component type being equal to the luminance component. The method of claim 1, wherein the more than one subdivision pattern for the first color component of the video is different from the other more than one subdivision pattern for the second color component of the video. The method according to claim 1, wherein the weighted average uses a weight, and the value of the weight depends on the component type. A video processing method includes: determining whether an interleaving prediction mode is suitable for conversion between the current video block and a bitstream representation of the current video block based on the prediction direction of the current video block; and in response to determining the interleaving prediction mode Apply to the current video block, perform the conversion by applying the interleaving prediction mode, and wherein applying the interleaving prediction includes subdividing part of the current video block into at least one sub-block using more than one subdivision pattern, and generating as the A weighted average predictor of the current video block of each determined predictor in a subdivision pattern. The method according to claim 5, wherein in response to the prediction direction being equal to one-way prediction, apply The interweaving prediction. The method of claim 5, wherein the interleaving prediction is applied in response to the prediction direction being equal to two-way. A video processing method includes: determining whether an interlaced prediction mode is suitable for conversion between a current video block in the current picture and a bitstream representation of the current video block based on a low delay mode of the current picture; and in response to the determination The interlaced prediction mode is applied to the current video block, and the conversion is performed by applying the interlaced prediction mode, and wherein applying the interlaced prediction includes subdividing part of the current video block into at least one sub-block using more than one subdivision pattern, and generating as The predictor of the current video block that is a weighted average of the determined predictors for each of the more than one subdivision patterns. The method according to claim 8, wherein the interleaving prediction is disabled for the low delay mode of the current picture. A video processing method includes: determining whether an interlaced prediction mode is suitable for conversion between the current video block and the bitstream representation of the current video block based on using a current picture containing the current video block as a reference; and in response to the determination The interlaced prediction mode is applied to the current video block, the conversion is performed by applying the interlaced prediction mode, and wherein applying the interlaced prediction includes subdividing part of the current video block into at least one sub-block pattern using more than one sub-block pattern, and generating The predictor of the current video block as a weighted average of the determined predictors for each of the more than one subdivision patterns. The method of claim 10, wherein the interlaced prediction is enabled when the current video block is predicted from the current picture. The method according to claim 1, 5, 8 or 10, wherein part of the current video block includes less than All of the current video block. The method according to any one of claim 1, 5, 8, or 10, wherein the interlaced prediction mode is applied to part of the current video block. The method according to claim 13, wherein at least one of the subdivision patterns only covers a part of the current video block. The method according to claim 14, wherein the part excludes samples located at the boundary of the current video block. The method according to claim 14, wherein the part excludes samples located at sub-blocks having a size different from most of the sub-block sizes within the current video block. The method according to any one of claim 1, 5, 8, or 10, wherein the subdividing part of the current video block further comprises: dividing the current video block into sub-blocks according to the first pattern of the subdivision style Set; generate a first intermediate prediction block based on the first set of sub-blocks; divide the current video block into a second set of sub-blocks according to the second pattern of the subdivision pattern, wherein at least one sub-block in the second set Not in the first set; and generating a second intermediate prediction block based on the second set of sub-blocks. The method according to claim 17, wherein the weight value w1 associated with the prediction sample generated by the first style is the same as the weight value w2 associated with the prediction sample generated by the second style, and the final prediction P The calculation is P=(P1+P2)>>1 or P=(P1+P2+1)>>1. The method according to claim 17, wherein the weight value w1 associated with the prediction sample generated by the first style is different from the weight value w2 associated with the prediction sample generated by the second style, and the final prediction P Calculate as P=(w1×P1+w2×P2+offset)>>N, where the offset is 1<<(N-1) or 0. The method according to claim 17, wherein the first weight Wa of the first intermediate prediction block and the second weight Wb of the second intermediate prediction block satisfy the condition Wa+Wb=2 ^N , where N is an integer. The method according to claim 17, wherein if the first sample is closer to the position of deriving the motion vector of the sub-block than the second sample, the weight value of the first sample in the sub-block is greater than the first sample in the sub-block The weight value of the second sample. A video processing method, comprising: selectively encoding based on video conditions, encoding one or more of the luminance component, the first chrominance component, and the second chrominance component of the video frame based on the interleaving prediction , Wherein performing the interleaving prediction includes determining a prediction block by the following current block which is a component of the video: selecting a set of pixels of the component of the video frame to form a block; dividing the block into the first sub-block according to the first pattern Set; generate a first intermediate prediction block based on the first set of sub-blocks; divide the block into a second set of sub-blocks according to the second pattern, wherein at least one sub-block in the second set is not in the first set ; Generate a second intermediate prediction block based on the second set of sub-blocks; and determine a prediction block based on the first intermediate prediction block and the second intermediate prediction block. The method according to claim 22, wherein interlaced prediction is used only for the luminance component to form the prediction block. The method according to claim 22, wherein different first styles or second styles are used to segment different components of the video. The method according to any one of Claims 22 to 24, wherein the video condition includes the direction of prediction, and wherein the interleaving prediction is performed only for one of unidirectional prediction or bidirectional prediction, and not for the unidirectional prediction Perform the interleaving prediction with the other of the bidirectional prediction. The method according to claim 22, wherein the video condition includes the use of a low-delay P coding mode, and wherein in the case of using the low-delay P mode, the method includes suppressing the interleaving prediction. The method according to claim 22, wherein the video condition includes using the current block The current picture is used as a reference for this prediction. The method according to any one of claim 1, 5, 8, 10, or 22, wherein the interleaving-based predictive coding includes using only a first set of the sub-block from a part of the current block and the first set of the sub-block Two sets. The method according to claim 28, wherein a smaller part of the current block excludes samples in the boundary area of the current block. The method according to claim 28, wherein using a part of the current block to perform interleaving-based predictive coding includes using the part of the current block to perform affine prediction. The method according to claim 22, wherein determining the prediction block includes using a weighted average of the first intermediate prediction block and the second intermediate prediction block to determine the prediction block. The method according to claim 31, wherein the first weight Wa of the first intermediate prediction block and the second weight Wb of the second intermediate prediction block satisfy the condition Wa+Wb=2 ^N , where N is an integer. The method according to claim 32, wherein Wa=3 and Wb=1. The method according to claim 22, wherein when the first sample is closer to the position where the motion vector of the sub-block is derived than the second sample, the weight value of the first sample in the sub-block is greater than that in the sub-block The weight value of the second sample. A device comprising a processor and a non-transitory memory with instructions thereon, wherein when the instruction is executed by the processor, the processor is caused to implement any one of request items 1, 5, 8, 10, or 22. The method described. A computer program product is stored on a non-transitory computer readable medium, the computer program product includes program code, and the program code is used to execute the method described in any one of claim items 1, 5, 8, 10 or 22.

Images