TWI833795B - Fast encoding methods for interweaved prediction - Google Patents

Abstract

Methods, systems, and devices related to sub-block based motion prediction in video coding are described. In one representative aspect, a method for video processing includes determining, based on a characteristic of a video block, a prediction block for the video block using a first intermediate prediction block and a second intermediate prediction block, and generating a coded representation of the video block using the prediction block. The first intermediate prediction block is generated by partitioning the video block into a first set of sub-blocks and the second intermediate prediction block is generated by partitioning the video block into a second set of sub-blocks. At least one sub-block in the second set has a different size than a sub-block in the first set.

Description

Fast coding method for interleaved prediction

This patent document relates to video processing technologies, devices and systems.

[Cross-reference to related applications]

This application timely claims priority and interest in International Patent Application No. PCT/CN2018/103770, filed on September 3, 2018, under applicable patent law and/or under the rules of the Paris Convention. The entire disclosure of International Patent Application No. PCT/CN2018/103770 is incorporated by reference as part of the disclosure of this patent document for all purposes.

Motion compensation (MC) is a technique in video processing that is used to predict frames in a video by taking into account the motion of the camera and/or objects in the video, given previous and/or future frames. Motion compensation can be used in the encoding of video data for video compression.

This document discloses methods, systems and apparatuses related to sub-block based motion prediction in video motion compensation.

In one representative aspect, a method of video processing is disclosed. the The method includes determining a prediction block of the video block using a first intermediate prediction block and a second intermediate prediction block based on characteristics of the video block, and generating a codec representation of the video block using the prediction block. A first intermediate prediction block is generated by subdividing the video block into a first set of sub-blocks, and a second intermediate prediction block is generated by subdividing the video block into a second set of sub-blocks. At least one sub-block in the second set has a different size than the sub-block in the first set.

In another representative aspect, a method for improving bandwidth usage and prediction accuracy of a block-based motion prediction video system is disclosed. The method includes selecting a set of pixels from a video frame to form a block, subdividing the block into a first set of sub-blocks based on a first pattern, generating a first intermediate prediction block based on the first set of sub-blocks, dividing the block based on a second pattern subdividing into a second set of sub-blocks, generating a second intermediate prediction block based on the second set of sub-blocks, and determining a prediction block based on the first intermediate prediction block and the second intermediate prediction block. At least one sub-block in the second set has a different size than the sub-block in the first set.

In another representative aspect, a method for improving block-based motion prediction in a video system is disclosed. Methods include selecting a set of pixels from a video frame to form a block, subdividing the block into a plurality of sub-blocks based on dimensions of the block or information from another block spatially or temporally adjacent to the block, and by applying a coding and decoding algorithm Generate motion vector predictions to multiple sub-blocks. At least one sub-block of the plurality of sub-blocks has a different size than other sub-blocks.

In another representative aspect, an apparatus is disclosed that includes a processor and non-transitory storage having instructions thereon. When instructions are executed by the processor, the processor Selecting a set of pixels from the video frame to form a block, subdividing the block into a first set of sub-blocks according to a first pattern, generating a first intermediate prediction block based on the first set of sub-blocks, subdividing the block into sub-blocks according to a second pattern a second set of blocks, wherein at least one sub-block in the second set has a different size than the sub-block in the first set, generating a second intermediate prediction block based on the second set of sub-blocks, and based on the first intermediate prediction block and the second intermediate prediction block determine the prediction block.

In yet another representative aspect, various techniques described herein may be implemented as a computer program product stored on a non-transitory computer-readable medium. A computer program product contains program code for performing the methods described herein.

In yet another representative aspect, a video decoder device can implement a method as described herein.

Details of one or more implementations are set forth in the accompanying attachments, drawings, and description below. Other features will become apparent from the description and drawings and from the claims.

100, 200, 300, 400: blocks

101: Sub-block

500:Current CU

501:Left

502:Up

503:upper right

504:lower left

505:upper left

600:CU

601: Sub-CU

650:Reference picture

651:Block

700:8×8CU

701~704:A~D

711~714:a~d

900: block

901: Fill area

1000, 1100: current CU

1001, 1002:MV0, MV1

1003, 1004:TD0, TD1

1010, 1011, 1100, 1110: Reference pictures

1200: Unilateral motion estimation

1300:Current block

1301, 1302: Style 0, Style 1

1303, 1304, 1305:P1, P2, P

1500, 1550: Method

1502~1512, 1552~1556:

1600:Computer systems

1605:Multiple processors

1610:Storage

1615:Network adapter

1625:Interconnects

1700:Mobile device

1701: Processor or controller

1702:Storage

1703:I/O interface and sensors

1704:Display device

1900:Method

1902, 1904: Operation

2000:System

2002:Enter

2004: Codec component

2006, 2008: Components

2010: Display interface

A, B, C, D, E: sub-blocks

V0, V1, v1~v4: vector

τ ₀ , τ ₁ : distance

FIG. 1 is a schematic diagram showing an example of sub-block based prediction.

Figure 2 shows an example of an affine motion field for a block described by two control point motion vectors.

Figure 3 shows an example of an affine motion vector field for each sub-block of a block.

Figure 4 shows an illustration of motion vector prediction for block 400 in AF_INTER mode. example.

FIG. 5A shows an example of the selection order of candidate blocks of the current coding and decoding unit (CU).

FIG. 5B shows another example of candidate blocks of the current CU in AF_MERGE mode.

Figure 6 shows an example of an alternative temporal motion vector prediction (ATMVP) motion prediction process for a CU.

Figure 7 shows an example of one CU with four sub-blocks and adjacent blocks.

Figure 8 shows an exemplary optical flow trajectory in the Bidirectional Optical Flow (BIO) method.

Figure 9A shows an example of an access location outside a block.

Figure 9B shows a padding area that can be used to avoid extra memory accesses and computations.

Figure 10 shows an example of bilateral matching used in the frame rate up conversion (FRUC) method.

Figure 11 shows an example of template matching used in the FRUC method.

Figure 12 shows an example of one-sided motion estimation (ME) in the FRUC method.

13 illustrates an example of interleaved prediction with two subdivision styles according to the disclosed technology.

14A illustrates an exemplary subdivision pattern for subdividing a block into 4×4 sub-blocks in accordance with the disclosed technology.

14B illustrates an exemplary subdivision pattern for subdividing a block into 8×8 sub-blocks in accordance with the disclosed technology.

Figure 14C illustrates an exemplary subdivision pattern for subdividing a block into 4x8 sub-blocks in accordance with the disclosed technology.

14D illustrates an exemplary subdivision pattern for subdividing a block into 8×4 sub-blocks in accordance with the disclosed technology.

14E illustrates an exemplary subdivision pattern for subdividing a block into non-uniform sub-blocks in accordance with the disclosed technology.

Figure 14F illustrates another exemplary subdivision pattern for subdividing a block into non-uniform sub-blocks in accordance with the disclosed technology.

14G illustrates yet another exemplary subdivision pattern for subdividing a block into non-uniform sub-blocks in accordance with the disclosed technology.

15A is an exemplary flow diagram of a method for improving bandwidth usage and prediction accuracy of a block-based motion prediction video system in accordance with the techniques of this disclosure.

15B is another exemplary flow diagram of a method for improving bandwidth usage and prediction accuracy of a block-based motion prediction video system in accordance with the techniques of this disclosure.

16 is a block diagram illustrating an example of the architecture of a computer system or other control device that may be used to implement portions of the techniques of this disclosure.

17 illustrates a block diagram of an exemplary embodiment of a mobile device that may be used to implement portions of the disclosed technology.

Figures 18A to 18C illustrate exemplary embodiments of partially interleaved prediction techniques.

Figure 19 is a flowchart representation of a method for improving block-based motion prediction in a video system.

20 is an illustration of an exemplary video processing system in which the techniques of the present disclosure may be implemented. block diagram.

Global motion compensation is one of many variations of motion compensation technology and can be used to predict camera motion. However, moving objects within a frame are not adequately represented by various implementations of global motion compensation. Local motion estimation, such as block motion compensation, which subdivides a frame into blocks of pixels for motion prediction, can be used to account for objects moving within a frame.

Sub-block-based prediction developed based on block motion compensation was first introduced into the video codec standard by High Efficiency Video Coding (HEVC) Annex I (3D-HEVC). FIG. 1 is a schematic diagram showing an example of sub-block based prediction. In the case of sub-block based prediction, a block 100 (such as a codec unit (CU) or a prediction unit (PU)) is subdivided into several non-overlapping sub-blocks 101. Different sub-blocks may be assigned different motion information, such as reference indices or motion vectors (MVs). Motion compensation is then performed individually for each sub-block.

In order to explore future video codec technologies beyond HEVC, the Joint Video Exploration Team (JVET) was jointly established in 2015 by the Video Codec Experts Group (VCEG) and the Moving Picture Experts Group (MPEG). Many methods have been adopted by JVET and added to a reference software called the Joint Exploration Model (JEM). In JEM, sub-block based prediction is adopted in several codec techniques, such as affine prediction, optional temporal motion vector prediction (ATMVP), spatial-temporal motion vector prediction (STMVP), bi-directional motion vector prediction, discussed in detail below. Optical flow (BIO), and frame rate upconversion (FRUC).

Affine prediction

In HEVC, only translational motion models are applied for motion compensated prediction (MCP). However, cameras and objects can have many types of motion, such as zoom in/out, rotation, perspective motion, and/or other unusual motion. JEM, on the other hand, applies a simplified affine transform for motion-compensated prediction. Figure 2 shows an example of an affine motion field for a block 200 described by two control point motion vectors V0 and V1. The motion vector field (MVF) of block 200 can be described by the following equation:

As shown in Figure 2, (v0x, v0y) is the motion vector of the upper left corner control point, and (v1x, v1y) is the motion vector of the upper right corner control point. To simplify motion compensated prediction, sub-block based affine transform prediction can be applied. The sub-block size M×N is derived as follows:

Here, MvPre is the motion vector fractional precision (eg, 1/16 in JEM). (v2x, v2y) is the motion vector of the lower left control point calculated according to equation (1). If necessary, M and N can be adjusted downwards so that they are divisors of w and h respectively.

Figure 3 shows an example of affine MVF for each sub-block of block 300. To derive the motion vector of each M × N sub-block, each sub-block can be calculated according to Equation (1) The motion vector of the central sample, and rounded to the motion vector fractional precision (e.g., 1/16 in JEM). A motion compensated interpolation filter can then be applied to generate predictions for each sub-block with the derived motion vectors. After MCP, the high-precision motion vector of each sub-block is rounded and saved with the same precision as the normal motion vector.

In JEM, there are two affine motion modes: AF_INTER mode and AF_MERGE mode. For CUs where both width and height are greater than 8, AF_INTER mode can be applied. An affine flag in the CU level is signaled in the bitstream to indicate whether to use AF_INTER mode. In AF_INTER mode, adjacent blocks are used to construct candidates with motion vector pairs {(v ₀ ,v ₁ )|v ₀ ={v _A ,v _B ,v _C },v ₁ ={v _D ,v _E }} list. Figure 4 shows an example of motion vector prediction (MVP) for block 400 in AF_INTER mode. As shown in Figure 4, v0 is selected from the motion vector of sub-block A, B or C. Motion vectors from neighboring blocks can be scaled according to the reference list. The motion vector may also be scaled according to the relationship between the picture order count (POC) of the neighboring block's reference, the POC of the current CU's reference, and the POC of the current CU. The scheme of selecting v1 from adjacent sub-blocks D and E is similar. If the number of candidate lists is less than 2, then the list is populated by motion vector pairs formed by copying each of the AMVP candidates. When the candidate list is larger than 2, the candidates may first be classified based on adjacent motion vectors (eg, based on the similarity of two motion vectors in a pair of candidates). In some implementations, the first two candidates are retained. In some embodiments, a rate-distortion (RD) cost check is used to determine which motion vector pair candidate is selected as the control point motion vector prediction (CPMVP) for the current CU. An index indicating the position of the CPMVP in the candidate list may be signaled in the bitstream. After the CPMVP of the current affine CU is determined, affine motion estimation is applied, and the control point motion vector (CPMV) is found. The difference between CPMV and CPMVP is then signaled in the bitstream.

When a CU is applied in AF_MERGE mode, it gets the first block encoded in affine mode from valid neighboring reconstructed blocks. FIG. 5A shows an example of the selection order of candidate blocks of the current CU 500. As shown in Figure 5A, the selection order may be from left (501), upper (502), upper right (503), lower left (504) to upper left (505) of the current CU 500. FIG. 5B shows another example of candidate blocks for the current CU 500 in AF_MERGE mode. If the adjacent lower left block 501 is coded in affine mode, as shown in Figure 5B, then the motion vectors v2, v3 and v4 of the upper left corner, upper right corner and lower left corner of the CU containing sub-block 501 are derived. The motion vector v0 of the upper left corner on the current CU 500 is calculated based on v2, v3 and v4. The upper right motion vector v1 of the current CU is calculated accordingly.

After the CPMV of the current CU v0 and v1 is calculated according to the affine motion model in Equation (1), the MVF of the current CU can be generated. In order to identify whether the current CU is coded in AF_MERGE mode, when there is at least one neighboring block coded in affine mode, an affine flag may be signaled in the bit stream.

Optional Temporal Motion Vector Prediction (ATMVP)

In the ATMVP method, the temporal motion vector prediction (TMVP) method is modified by retrieving multiple sets of motion information (containing motion vectors and reference indices) from blocks smaller than the current CU.

FIG. 6 shows an example of the ATMVP motion prediction process of the CU 600. The ATMVP method predicts motion vectors for sub-CUs 601 within CU 600 in two steps. The first step is to identify the corresponding block 651 in the reference picture 650 with a temporal vector. The reference picture 650 is also called a motion source picture. The second step is to partition the current CU 600 into sub-CUs 601 and obtain the motion vector and reference index of each sub-CU from the block corresponding to each sub-CU.

In the first step, the reference picture 650 and the corresponding block are determined from the motion information of the spatial neighboring blocks of the current CU 600 . In order to avoid repeated scanning processes of adjacent blocks, the first MERGE candidate in the MERGE candidate list of the current CU 600 is used. The first available motion vector and its associated reference index are set to the temporal vector and the index of the motion source picture. In this way, the corresponding block can be identified more accurately than in TMVP, where the corresponding block (sometimes called a collocation block) is always in the lower right or center position relative to the current CU.

In a second step, the corresponding block of the sub-CU 651 is identified by the time vector in the motion source picture 650 by adding the time vector to the coordinates of the current CU. For each sub-CU, the motion information of the sub-CU is derived using the motion information of its corresponding block (eg, the minimum motion grid covering the central sample). After identifying the motion information of the corresponding N×N block, it is converted into the motion vector and reference index of the current sub-CU in the same way as HEVC’s TMVP, where motion scaling and other processes are applicable. For example, the decoder checks whether the low latency condition is met (e.g., the POC of all reference pictures of the current picture is less than the POC of the current picture), and possibly uses the motion vector MVx (e.g., the motion vector corresponding to the reference picture list X) to predict each The motion vector MVy of the sub-CU (eg, X equals 0 or 1, and Y equals 1-X).

Spatial-temporal motion vector prediction (STMVP)

In the STMVP method, the motion vectors of sub-CUs are derived recursively following the raster scanning order. Figure 7 shows an example of one CU with four sub-blocks and adjacent blocks. Consider an 8x8 CU 700, which contains four 4x4 sub-CUs, A(701), B(702), C(703), and D(704). Adjacent 4x4 blocks in the current frame are labeled a (711), b (712), c (713), and d (714).

The motion derivation of sub-CU A starts by identifying its two spatial neighbors. The first neighborhood is the NxN block above sub-CU A 701 (block c 713). If block c (713) is not available or is intra-coded, then other NxN blocks above sub-CU A (701) are checked (from left to right, starting with block c 713). The second neighborhood is the block to the left of sub-CU A 701 (block b 712). If block b (712) is not available or is intra-coded, then other blocks to the left of sub-CU A 701 are checked (from top to bottom, starting with block b 712). The motion information obtained from neighboring blocks for each list is scaled to the first reference frame for the given list. Next, the temporal motion vector prediction (TMVP) of sub-block A 701 is derived by following the same process as TMVP as specified in HEVC. The motion information of the collocated block at block D 704 is retrieved and scaled accordingly. Finally, after extracting and scaling the motion information, all available motion vectors are averaged separately for each reference list. The average motion vector is assigned as the motion vector of the current sub-CU.

Bidirectional Optical Flow (BIO)

The Bidirectional Optical Flow (BIO) method is based on bidirectionally predicted blocks. Sample-wise motion refinement on top of (block-wise) motion compensation. In some implementations, sample-level motion refinement does not use signaling.

I ^(k)/

x、

I ^(k)/

y分別為I ^(k)梯度的水平和垂直分量。假設光流是有效的，運動向量場(v _x ,v _y )由以下給出：

Let I ^{( k )} be the brightness value from reference k (k=0,1) after block motion compensation, and

I ^{( k )} /

x ,

I ^{( k )} /

y are the horizontal and vertical components of the gradient of I ^{( k )} respectively. Assuming that optical flow is effective, the motion vector field ( v _x , v _y ) is given by:

I ^(k)/

x,

I ^(k)/

y兩者。該多項式在t=0處的值是BIO預測：

Combining this optical flow equation with Hermite interpolation for each sample's motion trajectory results in a unique third-order polynomial that matches the function value I ^{( k )} and the derivative at the ends

I ^{( k )} /

x ,

I ^{( k )} /

y both. The value of this polynomial at t=0 is the BIO prediction:

Figure 8 shows an exemplary optical flow trajectory in the Bidirectional Optical Flow (BIO) method. Here, τ ₀ and τ ₁ refer to the distance from the reference frame. Calculate the distance τ ₀ and τ ₁ based on the POC of Ref0 and Ref1: τ 0 =POC(current)-POC(Ref0), τ 1 =POC(Ref1)-POC(current). If two predictions come from the same direction in time (both from the past or from the future), the signs are different (e.g., τ ₀ · τ ₁ <0). In this case, BIO is applied if the predictions are not from the same time instant (eg, τ ₀ ≠ τ ₁ ). Both reference regions have non-zero motion (e.g., MVx ₀ , MVy ₀ , MVx ₁ , MVy ₁ ≠0), and the block motion vector is proportional to the temporal distance (e.g., MVx ₀ / MVx ₁ = MVy ₀ / MVy ₁ =-τ ₀ /τ ₁ ).

The motion vector field ( v _x , v _y ) is determined by minimizing the difference Δ between the values in point A and point B. 9A-9B illustrate examples of intersections of motion trajectories and reference frame planes. For Δ, the model uses only the first linear term of the local Taylor expansion:

All values in the above equation depend on the sample location, referred to as ( i',j' ). Assuming that the motion is consistent in the local surrounding area, then △ can be minimized within a (2M+1)×(2M+1) square window Ω centered on the currently predicted point ( i,j ), where M equals 2:

For this optimization problem, JEM uses a simplified scheme, minimizing first in the vertical direction and then in the horizontal direction. This results in the following:

in,

To avoid division by zero or very small values, regularization parameters r and m can be introduced into Equation (7) and Equation (8).

r =500.4 ^{d -8Equation} (10)

m =700.4 ^{d -8} Equation (11) Here, d is the bit depth of the video sample.

I ^(k)/

x,

I ^(k)/

y。圖9A示出了塊900之外的存取位置的示例。如圖9A中所示，在等式(9)中，預測的塊的邊界上的當前預測的點為中心的(2M+1)×(2M+1)正方形窗Ω需要存取塊之外的位置。在JEM中，塊之外的I ^(k),

I ^(k)/

x,

I ^(k)/

y的值被設定為等於塊內的最接近可用值。例如，這可以實現為填充區域901，如圖9B中所示。 To make memory access like BIO the same as conventional bidirectional predictive motion compensation, all prediction and gradient values I ^{( k )} are calculated for the position within the current block,

I ^{( k )} /

x ,

I ^{( k )} /

y . Figure 9A shows an example of access locations outside of block 900. As shown in Figure 9A, in Equation (9), a (2M+1)×(2M+1) square window Ω centered on the currently predicted point on the boundary of the predicted block requires access outside the block. Location. In JEM, I ^{( k )} outside the block,

I ^{( k )} /

x ,

I ^{( k )} /

The value of y is set equal to the closest available value within the block. For example, this may be implemented as filling area 901, as shown in Figure 9B.

Using BIO, the playing field can be refined for each sample. To reduce computational complexity, block-based designed BIOs are used in JEM. Motion refinement can be calculated based on 4×4 blocks. In block-based BIO, the values of sn in Equation (9) for all samples in a 4×4 block can be aggregated, and then the aggregated values of sn are used to derive the BIO motion vector bias of the 4×4 block. shift. More specifically, the following formula can be used for block-based BIO export:

Here, bk refers to the set of samples belonging to the k-th 4×4 block of the predicted block. sn in Equation (7) and Equation (8) is replaced by ((sn,bk)>>4) to derive the associated motion vector offset.

In some situations, the BIO's MV regiment may be unreliable due to noise or irregular movement. Therefore, in BIO, the size of the MV clump is trimmed to a threshold. The threshold is determined based on whether the reference pictures of the current picture all come from one direction. For example, if all reference pictures of the current picture come from one direction, the value of the threshold is set to 12×2 ^{14- d} ; otherwise, it is set to 12×2 ^{13- d} .

I/

x，首先使用BIOfilterS對信號進行垂直地插值，其對應於具有去縮放位移d-8的分數位置fracY。然後將梯度濾波器BIOfilterG應用於對應於具有去縮放位移18-d的分數位置fracX的水平方向上。對於垂直梯度

I/

y，對應於具有去縮放位移d-8的分數位置fracY，使用BIOfilterG垂直地應用梯度濾波器。然後對應於具有去縮放位移18-d的分數位置fracX，在水平方向使用BIOfilterS進行信號置換。梯度計算的插值濾波器的長度BIOfilterG和信號置換BIOfilterF可以更短(例如，6-tap)，以便保持合理的複雜度。表1示出了可以用於BIO中的塊運動向量的不同分數位置的梯度計算的示例性濾波器。表2示出了可以用於BIO中的預測信號生成的示例性插值濾波器。 The gradient of the BIO may be calculated simultaneously with motion compensated interpolation using operations consistent with the HEVC motion compensation process (eg, 2D separable finite impulse response (FIR)). In some embodiments, the input to the 2D separable FIR is the same reference frame sample as the motion compensation process and the fractional position (fracX, fracY) according to the fractional part of the block motion vector. For horizontal gradient

I /

x , the signal is first interpolated vertically using BIOfilterS, which corresponds to the fractional position fracY with a descaled displacement d-8. The gradient filter BIOfilterG is then applied in the horizontal direction corresponding to the fractional position fracX with a descaled displacement of 18-d. for vertical gradient

I /

y , corresponding to the fractional position fracY with descaled displacement d-8, apply the gradient filter vertically using BIOfilterG. Signal permutation is then performed using BIOfilterS in the horizontal direction corresponding to the fractional position fracX with a descaled displacement of 18-d. The lengths of the gradient calculation interpolation filter BIOfilterG and the signal permutation BIOfilterF can be shorter (e.g., 6-tap) in order to keep the complexity reasonable. Table 1 shows exemplary filters that can be used for gradient calculation of different fractional positions of block motion vectors in BIO. Table 2 shows exemplary interpolation filters that may be used for prediction signal generation in BIO.

Table 1: Exemplary filters for gradient calculation in BIO

In JEM, BIO can be applied to all bidirectional prediction blocks when the two predictions come from different reference pictures. BIO can be disabled when Local Illumination Compensation (LIC) is enabled for the CU.

In some embodiments, OBMC is applied to the block after the normal MC process. To reduce computational complexity, BIO may not be applied during the OBMC process. This means that during the OBMC process, when using its own MV, the BIO is applied during the MC process of a block, and when the MV of an adjacent block is used, BIO is not applied during the MC process.

Frame rate upconversion (FRUC)

When the CU's Merge flag is true, the FRUC flag may be signaled to the CU. When the FRUC flag is false, the Merge index can be signaled and regular Merge mode is used. When the FRUC flag is true, an additional FRUC mode flag may be signaled to indicate which method (eg, bilateral matching or template matching) is to be used to derive the motion information of the block.

On the encoder side, the decision whether to use FRUC Merge mode for a CU is based on RD cost selection, as done for normal Merge candidates. For example, by using RD cost selection, multiple matching modes (eg, bilateral matching and template matching) are checked for CU. The one pointing to the minimum cost is further compared with other CU modes. If the FRUC matching mode is the most efficient one, then the FRUC flag is set to true for the CU and the associated matching mode is used.

Typically, the motion derivation process in FRUC Merge mode has two steps: first CU-level motion search, followed by sub-CU-level motion refinement. At the CU level, initial motion vectors are derived for the entire CU based on bilateral matching or template matching. First, a list of MV candidates is generated, and the candidate pointing to the minimum matching cost is selected as the starting point for further CU-level refinement. A local search based on bilateral matching or template matching is then performed around the starting point. The MV that results in the minimum matching cost is taken as the MV of the entire CU. Subsequently, the motion information is further refined at the sub-CU level using the derived CU motion vectors as a starting point.

For example, the following derivation process is performed for W×H CU motion information derivation. In the first stage, the MV of the entire W×H CU is derived. In the second stage, the CU is further subdivided into M×M sub-CUs. The value of M is calculated as in (16), and D is the predetermined segmentation depth, which is set to 3 by default in JEM. Then export the MV of each sub-CU.

Figure 10 shows an example of bilateral matching used in the frame rate up conversion (FRUC) method. Bilateral matching is used to derive the motion information of the current CU by finding the closest match between two blocks along the motion trajectory of the current CU (1000) in two different reference pictures (1010, 1011). Under the assumption of continuous motion trajectories, the motion vectors MV0 (1001) and MV1 (1002) pointing to the two reference blocks are related to the temporal distance between the current picture and the two reference pictures - for example, TD0 (1003) and TD1 ( 1004) in proportion. In some embodiments, when the current picture 1000 is temporally between two reference pictures (1010, 1011) and the temporal distance from the current picture to the two reference pictures is the same, the bilateral matching becomes bidirectional MV based mirroring . Figure 11 shows an example of template matching used in the FRUC method. Template matching may be used by finding the closest match between a template in the current picture (e.g., the upper and/or left neighbor block of the current CU) and a block in the reference picture 1110 (e.g., the same size as the template). Export the current motion information of CU 1100. In addition to the above FRUC Merge mode, template matching can also be applied to AMVP mode. AMVP has two candidates among both JEM and HEVC. Using template matching methods, new candidates can be derived. If the newly exported candidate is matched by template with the first existing If the AMVP candidate is different, it is inserted at the beginning of the AMVP candidate list, and the list size is then set to 2 (eg, by removing the second existing AMVP candidate). When applied to AMVP mode, only CU-level searches are applied.

The MV candidates set at the CU level can include the following: (1) If the current CU is in AMVP mode, it is the original AMVP candidate, (2) all Merge candidates, (3) several MVs in the interpolated MV field (described later), and Up and left neighboring motion vectors.

When using bilateral matching, each valid MV of the Merge candidate can be used as input to generate MV pairs under the assumption of bilateral matching. For example, a valid MV for the Merge candidate is (MVa, refa) at reference list A. Then, the reference picture refb whose paired bilateral MV is found in other reference list B is such that refa and refb are on different sides of the current picture in time. If such refb is not available in reference list B, refb is determined as a reference different from refa and its temporal distance to the current picture is the smallest one in list B. After determining refb, based on the time domain distance between the current picture and refa and refb, MVb is derived by scaling MVa.

In some implementations, four MVs from the interpolated MV field may also be added to the CU-level candidate list. More specifically, the interpolated MVs at the positions (0,0), (W/2,0), (0,H/2), and (W/2,H/2) of the current CU are added. When FRUC is applied in AMVP mode, the original AMVP candidates are also added to the CU-level MV candidate set. In some implementations, at the CU level, 15 MVs for AMVP CUs and 13 MVs for Merge CUs may be added to the candidate list.

The MV candidate set at the sub-CU level includes MVs determined from (1) CU-level searches, (2) top, left, top left and top right adjacent MVs, (3) scaled versions of collocated MVs from reference pictures, (4) One or more ATMVP candidates (eg, up to four), and (5) one or more STMVP candidates (eg, up to four). The scaled MV from the reference image is exported as follows. Iterate over the reference images in both lists. The MV at the collocation position of the sub-CU in the reference picture is scaled to the reference of the starting CU-level MV. ATMVP and STMVP candidates can be the top four. At the sub-CU level, one or more MVs (eg, up to 17) are added to the candidate list.

Generation of interpolated MV fields

Before encoding and decoding the frame, an interpolated motion field is generated for the entire picture based on single-sided ME. The sports fields can then be used later as CU-level or sub-CU-level MV candidates.

In some embodiments, the motion field for each reference picture in both reference lists is traversed at the 4x4 block level. Figure 12 shows an example of one-sided motion estimation (ME) 1200 in the FRUC method. For each 4×4 block, if the motion associated with the block passes through the 4×4 block in the current picture (as shown in Figure 12) and the block has not been assigned any interpolation motion, the motion of the reference block is based on the temporal distance TD0 and TD1 (in the same way as TMVP's MV scaling in HEVC) scales to the current picture, and the scaled motion is assigned to the blocks in the current frame. If no scaled MV is assigned to a 4×4 block, the motion of the block is marked as unavailable in the interpolated motion field.

Interpolation and matching costs

When motion vectors point to fractional sample locations, motion compensated interpolation is required. To reduce complexity, bilinear interpolation can be used instead of conventional 8-tap HEVC interpolation for both bilateral matching and template matching.

The calculation of matching costs differs slightly at different steps. When selecting candidates from a CU-level candidate set, the matching cost can be the sum of absolute differences (SAD) for bilateral matching or template matching. After determining the starting MV, the matching cost C of bilateral matching for sub-CU level search is calculated as follows: C = SAD + w . (| MV _x - MV _x ^s |+| MV _y - MV _y ^s |) Equation (14)

Here, w is the weight factor. In some embodiments, w can be empirically set to 4. MV and MVs indicate the current MV and starting MV respectively. SAD can still be used as the matching cost for template matching for sub-CU level searches.

In FRUC mode, MV is derived by using luma samples only. The derived motion will be used for both luma and chroma of MC inter prediction. After deciding the MV, the final MC is performed using an 8-tap interpolation filter for luma and a 4-tap interpolation filter for chroma.

MV refinement is a style based on MV search with criteria of bilateral matching cost or template matching cost. In JEM, two search styles are supported - Unrestricted Central Biased Diamond Search (UCBDS) and adaptive cross search for MV refinement at the CU level and sub-CU level respectively. For CU and sub-CU level MV refinement, MVs are searched directly with one-quarter luma sample MV precision, and followed by one-eighth luma sample MV refinement. The search range for MV refinement for CU and sub-CU steps is set equal to 8 luma samples.

In bilateral matching Merge mode, bidirectional prediction is applied because CU's The motion information is derived based on the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures. In template matching Merge mode, the encoder can choose between unidirectional prediction from list0, unidirectional prediction from list1, or bidirectional prediction for the CU. The selection can be based on template matching cost as follows: if costBi<=factor * min(cost0, cost1) then use bidirectional prediction; otherwise, if cost0<=cost1 then use unidirectional prediction from list0; otherwise, use unidirectional prediction from list1 Unidirectional prediction; here, cost0 is the SAD for list0 template matching, cost1 is the SAD for list1 template matching, and costBi is the SAD for bidirectional prediction template matching. For example, when the value of factor is equal to 1.25, it means that the selection process is biased towards bidirectional prediction. Inter prediction direction selection can be applied to the CU-level template matching process.

When the size of sub-blocks is smaller, the sub-block-based prediction techniques discussed above can be used to obtain more accurate motion information for each sub-block. However, smaller sub-blocks impose higher bandwidth requirements in motion compensation. On the other hand, the motion information derived for smaller sub-blocks may be inaccurate, especially when there is some noise in the block. Therefore, having fixed sub-block sizes within a block may be suboptimal.

This document describes techniques that can be used in various embodiments to address the bandwidth and accuracy introduced by fixed sub-block sizes using non-uniform and/or variable sub-block sizes. problem. These techniques (also known as interleaved prediction) use different ways of subdividing blocks, allowing for more robust acquisition of motion information without increasing bandwidth consumption.

Using interleaved prediction techniques, blocks are subdivided into sub-blocks in one or more subdivision modes. A subdivision style represents the way a block is subdivided into subblocks, including the size of the subblocks and the location of the subblocks. For each subdivision style, a corresponding prediction block can be generated by deriving motion information of each sub-block based on the subdivision style. Therefore, in some embodiments, multiple prediction blocks may be generated by multiple subdivision patterns even for one prediction direction. In some embodiments, only one segmentation style may be applied for each prediction direction.

13 illustrates an example of interleaved prediction with two subdivision patterns according to the techniques of this disclosure. The current block 1300 may be subdivided into multiple styles. For example, as shown in Figure 13, the current block is subdivided into both Pattern 0 (1301) and Pattern 1 (1302). Two prediction blocks P0 (1303) and P1 (1304) are generated. The final prediction block P (1305) of the current block 1300 may be generated by calculating the weighted sum of P0 (1303) and P1 (1304).

More generally, given X subdivision styles, X prediction blocks of the current block can be generated by sub-block based prediction with . The final prediction of the current block, denoted as P, can be generated as

Here, (x, y) are the coordinates of the pixels in the block, and w _i ( x, y ) are the weight values of Pi. By way of example and without limitation, the weights can be expressed as:

N is a non-negative value. Alternatively, the bit shift operation in equation (16) can also be expressed as:

The sum of the weights as a power of 2 allows the weighted sum P to be calculated more efficiently by performing bit shifting operations instead of floating point division.

Subdivision styles can have different shapes, sizes, or locations of subblocks. In some embodiments, the subdivision pattern may include irregular sub-block sizes. Figures 14A-14G illustrate several examples of subdivision patterns for 16x16 blocks. In Figure 14A, the block is subdivided into 4x4 sub-blocks according to the disclosed techniques. This style is also used in JEM. Figure 14B shows an example of subdividing a block into 8x8 sub-blocks in accordance with the techniques of this disclosure. Figure 14C shows an example of subdividing a block into 8x4 sub-blocks in accordance with the disclosed technology. Figure 14D shows an example of subdividing a block into 4x8 sub-blocks in accordance with the techniques of this disclosure. In Figure 14E, a portion of the block is subdivided into 4x4 sub-blocks in accordance with the techniques of this disclosure. Pixels at block boundaries are subdivided into smaller sub-blocks of size such as 2x4, 4x2, or 2x2. Some sub-chunks can be merged to form larger sub-chunks. Figure 14F shows examples of adjacent sub-blocks, such as 4x4 sub-blocks and 2x4 sub-blocks, which are merged to form larger sub-blocks with dimensions such as 6x4, 4x6, or 6x6 . In Figure 14G, part of the block is subdivided into 8×8 sub-blocks. Pixels at block boundaries are subdivided into smaller sub-blocks, with dimensions such as 8×4, 4×8, or 4×4.

The shape and size of the sub-block in the sub-block based prediction may be determined based on the shape and/or size of the codec block and/or the codec block information. For example, in some embodiments, when the current block has a size of M×N, the sub-block has a size of 4×N (or 8×N, etc.) the size of. That is, the child block has the same height as the current block. In some embodiments, when the current block has a size of M×N, the sub-block has a size of M×4 (or M×8, etc.). That is, the subblock has the same width as the current block. In some embodiments, when the current block has a size of M×N (where M>N), the sub-block has a size of A×B, where A>B (eg, 8×4). Alternatively, the sub-blocks may be of size B×A (eg 4×8).

In some embodiments, the current block has a size of MxN. When M×N<=T (or Min(M,N)<=T, or Max(M,N)<=T, etc.), the sub-block has the size of A×B, and when M×N>T( or Min(M, N)>T, or Max(M, N)>T, etc.), the sub-block has a size of C×D, where A<=C and B<=D. For example, if M×N<=256, the sub-block may be 4×4 in size. In some implementations, the sub-blocks have a size of 8x8.

In some embodiments, whether to apply interleaved prediction may be determined based on the inter prediction direction. For example, in some embodiments, interleaved prediction may be applied to bidirectional prediction but not to unidirectional prediction. As another example, when applying multiple hypotheses, interleaved prediction may be applied to one prediction direction when there is more than one reference block.

In some embodiments, how to apply interleaved prediction may also be determined based on the inter prediction direction. In some embodiments, a bidirectionally predicted block with sub-block based prediction is subdivided into sub-blocks in two different subdivision patterns for two different reference lists. For example, when predicting from reference list 0 (L0), the bidirectional prediction block is subdivided into 4×8 sub-blocks, as shown in Figure 14D. When predicting from reference list 1 (L1), the same block is subdivided into 8×4 sub-blocks, as shown in Figure 14C. The final prediction P is calculated as

Here, P0 and P1 are predictions from L0 and L1 respectively. w0 and w1 are the weight values of L0 and L1 respectively. As shown in equation (16), the weight value can be determined as: w0(x,y)+w1(x,y)=1<<N (where N is a non-negative integer value). Because there are fewer sub-blocks used for prediction in each direction (e.g., 4×8 sub-blocks compared to 8×8 sub-blocks), the computation requires less frequency than existing sub-block-based methods. wide. By using larger sub-blocks, the prediction results are also less susceptible to noise interference.

In some embodiments, a unidirectional prediction block with sub-block based prediction is subdivided into sub-blocks in two or more different subdivision patterns for the same reference list. For example, the predicted PL for the list L (L=0 or 1) is calculated as

(x,y)是用第i細 分樣式預測生成的，並且

(x,y)是

(x,y)的權重值。例如，當XL 為2時，兩種細分樣式應用於列表L。在第一細分樣式中，塊被細分為4×8子塊，如圖14D所示。在第二細分樣式中，塊被細分為8×4子塊如圖14D所示。 Here XL is the number of subdivision styles of list L.

( x,y ) is generated using the i-th subdivision style prediction, and

( x,y ) is

The weight value of ( x,y ). For example, when XL is 2, two subdivision styles are applied to list L. In the first subdivision style, the block is subdivided into 4×8 sub-blocks, as shown in Figure 14D. In the second subdivision style, the block is subdivided into 8×4 sub-blocks as shown in Figure 14D.

In one embodiment, a bi-directional prediction block with sub-block based prediction is considered to be a combination of two uni-directional prediction blocks from L0 and L1 respectively. Predictions from each list can be exported as described in the example above. The final prediction P can be calculated as

Here, parameters a and b are two additional weights applied to the two intra prediction blocks. In this specific example, a and b can both be set to 1. Similar to the example above, because fewer sub-blocks are used in each direction for prediction (e.g., 4×8 sub-blocks compared to 8×8 sub-blocks), the bandwidth usage is higher than that of existing sub-block based methods. Better or equivalent. At the same time, the prediction results can be improved by using larger sub-blocks.

In some embodiments, a single non-uniform pattern may be used in each unidirectional prediction block. For example, for each list L (eg, L0 or L1), the blocks are divided into different patterns (eg, as shown in Figure 14E or Figure 14F). Using a smaller number of sub-blocks reduces bandwidth requirements. The non-uniformity of sub-blocks also increases the robustness of the prediction results.

In some embodiments, for multi-hypothesis codec blocks, there may be more than one prediction block generated by different subdivision styles for each prediction direction (or reference picture list). Multiple prediction blocks can be used to generate a final prediction with additional weights applied. For example, the additional weight can be set to 1/M, where M is the total number of generated prediction blocks.

In some embodiments, the encoder may determine whether and how to apply interleaved prediction. The encoder can then operate at sequence level, picture level, view level, slice level, codec tree unit (CTU) (also known as largest codec unit (LCU)) level, CU level, PU level, tree unit (TU) ) level, or regional level (which may include multiple CU/PU/Tu/LCU), sends information corresponding to the determination to the decoder. Can be ordered The information is contained in the column parameter set (SPS), view parameter set (VPS), picture parameter set (PPS), slice header (SH), CTU/LCU, CU, PU, TU, or the first block of the region. order notice.

In some implementations, interleaved prediction is adapted to existing sub-block methods, such as affine prediction, ATMVP, STMVP, FRUC, or BIO. In this case, no additional signaling cost is required. In some implementations, new sub-block Merge candidates generated by interleaved prediction can be inserted into the Merge list, for example, interleaved prediction + ATMVP, interleaved prediction + STMVP, interleaved prediction + FRUC, etc.

In some embodiments, the subdivision pattern to be used for the current block may be derived based on information from spatial and/or temporal neighboring blocks. For example, instead of relying on the encoder to signal the relevant information, both the encoder and decoder could employ a set of predetermined rules to determine whether the code is based on temporal contiguity (e.g., subdivision pattern of the same block previously used) or spatial contiguity ( For example, the subdivision style used by adjacent blocks) to obtain the subdivision style.

In some embodiments, the weight value w may be fixed. For example, all subdivision styles can be weighted equally: w _i ( x,y ) = 1. In some embodiments, the weight value may be determined based on the location of the block and the subdivision style used. For example, w _i (x,y) can be different for different ( x,y ). In some embodiments, the weight value may also depend on the sub-block prediction based codec technology (eg, affine or ATMVP) and/or other codec information (eg, skip or non-skip mode, and/or MV information). In some embodiments, the encoder may determine the weight value and perform the processing at the sequence level, picture level, slice level, CTU/LCU level, CU level, PU level, or region level (which may include multiple CU/PU/Tu/ The value is sent to the decoder in the LCU)). The weight value may be signaled in the Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Slice Header (SH), CTU/LCU, CU, PU or the first block of the region. In some embodiments, the weight values may be derived from the weight values of spatial and/or temporal neighboring blocks.

Note that the interleaved prediction techniques disclosed herein may be applied to one, some, or all coding and decoding techniques in sub-block based prediction. For example, interleaved prediction techniques may be applied to affine prediction, while other codec techniques for sub-block based prediction (eg, ATMVP, STMVP, FRUC, or BIO) do not use interleaved prediction. As another example, Affine, ATMVP, and STMVP all apply the interleaved prediction techniques disclosed herein.

15A is an exemplary flow diagram of a method 1500 for improving motion prediction in a video system in accordance with the disclosed technology. Method 1500 includes selecting, at 1502, a set of pixels from a video frame to form a block. Method 1500 includes, at 1504, subdividing the block into a first set of sub-blocks according to the first pattern. Method 1500 includes generating, at 1506, a first intermediate prediction block based on a first set of sub-blocks. Method 1500 includes, at 1508, subdividing the block into a second set of sub-blocks according to the second pattern. At least one sub-block in the second set has a different size than the sub-block in the first set. Method 1500 includes generating, at 1510, a second intermediate prediction block based on a second set of sub-blocks. Method 1500 also includes determining a prediction block based on the first intermediate prediction block and the second intermediate prediction block at 1512 .

In some embodiments, (1) affine prediction method, (2) alternative temporal motion vector prediction method, (3) spatial-temporal motion vector prediction method, (4) bidirectional optical flow method, or (5 ) frame rate up-conversion method to generate the first intermediate prediction block or the second intermediate prediction block.

In some embodiments, the sub-blocks in the first set or the second set have a rectangular shape. In some embodiments, the sub-blocks in the first set of sub-blocks have a non-uniform shape. In some embodiments, the sub-blocks in the second set of sub-blocks have a non-uniform shape.

In some embodiments, the method includes determining the first pattern or the second pattern based on the size of the block. In some embodiments, the method includes determining the first pattern or the second pattern based on information from a second block that is temporally or spatially adjacent to the block.

In some embodiments, subdividing the block into a first set of sub-blocks is performed for motion prediction of the block in a first direction. In some embodiments, the block is subdivided into a second set of sub-blocks for motion prediction of the block in the second direction.

In some embodiments, subdividing the block into a first set of sub-blocks and subdividing the block into a second set of sub-blocks is performed for motion prediction of the block in a first direction. In some embodiments, the method further includes performing motion prediction of the block in the second direction by subdividing the block into a third set of sub-blocks according to the third pattern and generating a third intermediate based on the third set of sub-blocks. Predicting the block, subdividing the block into a fourth set of sub-blocks according to a fourth pattern, wherein at least one sub-block in the fourth set has a different size than a sub-block in the third set, generating a third set based on the fourth set of sub-blocks. Four intermediate prediction blocks, the second prediction block is determined based on the third intermediate prediction block and the fourth intermediate prediction block, and the third prediction block is determined based on the prediction block and the second prediction block.

In some embodiments, the method includes transmitting information of the first pattern and the second pattern of the subdivided blocks to a codec device in a block-based motion prediction video system. In some embodiments, transmitting the first style and the second style of information is between Conducted in one place: (1) sequence level, (2) picture level, (3) view level, (4) stripe level, (5) codec tree unit, (6) maximum codec unit level, (7) codec decoding unit level, (8) prediction unit level, (10) tree unit level, or (11) region level.

In some embodiments, determining the prediction result includes applying a first set of weights to a first intermediate prediction block to obtain a first weighted prediction block, and applying a second set of weights to a second intermediate prediction block to obtain a second weighting the prediction block, and calculating a weighted sum of the first weighted prediction block and the second weighted prediction block to obtain the prediction block.

In some embodiments, the first set of weights or the second set of weights includes fixed weight values. In some embodiments, the first set of weights or the second set of weights is determined based on information from another block that is temporally or spatially adjacent to the block. In some embodiments, the first set of weights or the second set of weights are determined using a coding and decoding algorithm used to generate the first prediction block or the second prediction block. In some implementations, at least one value in the first set of weights is different from another value in the first set of weights. In some implementations, at least one value in the second set of weights is different from another value in the second set of weights. In some implementations, the sum of the weights is equal to a power of two.

In some embodiments, the method includes transmitting weights to a codec device in a block-based motion prediction video system. In some embodiments, transmission weighting is performed at one of: (1) sequence level, (2) picture level, (3) view level, (4) slice level, (5) codec tree unit, (6 ) maximum codec unit level, (7) codec unit level, (8) prediction unit level, (10) tree unit level, or (11) region level.

Figure 15B is a basic diagram for improving the video system in accordance with the disclosed technology. An example flowchart of a method 1550 for block-based motion prediction. Method 1550 includes selecting, at 1552, a set of pixels from the video frame to form a block. Method 1550 includes, at 1554, subdividing the block into a plurality of sub-blocks based on the size of the block or information from another block that is spatially or temporally adjacent to the block. At least one sub-block of the plurality of sub-blocks has a different size than other sub-blocks. Method 1550 also includes generating, at 1556, a motion vector prediction by applying a coding and decoding algorithm to the plurality of sub-blocks. In some embodiments, the encoding and decoding algorithms include (1) affine prediction method, (2) optional temporal motion vector prediction method, (3) spatial-temporal motion vector prediction method, (4) bidirectional optical flow method, or (5) at least one of the frame rate upconversion methods.

In methods 1500 and 1550, partial interleaving may be achieved. Using this scheme, the samples in a first subset of prediction samples are computed as a weighted combination of the first intermediate prediction block, and the samples in a second subset of prediction samples are copied from the sub-block based prediction, where the first subset and The second subset is based on subdivision styles. The first subset and the second subset may together constitute the entire prediction block, eg, the block currently being processed. As shown in Figures 18A-18C, in various examples, the second subset excluded from interleaving may consist of (a) the corner sub-block or (b) the uppermost and lowermost rows of the sub-block or (c) the Composed of the leftmost or right column. The size of the block currently being processed can be used as a condition for deciding whether to exclude certain sub-blocks from interleaved prediction. For example, some of the conditions described below.

As described further in this document, the encoding process can avoid checking the affine pattern of blocks subdivided from a parent block, where the parent block itself is encoded with a different pattern than the affine pattern.

16 is an illustration of a computer that may be used to implement portions of the disclosed technology. A block diagram of an example of the architecture of a system or other control device 1600. In FIG. 16 , computer system 1600 includes one or more processors 1605 and storage 1610 connected via interconnect 1625 . Interconnect 1625 may represent any one or more separate physical buses, point-to-point connections, or both, connected by appropriate bridges, adapters, or controllers. Accordingly, interconnect 1625 may include, for example, a system bus, a Peripheral Component Interconnect (PCI) bus, a HyperTransport or Industry Standard Architecture (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus Line bus (USB), IIC (I2C) bus, or Institute of Electrical and Electronics Engineers (IEEE) standard 674 bus, sometimes called "Firewire".

Processor(s) 1605 may include a central processing unit (CPU) to control, for example, the overall operation of a host computer. In some embodiments, processor(s) 1605 accomplish this through execution software or firmware stored in memory 1610 . Processor(s) 1605 may be or may include one or more programmable general or special purpose microprocessors, digital signaling processors (DSPs), programmable controllers, application specific integrated circuits (ASICs), programmable logic device (PLD), etc., or a combination of such devices.

Storage 1610 may be or include the main storage of the computer system. Storage 1610 represents any suitable form of random access memory (RAM), read only memory (ROM), flash memory, etc., or a combination of such devices. In use, storage 1610 may contain, among other things, a set of machine instructions that, when executed by processor 1605, cause processor 1605 to operate to implement embodiments of the disclosed technology.

Also connected to processor(s) 1605 via interconnect 1625 is an (optional) network adapter 1615. Network adapter 1615 provides computer system 1600 with the ability to communicate with remote devices (such as storage clients, and/or other storage servers), and may be, for example, an Ethernet adapter or a Fiber Channel adapter.

17 illustrates a block diagram of an exemplary embodiment of a mobile device 1700 that may be used to implement portions of the disclosed technology. Mobile device 1700 may be a laptop computer, smartphone, tablet computer, camcorder, or other type of device capable of processing video. Mobile device 1700 includes a processor or controller 1701 to process data, and a memory 1702 in communication with processor 1701 to store and/or buffer data. For example, processor 1701 may include a central processing unit (CPU) or a microcontroller unit (MCU). In some implementations, processor 1701 may include a field programmable gate array (FPGA). In some implementations, the mobile device 1700 includes or is in communication with a graphics processing unit (GPU), a video processing unit (VPU), and/or wireless communication for various visual and/or communication data processing functions of the smartphone device. unit. For example, memory 1702 may contain and store processor executable code that, when executed by processor 1701, configures mobile device 1700 to perform various operations, such as, for example, receiving information, commands, and/or data, processing information and data, as well as transmitting or providing the processed information/data to another device, such as an actuator or external display. To support various functions of mobile device 1700 , storage 1702 may store information and data, such as instructions, software, values, images, and other data processed or referenced by processor 1701 . For example, various types of random access memory (RAM) devices, read only memory (ROM) devices, flash memory devices Configuration, and other appropriate storage media may be used to implement the storage function of the storage 1702. In some implementations, mobile device 1700 includes an input/output (I/O) unit 1703 to interface processor 1701 and/or memory 1702 with other modules, units, or devices. For example, I/O unit 1703 may interface and utilize processor 1701 and memory 1702 with various types of wireless interfaces that are compatible with typical data communications standards, for example, such as between one or more computers in the cloud and a user. between devices. In some implementations, mobile device 1700 may interface with other devices using wired connections via I/O unit 1703 . Mobile device 1700 may also interface with other external interfaces (such as data storage, and/or visual or audio display device 1704) to retrieve and transmit data and information that may be processed by a processor, stored in memory, or displayed on on the output unit of the display device 1704 or on an external device. For example, display device 1704 may display video frames based on MVP modifications in accordance with the disclosed technology (eg, video frames including prediction block 1305, as shown in FIG. 13).

In some embodiments, a video decoder device may implement a method of video decoding in which improved block-based motion prediction as described herein is used for video decoding. The method may include using a collection of pixels from a video frame to form blocks of the video. The block may be subdivided into a first set of sub-blocks according to the first pattern. The first intermediate prediction block may correspond to the first set of sub-blocks. The block may contain a second set of sub-blocks according to a second pattern. At least one sub-block in the second set has a different size than the sub-block in the first set. The method may also determine the prediction block based on the first intermediate prediction block and a second intermediate prediction block generated from the second set of sub-blocks. Other features of the method may be similar to method 1500 described above.

In some embodiments, a decoder-side approach to video decoding may use block-based motion prediction for improving video quality by using blocks of video frames for prediction, where blocks correspond to sets of blocks of pixels. The block may be subdivided into a plurality of sub-blocks based on the size of the block or information from another block spatially or temporally adjacent to the block, wherein at least one sub-block of the plurality of sub-blocks has a different size than the other sub-blocks. The decoder may use motion vector predictions generated by applying a coding and decoding algorithm to multiple sub-blocks. Additional features of this method are described with respect to Figure 15B and the corresponding description.

In some embodiments, the video decoding method may be implemented using a decoding device implemented on the hardware platform described with respect to FIGS. 16 and 17 .

partially intertwined

In some embodiments, partially interleaved prediction may be implemented as follows.

In some embodiments, interleaved prediction is applied to a portion of the current block. Prediction samples at some locations are calculated as a weighted sum of two or more sub-block based predictions. Predicted samples at other locations are not used in the weighted sum. For example, these prediction samples are copied from sub-block based predictions with a specific segmentation style.

In some embodiments, the current block is predicted from sub-block based predictions P1 and P2 with subdivision pattern D0 and subdivision pattern D1 respectively. The final prediction is calculated as P=w0×P0+w1×P1. At some locations, w0≠0 and w1≠0. But at some other positions, w0=1 and w1=0, i.e., interleaved prediction should not be applied at those positions.

In some embodiments, interleaved prediction is not applied to the four corner sub-blocks as shown in Figure 18A.

In some embodiments, interleaved prediction is not applied to the leftmost column of the subblock and the rightmost column of the subblock as shown in Figure 18B.

In some embodiments, interleaved prediction is not applied to the topmost row of a subblock and the bottommost row of a subblock as shown in Figure 18C.

Examples of techniques integrated within encoder embodiments

In some embodiments, interleaved prediction is not used in the motion estimation (ME) process.

For example, for 6-parameter affine prediction, interleaved prediction should not be used in the ME process.

For example, if the size of the current block satisfies certain conditions such as the following, interleaved prediction should not be applied to the ME process. Here, assume that the width and height of the current block are W and H respectively, and T, T1, and T2 are integer values: W>=T1 and H>=T2; W<=T1 and H<=T2; W>=T1 Or H>=T2; W<=T1 or H<=T2; W+H>=T

W+H<=T

W×H>=T

W×H<=T

For example, if the current block is split from a parent block, and the parent block does not select affine mode at the encoder, interleaving prediction is omitted in the ME process.

Figure 19 is a flowchart representation of a method 1900 for improving block-based motion prediction in a video system. Method 1900 includes, at operation 1902, determining a prediction block for the video block using a first intermediate prediction block and a second intermediate prediction block based on characteristics of the video block. Method 1900 includes, at operation 1904, generating a codec representation of the video block using the prediction block. A first intermediate prediction block is generated by subdividing the video block into a first set of sub-blocks, and a second intermediate prediction block is generated by subdividing the video block into a second set of sub-blocks. At least one sub-block in the second set has a different size than the sub-block in the first set.

In some embodiments, the determination is made during the motion estimation stage of the encoding process. In some embodiments, the characteristics of the video block indicate that the prediction block is not determined based on affine prediction.

T1且H

T2，T1和T2為預定整數值。在一些實施例中，一個或多個條件包含W

T1且H

T2，T1和T2為預定整數值。在一些實施例中，一個或多個條件包含W

T1或H

T2，T1和T2為預定整數值。在一些實施例中，一個或多個條件包含W

T1或H

T2，T1和T2為預定整數值。在一些實施例中，一個或多個條件包含W+H

T，T為預定整數值。在一些實施例中，一個或多個條件包含W+H

T，T為預定整數值。在一些實施例中，一個或多個條件包含W×H

T，T為預定整數值。在一些實施例中，一個或多個條件包含 W×H

T，T為預定整數值。 In some embodiments, the block has a width of W and a height of H, and the characteristics of the video block indicate that the prediction block is determined because W and H do not satisfy one or more conditions. In some embodiments, one or more conditions include W

T1 and H

T2, T1 and T2 are predetermined integer values. In some embodiments, one or more conditions include W

T1 and H

T2, T1 and T2 are predetermined integer values. In some embodiments, one or more conditions include W

T1 or H

T2, T1 and T2 are predetermined integer values. In some embodiments, one or more conditions include W

T1 or H

T2, T1 and T2 are predetermined integer values. In some embodiments, one or more conditions include W+H

T, T is a predetermined integer value. In some embodiments, one or more conditions include W+H

T, T is a predetermined integer value. In some embodiments, one or more conditions include W×H

T, T is a predetermined integer value. In some embodiments, one or more conditions include W×H

T, T is a predetermined integer value.

In some embodiments, the characteristics of the video block indicate that the video block is not split from a parent block. In some embodiments, the characteristics of the video block indicate that the block is partitioned from an affine-coded parent block.

Alternatively, if the current block is split from a parent block and the parent block does not select affine mode at the encoder, then no affine mode is checked at the encoder.

20 is a block diagram illustrating an example video processing system 2000 that may implement various techniques disclosed herein. Various implementations may include some or all components of system 2000. System 2000 may include an input 2002 for receiving video content. Video content may be received in raw or uncompressed format, such as 8 or 10-bit multi-component pixel values, or may be in compressed or encoded format. Input 2002 can represent a network interface, peripheral bus interface, or storage interface. Examples of network interfaces include wired interfaces such as Ethernet, Passive Optical Network (PON), etc. and wireless interfaces such as Wi-Fi or cellular interfaces.

System 2000 may include a codec component 2004 that may implement the various codecs or encoding methods described in this document. Codec component 2004 may reduce the average bitrate of the video from input 2002 to the output of codec component 2004 to produce a codec representation of the video. Therefore, codec technology is sometimes called video compression or video transcoding technology. The output of the codec component 2004 may be stored or transmitted via connected communications, as represented by component 2006 . The stored or communicated bitstream (or codec) representation of the video received at input 2002 may be used by component 2008 to generate pixel values or displayable video that is sent to display interface 2010 . from bit stream table The process of generating user-viewable video is sometimes called video decompression. Additionally, while certain video processing operations are referred to as "encoding" operations or tools, it is understood that the codec tools or operations are used at the encoder and that the corresponding decoding tools or operations that reverse the codec results would be performed by the decoder .

Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), or Display Port (Displayport). Examples of storage interfaces include SATA (Serial Advanced Technology Attachment), PCI, IDE interfaces, etc. The techniques described in this document may be implemented in a variety of electronic devices, such as mobile phones, laptop computers, smartphones, or other devices capable of performing digital data processing and/or video display.

It will be understood from the foregoing that specific embodiments of the disclosed technology have been described herein for purposes of illustration, but that various modifications may be made without departing from the scope of the invention. Accordingly, the technology disclosed in the present invention is not limited except for the scope of the appended claims.

The disclosed and other embodiments, modules, and functional operations described in this document may be implemented in digital electronic circuits, or in computer software, firmware, or hardware, including the structures disclosed in this document and their structural equivalents, or a combination of one or more of them. The disclosed and other embodiments may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium for execution by or to control data processing equipment. operation. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a storage device, a combination of substances that effect a machine-readable propagated signal, or or a combination of one or more of them. The term "data processing apparatus" includes all equipment, apparatus and machines for processing data, including for example a programmable processor, a computer or a plurality of processors or computers. In addition to hardware, the device may contain code that creates the execution environment for the computer program in question, such as code that makes up the processor's firmware, a protocol stack, a database management system, an operating system, or one or more of them. a combination of. Propagated signaling is an artificially generated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a suitable receiver device.

A computer program (also called a program, software, software application, script, or code) may be written in any form of programming language, including a compiled or interpreted language, and it may be deployed in any form, including as a stand-alone program or as suitable in a computing environment Modules, components, subroutines or other units used in. Computer programs do not necessarily correspond to files in a file system. Programs may be stored as part of a file that holds other programs or data (for example, one or more scripts stored in a markup language file), in a single file dedicated to the program in question, or in multiple coordinated In a file (for example, a file that stores one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one computer or on multiple computers located at a site or distributed across multiple sites and interconnected by a communications network.

The processes and logic flows described in this document may be executed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. Procedures and logic flows can also be represented by dedicated Logic circuits (eg, FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit)) are implemented, and the device may also be implemented as an application specific logic circuit.

Processors suitable for the execution of a computer program include, for example, general and special purpose microprocessors, and any one or more processors of any type of digital computer. Typically, a processor will receive instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also contain or be operably coupled to one or more mass storage devices (such as magnetic, magneto-optical, or optical disks) for storing data or to receive data from or to one or more mass storage devices. Transfer to one or more bulk device devices. However, computers do not require such equipment. Computer-readable media suitable for storage of computer program instructions and data includes all forms of non-volatile storage, media, and storage devices, including, for example, semiconductor storage devices, such as EPROM, EEPROM, and flash memory; magnetic disks, such as internal Hard or removable disks; magneto-optical disks; and, CD ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into special purpose logic circuitry.

Although this patent document contains many details, these details should not be construed as limitations on the scope of any invention or that may be claimed, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features that are described in this patent document in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, despite the above special Features may be described as functioning in certain combinations and even initially claimed as such, but in some cases one or more features from the claimed combination may be removed from the combination and the claimed combination may be sub- Variations of combinations or subcombinations.

Similarly, although operations are depicted in the drawings in a specific order, this should not be understood to imply that achieving desirable results requires that such operations be performed in the specific order shown, or in sequence, or that all illustrated operations are performed. Furthermore, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

Only a few implementations and examples are described, and other implementations, enhancements, and changes may be made based on what is described and illustrated in this patent document.

1500:Method

1502: Select a collection of pixels from a video frame to form a block

1504: Subdivide the block into a first set of sub-blocks according to the first style

1506: Generate a first intermediate prediction block based on the first set of sub-blocks

1508: Subdivide the block into a second set of sub-blocks according to the second style

1510: Generate a second intermediate prediction block based on the second set of sub-blocks

1512: Determine the prediction block based on the first intermediate prediction block and the second intermediate prediction block.

Claims (7)

A method of processing video, comprising: using a first intermediate prediction block and a second intermediate prediction block to determine a prediction block of the video block based on characteristics of the video block; and using the prediction block to generate a codec of the video block, wherein the first intermediate prediction block is generated by subdividing the video block into a first set of sub-blocks, wherein the second intermediate prediction block is generated by subdividing the video block into a second set of sub-blocks, and wherein at least one sub-block in the second set has a different size than the sub-block in the first set, wherein the video block has a width of W and a height of H, and wherein the characteristics of the video block Indicates that the prediction block is determined because W and H do not satisfy one or more conditions, and wherein the one or more conditions include: W

T1 and H

T2, T1 and T2 are predetermined integer values; W

T1 or H

T2, T1 and T2 are predetermined integer values; W

T1 or H

T2, T1 and T2 are predetermined integer values; W+H

T, T is a predetermined integer value; W+H

T, T is a predetermined integer value; or, W×H

T, T is a predetermined integer value. A method as claimed in claim 1, wherein said determination is performed during the motion estimation stage of the encoding process. The method of claim 1, wherein characteristics of the video block indicate that the prediction block is not determined based on affine prediction. The method of claim 1 or 2, wherein the characteristics of the video block indicate that the video block is not split from a parent block. The method of claim 1 or 2, wherein the characteristics of the video block indicate that the video block is split from an affine coded parent block. A device in a video system, including a processor and a non-transitory storage having instructions thereon, wherein when the instructions are executed by the processor, the processor is caused to implement items 1 to 5 of the patent application scope. any of the methods described. A non-transitory computer-readable medium comprising the computer code stored thereon, the computer program code being used to execute the method described in any one of items 1 to 5 of the patent application.

Images