CN111010580B

CN111010580B - Size limitation based on motion information

Info

Publication number: CN111010580B
Application number: CN201910951478.5A
Authority: CN
Inventors: 刘鸿彬; 张莉; 张凯; 王悦
Original assignee: Beijing ByteDance Network Technology Co Ltd; ByteDance Inc
Current assignee: Beijing ByteDance Network Technology Co Ltd; ByteDance Inc
Priority date: 2018-10-06
Filing date: 2019-10-08
Publication date: 2023-08-18
Anticipated expiration: 2039-10-08
Also published as: CN111010570A; WO2020070730A2; WO2020070729A1; WO2020070730A3; CN111010580A; CN111010570B

Abstract

Apparatus, systems, and methods for digital video encoding are described that include size limitations based on motion information. An exemplary method for video encoding includes: for a transition between an encoding unit of a video and a bitstream representation of the video, determining a sub-block size of a sub-block of the encoding unit based on motion information associated with the encoding unit; and performing the conversion based on the determined sub-block size.

Description

Size limitation based on motion information

Cross Reference to Related Applications

The present application is for the purpose of timely claiming priority and benefit of international patent application No. pct/CN2018/109255 filed on 10/6 2018, in accordance with applicable patent laws and/or rules in accordance with paris convention. The entire contents of the foregoing patent application are incorporated by reference as part of the disclosure of this patent document for all purposes in accordance with the relevant law.

Technical Field

This patent document relates to video coding techniques, apparatuses, and systems.

Background

Despite advances in video compression technology, digital video still occupies the largest bandwidth usage on the internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth requirements for digital video usage are expected to continue to increase.

Disclosure of Invention

Devices, systems, and methods related to digital video coding, and in particular to memory bandwidth reduction of sub-block coding tools in video coding, are described. The described methods may be applied to existing video coding standards (e.g., high efficiency video coding (High Efficiency Video Coding, HEVC)) and future video coding standards or video codecs.

In one representative aspect, the disclosed techniques may be used to provide a method for video processing. The method comprises the following steps: for a transition between an encoding unit of a video and a bitstream representation of the video, determining a sub-block size of a sub-block of the encoding unit based on motion information associated with the encoding unit; and performing the conversion based on the determined sub-block size.

In another representative aspect, the disclosed techniques may be used to provide a method for video processing. The method comprises the following steps: for a transition between an encoding unit of a video comprising sub-blocks and a bitstream representation of the video, determining whether to use a storage bandwidth reduction tool for the transition based on a size of the encoding unit; and performing the conversion based on the determination.

In another representative aspect, the disclosed techniques may be used to provide a method for video processing. The method comprises the following steps: for a transition between an encoding unit of video and a bitstream representation of video in affine mode, a sub-block size is determined based on information associated with the encoding unit. The method further includes performing a conversion in the affine mode based on the sub-block size.

In another representative aspect, the disclosed techniques may be used to provide a method for video encoding. The method comprises the following steps: for a bitstream representation of a current block, selecting a sub-block size based on motion information associated with the current block; and performing conversion between the bitstream representation and the current block based on the sub-block size.

In another representative aspect, the above-described methods are embodied in the form of processor-executable code and stored in a computer-readable program medium.

In yet another representative aspect, an apparatus configured or operable to perform the above-described method is disclosed. The apparatus may include a processor programmed to implement the method.

In yet another representative aspect, a video decoder device may implement the methods as described herein.

The above and other aspects and features of the disclosed technology are described in more detail in the accompanying drawings, description and claims.

Drawings

Fig. 1 shows an example of constructing a Merge candidate list.

Fig. 2 shows an example of the location of spatial candidates.

Fig. 3 shows an example of candidate pairs for performing redundancy check of the spatial Merge candidate.

Fig. 4A and 4B illustrate examples of the location of a second Prediction Unit (PU) based on the size and shape of a current block.

Fig. 5 shows an example of motion vector scaling for a temporal Merge candidate.

Fig. 6 shows an example of candidate locations of the time domain Merge candidates.

Fig. 7 shows an example of generating combined bi-predictive Merge candidates.

Fig. 8 shows an example of constructing a motion vector prediction candidate.

Fig. 9 shows an example of motion vector scaling for spatial motion vector candidates.

Fig. 10 shows an example of motion prediction using an alternative temporal motion vector prediction (Alternative Temporal Motion Vector Prediction, ATMVP) algorithm for Coding Units (CUs).

Fig. 11 shows an example of a Coding Unit (CU) with sub-blocks and neighboring blocks used by the Spatial-motion vector prediction (STMVP) algorithm.

Fig. 12 shows an example flow chart for encoding with different MV precision.

Fig. 13A and 13B show example snapshots of sub-blocks when using an overlapped block motion compensation (Overlapped Block Motion Compensation, OBMC) algorithm.

Fig. 14 shows an example of neighboring samples used to derive parameters for the local illumination compensation (Local Illumination Compensation, LIC) algorithm.

Fig. 15 shows an example of a simplified affine motion model.

Fig. 16 shows an example of an affine motion vector field (Affine Motion Vector Field, MVF) for each sub-block.

Fig. 17 shows an example of motion vector prediction (Motion Vector Prediction, MVP) of an AF INTER affine motion mode.

Fig. 18A and 18B show examples of a 4-parameter affine model and a 6-parameter affine model, respectively.

Fig. 19A and 19B show example candidates of the af_merge affine motion mode.

Fig. 20 shows an example of bilateral matching in a pattern-matched motion vector derivation (Pattern Matched Motion Vector Derivation, PMMVD) pattern, which is a special mere pattern based on the Frame-rate up-conversion (Frame-Rate Up Conversion, FRUC) algorithm.

Fig. 21 shows an example of template matching in a FRUC algorithm.

Fig. 22 shows an example of single sided motion estimation in a FRUC algorithm.

FIG. 23 shows an example of an optical flow trace used by the bi-directional optical flow (bi-directional optical flow, BIO) algorithm.

FIGS. 24A and 24B illustrate example snapshots using a bi-directional optical flow (BIO) algorithm without block expansion.

Fig. 25 shows an example of Decoder-side motion vector refinement (Decoder-side Motion Vector Refinement, DMVR) algorithm based on bilateral template matching.

Fig. 26 shows an example of a reference rectangular region of an encoding unit.

Fig. 27 shows an example of motion vector constraints in affine mode.

Fig. 28 shows a flow chart of an example method for video encoding.

Fig. 29 is a block diagram of an example of a hardware platform for implementing the visual media decoding or visual media encoding techniques described in this document.

FIG. 30 is a block diagram of an example video processing system in which the disclosed techniques may be implemented.

Fig. 31 is a flowchart representation of a method for video processing in accordance with the present technique.

Fig. 32 is a flow chart representation of another method for video processing in accordance with the present technique.

Fig. 33 is a flow chart representation of yet another method for video processing in accordance with the present technique.

Detailed Description

Video coding methods and techniques are ubiquitous in modern technology due to the increasing demand for higher resolution video. Video codecs typically include electronic circuitry or software that compresses or decompresses digital video, and are continually being improved to provide higher coding efficiency. The video codec converts uncompressed video into a compressed format and vice versa. There are complex relationships between video quality, the amount of data used to represent the video (determined by the bit rate), the complexity of the encoding and decoding algorithms, the sensitivity to data loss and errors, the ease of editing, random access and end-to-end delay (latency). The compression format typically conforms to standard video compression specifications, such as the High Efficiency Video Coding (HEVC) standard (also known as h.265 or MPEG-H part 2), the universal video coding standard to be completed, or other current and/or future video coding standards.

Embodiments of the disclosed technology may be applied to existing video coding standards (e.g., HEVC, h.265) and future standards to improve compression performance. Chapter titles are used in this document to enhance the readability of the description and in no way limit the discussion or embodiments (and/or implementations) to individual chapters only.

Examples of inter prediction in HEVC/H.265

Video coding standards have improved significantly over the years and now offer, in part, high coding efficiency and support for higher resolutions. Recent standards such as HEVC and h.265 are based on hybrid video coding structures in which temporal prediction plus transform coding is utilized.

1.1 examples of prediction modes

Each inter-predicted PU (prediction unit) has motion parameters of one or two reference picture lists. In some embodiments, the motion parameters include a motion vector and a reference picture index. In other embodiments, inter predidc may also be used to signal the use of one of the two reference picture lists. In still other embodiments, the motion vector may be explicitly encoded as an increment (delta) relative to the predicted value.

When a CU is encoded in skip mode, one PU is associated with the CU and there are no significant residual coefficients, no encoded motion vector delta, or reference picture index. The Merge mode is specified so that the current PU's motion parameters, including spatial and temporal candidates, are obtained from neighboring PUs. The Merge mode may be applied to any inter-predicted PU, not just the skip mode. An alternative to the Merge mode is explicit transmission of motion parameters, where for each PU, motion vectors, corresponding reference picture indices for each reference picture list, and reference picture list usage are explicitly signaled.

When the signaling indicates that one of the two reference picture lists is to be used, a PU is generated from a block of one sample. This is called "uni-prediction". Unidirectional prediction may be used for both P-stripes and B-stripes.

When the signaling indicates that two reference picture lists are to be used, a PU is generated from two sample blocks. This is called "bi-prediction". Bi-prediction is only available for B slices.

1.1.1 embodiment of constructing candidates for Merge mode

When predicting a PU using the Merge mode, an index to an entry in the Merge candidate list is parsed from the bitstream and used to retrieve motion information. The construction of this list can be summarized according to the following sequence of steps:

step 1: initial candidate derivation

Step 1.1: spatial candidate derivation

Step 1.2: redundancy check of airspace candidates

Step 1.3: time domain candidate derivation

Step 2: additional candidate inserts

Step 2.1: creating bi-prediction candidates

Step 2.2: inserting zero motion candidates

Fig. 1 shows an example of constructing a Merge candidate list based on the above summarized sequence of steps. For spatial-domain Merge candidate derivation, a maximum of four Merge candidates are selected among the candidates located at five different positions. For time domain Merge candidate derivation, a maximum of one Merge candidate is selected among the two candidates. Since a constant number of candidates per PU is assumed at the decoder, additional candidates are generated when the number of candidates does not reach the maximum number of Merge candidates signaled in the slice header (MaxNumMergeCand). Since the number of candidates is constant, truncated unary binarization (Truncated Unary binarization, TU) is used to encode the index of the best Merge candidate. If the size of the CU is equal to 8, then all PUs of the current CU share a single Merge candidate list, which is the same as the Merge candidate list of the 2Nx2N prediction unit.

1.1.2 construction of airspace Merge candidates

In the derivation of the spatial-domain Merge candidates, up to four Merge candidates are selected among candidates located at the positions depicted in fig. 2. The deduced sequence is A ₁ 、B ₁ 、B ₀ 、A ₀ And B ₂ . Only when position A ₁ 、B ₁ 、B ₀ 、A ₀ Position B is only considered if any PU of (e.g. because it belongs to another slice or slice) is not available or is intra-coded ₂ . In addition position A ₁ After the candidates at this point, a redundancy check is performed on the addition of the remaining candidates, which ensures that candidates with the same motion information are excluded from the listIn addition, the coding efficiency is improved.

In order to reduce the computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. Instead, only the pairs joined by arrows in fig. 3 are considered, and candidates are added to the list only when the corresponding candidates for redundancy check have different motion information. Another source of duplicate motion information is a "second PU" associated with a partition other than 2nx2n. As an example, fig. 4A and 4B depict the second PU for cases nx2n and 2nxn, respectively. When the current PU is partitioned into N2N, position A ₁ Candidates at this point are not considered for list construction. In some embodiments, adding the candidate may result in two prediction units having the same motion information, which is redundant for having only one PU in the coding unit. Similarly, when the current PU is partitioned into 2N×N, position B is not considered ₁ 。

1.1.3 construction of time Domain Merge candidates

In this step, only one candidate is added to the list. In particular, in the derivation of the temporal Merge candidate, the scaled motion vector is derived based on the collocated PU belonging to the picture within the given reference picture list having the smallest POC difference from the current picture. The derived reference picture list to be used for the collocated PU is explicitly signaled in the slice header.

Fig. 5 shows an example of the derivation of a scaled motion vector of a temporal Merge candidate (as a dashed line) scaled from a motion vector of a collocated PU using POC distances tb and td, where tb is defined as the POC difference between a reference picture of the current picture and td is defined as the POC difference between a reference picture of the collocated picture and the collocated picture. The reference picture index of the temporal Merge candidate is set equal to zero. For the B slice, two motion vectors are obtained, one for reference picture list 0 and the other for reference picture list 1, and the two motion vectors are combined to obtain the bi-prediction Merge candidate.

In the collocated PU (Y) belonging to the reference frame, in the candidate C ₀ And C ₁ Between which the location of the time domain candidate is selected, as depicted in fig. 6. If position C ₀ The PU at is unavailable, intra-coded, or outside the current CTU, then position C is used ₁ . Otherwise, position C ₀ For the derivation of time domain mere candidates.

1.1.4 construction of additional types of Merge candidates

In addition to the space-time Merge candidates, there are two additional types of Merge candidates: combined bi-predictive Merge candidate and zero Merge candidate. A combined bi-predictive Merge candidate is generated by utilizing the space-time Merge candidate. The combined bi-predictive Merge candidate is only for the B stripe. A combined bi-prediction candidate is generated by combining the first reference picture list motion parameter of the initial candidate with the second reference picture list motion parameter of the other candidate. If the two tuples provide different motion hypotheses they will form new bi-prediction candidates.

Fig. 7 shows an example of this process, where two candidates with mvL0 and refIdxL0 or mvL1 and refIdxL1 in the original list (710 on the left) are used to create a combined bi-prediction Merge candidate that is added to the final list (720 on the right).

Zero motion candidates are inserted to fill the remaining entries in the Merge candidate list, thereby achieving MaxNumMergeCand capacity. These candidates have zero spatial displacement and a reference picture index that starts from zero and increases each time a new zero motion candidate is added to the list. For uni-directional and bi-directional prediction, the number of reference frames used by these candidates is one and two, respectively. In some embodiments, no redundancy check is performed on these candidates.

1.1.5 examples of motion estimation regions for parallel processing

In order to speed up the encoding process, motion estimation may be performed in parallel, thereby deriving motion vectors for all prediction units within a given region at the same time. Deriving Merge candidates from spatial neighborhood may interfere with parallel processing because one prediction unit cannot derive motion parameters from neighboring PUs until its associated motion estimation is complete. To mitigate the tradeoff between coding efficiency and processing latency, a motion estimation region (Motion Estimation Region, MER) may be defined. The size of the MER may be signaled in the picture parameter set (Picture Parameter Set, PPS) using a "log2_parameter_merge_level_minus2" syntax element. When defining MERs, mere candidates that fall into the same region are marked as unavailable and are therefore not considered in list construction.

1.2 example of advanced motion vector prediction (Advanced Motion Vector Prediction, AMVP)

AMVP exploits the spatial-temporal correlation of motion vectors with neighboring PUs, which is used for explicit transmission of motion parameters. It constructs a motion vector candidate list by first checking the availability of the left, upper side of temporally adjacent PU locations, removing redundant candidates and adding zero vectors to make the candidate list a constant length. The encoder may then select the best predictor from the candidate list and send a corresponding index indicating the selected candidate. Similar to the Merge index signaling, truncated unary is used to encode the index of the best motion vector candidate. The maximum value to be encoded in this case is 2 (see fig. 8). In the following section, details are provided regarding the derivation process of motion vector prediction candidates.

1.2.1 examples of constructing motion vector prediction candidates

Fig. 8 summarizes the derivation process of the motion vector prediction candidates, and may be implemented for each reference picture list with refidx as an input.

In motion vector prediction, two types of motion vector candidates are considered: spatial domain motion vector candidates and temporal motion vector candidates. For spatial domain motion vector candidate derivation, two motion vector candidates are ultimately derived based on the motion vector of each PU located at five different locations as previously shown in fig. 2.

For temporal motion vector candidate derivation, one motion vector candidate is selected from the two candidates, which is derived based on two different collocated positions. After the first list of space-time candidates is generated, duplicate motion vector candidates in the list are removed. If the number of potential candidates is greater than two, motion vector candidates within the associated reference picture list whose reference picture index is greater than 1 are removed from the list. If the number of space-time motion vector candidates is less than two, additional zero motion vector candidates are added to the list.

1.2.2 construction of spatial motion vector candidates

In the derivation of spatial motion vector candidates, a maximum of two candidates are considered among five potential candidates, which are derived from PUs located at positions as previously shown in fig. 2, those positions being the same as the positions of motion Merge. The derivation order of the left side of the current PU is defined as A ₀ 、A ₁ And scaled A ₀ Scaled A ₁ . Defining the derivation order of the upper side of the current PU as B ₀ 、B ₁ 、B ₂ Scaled B ₀ Scaled B ₁ Scaled B ₂ . Thus, for each side, there are four cases that can be used as motion vector candidates, two of which do not require the use of spatial scaling and two of which use spatial scaling. Four different cases are summarized as follows:

-non-spatial scaling

(1) Identical reference picture list and identical reference picture index (identical POC)

(2) Different reference picture lists, but the same reference picture (same POC)

-spatial domain scaling

(3) The same reference picture list, but different reference pictures (different POCs)

(4) Different reference picture lists, and different reference pictures (different POCs)

First, the case without spatial scaling is checked, and then the case where spatial scaling is allowed is checked. When POC is different between the reference picture of the neighboring PU and the reference picture of the current PU, spatial scaling is considered regardless of the reference picture list. If all PUs of the left candidate are not available or are intra coded, then the upper motion vector is allowed to be scaled to aid in the parallel derivation of the left and upper MV candidates. Otherwise, spatial scaling of the upper motion vector is not allowed.

As shown in the example in fig. 9, for the spatial scaling case, the motion vectors of neighboring PUs are scaled in a similar manner as the temporal scaling. One difference is to give as input the reference picture list and the index of the current PU; the actual scaling process is the same as the time domain scaling process.

1.2.3 construction of temporal motion vector candidates

All procedures for deriving temporal Merge candidates are the same as those for deriving spatial motion vector candidates (as shown in the example in fig. 6), except for reference picture index derivation. In some embodiments, the reference picture index is signaled to the decoder.

2. Examples of inter prediction methods in joint exploration model (Joint Exploration Model, JEM)

In some embodiments, future video coding techniques are explored using reference software known as Joint Exploration Model (JEM). In JEM, sub-block based predictions such as affine prediction, optional temporal motion vector prediction (ATMVP), spatial motion vector prediction (STMVP), bi-directional optical flow (BIO), frame rate up-conversion (FRUC), local adaptive motion vector resolution (Locally Adaptive Motion Vector Resolution, LAMVR), overlapped Block Motion Compensation (OBMC), local Illumination Compensation (LIC), and Decoder side motion vector refinement (Decoder-side Motion Vector Refinement, DMVR) are employed in several coding tools.

2.1 example of sub-CU based motion vector prediction

In a JEM with a quadtree plus binary tree (Quadtrees Plus Binary Trees, QTBT), each CU may have at most one set of motion parameters for each prediction direction. In some embodiments, two sub-CU level motion vector prediction methods are considered in the encoder by dividing the large CU into sub-CUs and deriving motion information for all sub-CUs of the large CU. An optional temporal motion vector prediction (ATMVP) method allows each CU to extract multiple sets of motion information from multiple blocks smaller than the current CU in the collocated reference picture. In the space-time motion vector prediction (STMVP) method, a motion vector of a sub-CU is recursively derived by using a time-domain motion vector predictor and a spatial-domain neighboring motion vector. In some embodiments and in order to preserve a more accurate motion field for sub-CU motion prediction, motion compression of the reference frame may be disabled.

2.1.1 example of Alternative Temporal Motion Vector Prediction (ATMVP)

In the ATMVP method, a temporal motion vector prediction (Temporal Motion Vector Prediction, TMVP) method is modified by extracting a plurality of sets of motion information (including a motion vector and a reference index) from a block smaller than a current CU.

Fig. 10 shows an example of an ATMVP motion prediction procedure of the CU 1000. The ATMVP method predicts the motion vector of the sub-CU 1001 within the CU 1000 in two steps. The first step is to identify the corresponding block 1051 in the reference picture 1050 using the temporal vector. The reference picture 1050 is also referred to as a motion source picture. The second step is to divide the current CU 1000 into sub-CUs 1001 and obtain a motion vector and a reference index of each sub-CU from a block corresponding to each sub-CU.

In a first step, the reference picture 1050 and corresponding block are determined from motion information of spatial neighboring blocks of the current CU 1000. To avoid the repeated scanning process of neighboring blocks, the first Merge candidate in the Merge candidate list of the current CU 1000 is used. The first available motion vector and its associated reference index are set as the temporal vector and the index of the motion source picture. In this way, the corresponding block (sometimes referred to as a collocated block) may be more accurately identified than the TMVP, with the corresponding block always being in a lower right or center position relative to the current CU.

In a second step, by adding a temporal vector to the coordinates of the current CU, the corresponding block of the sub-CU 1051 is identified by the temporal vector in the motion source picture 1050. For each sub-CU, the motion information of its corresponding block (e.g., the smallest motion grid covering the center sample) is used to derive the motion information of the sub-CU. After identifying the motion information of the corresponding nxn block, it is converted into a motion vector and a reference index of the current sub-CU in the same manner as TMVP of HEVC, where motion scaling and other processes are applicable. For example, the decoder checks whether a low delay condition (e.g., POC of all reference pictures of the current picture is smaller than POC of the current picture) is satisfied and possibly uses the motion vector MV _x Predicting motion vectors MV for each sub-CU (e.g., motion vectors corresponding to reference picture list X) _y (e.g., X equals 0 or 1, Y equals 1-X).

2.1.2 example of spatial motion vector prediction (STMVP)

In the STMVP method, motion vectors of sub-CUs are recursively deduced in raster scan order. Fig. 11 shows an example of one CU and neighboring blocks with four sub-blocks. Consider an 8 x 8CU 1100 that includes four 4 x 4 sub-CUs a (1101), B (1102), C (1103), and D (1104). Adjacent 4 x 4 blocks in the current frame are labeled a (1111), b (1112), c (1113), and d (1114).

The motion derivation of sub-CU a begins by identifying its two spatial neighbors. The first neighbor is an n×n block (block c 1113) on the upper side of the sub CU a (1101). If this block c (1113) is not available or intra-coded, the other nxn blocks on the upper side of sub-CU a (1101) are checked (starting from block c 1113, left to right). The second neighbor is the block to the left of sub-CU a1101 (block b 1112). If block b (1112) is not available or intra coded, then other blocks on the left side of sub-CU a1101 are checked (starting from block b 1112, top to bottom). The motion information obtained from neighboring blocks of each list is scaled to the first reference frame of the given list. Next, a temporal motion vector prediction value (Temporal Motion Vector Predictor, TMVP) of the sub-block a1101 is derived by following the same procedure as that of TMVP derivation specified in HEVC. The motion information of the collocated block at block D1114 is extracted and scaled accordingly. Finally, after retrieving and scaling the motion information, all available motion vectors are averaged for each reference list, respectively. The average motion vector is assigned as the motion vector of the current sub-CU.

2.1.3 examples of sub-CU motion prediction mode Signaling

In some embodiments, the sub-CU mode is enabled as an additional Merge candidate and no additional syntax elements are needed to signal the mode. Two additional Merge candidates are added to the Merge candidate list for each CU to represent ATMVP mode and STMVP mode. In other embodiments, up to seven Merge candidates are used if the sequence parameter set indicates that ATMVP and STMVP are enabled. The coding logic of the additional Merge candidate is the same as the Merge candidate in the HM, which means that for each CU in the P-slice or B-slice, two further RD checks may be required for two additional Merge candidates. In some embodiments, all binary bits, such as JEM, merge index, are context coded by CABAC (context-based adaptive binary arithmetic coding). In other embodiments, for example, HEVC, only the first binary bit is context coded and the remaining binary bits are context bypass coded.

2.2 examples of adaptive motion vector difference resolution

In some embodiments, when use_integer_mv_flag in the slice header is equal to 0, the motion vector difference (Motion Vector Difference, MVD) between the motion vector of the PU and the predicted motion vector) is signaled in quarter luma samples. In JEM, locally Adaptive Motion Vector Resolution (LAMVR) is introduced. In JEM, MVDs may be encoded in units of quarter-luminance samples, integer-luminance samples, or four-luminance samples. The MVD resolution is controlled at the Coding Unit (CU) level and a MVD resolution flag is conditionally signaled for each CU having at least one non-zero MVD component.

For a CU with at least one non-zero MVD component, a first flag is signaled to indicate whether quarter-luma sample MV precision is used in the CU. When the first flag (equal to 1) indicates that the quarter-luma sample MV precision is not used, another flag is signaled to indicate whether integer-luma sample MV precision or four-luma sample MV precision is used.

When the first MVD resolution flag of a CU is zero or is not coded for the CU (meaning that all MVDs in the CU are zero), the quarter luma sample MV resolution is used for the CU. When the CU uses integer luminance sample MV precision or four luminance sample MV precision, the MVPs in the AMVP candidate list of the CU are rounded to the corresponding precision.

In the encoder, a CU level RD check is used to determine which MVD resolution is to be used for the CU. That is, three CU level RD checks are performed for each MVD resolution. In order to accelerate the encoder speed, the following coding scheme is applied in JEM:

during RD checking of a CU with conventional quarter-luma sample MVD resolution, the motion information (integer luma sample precision) of the current CU is stored. The stored motion information (after rounding) is used as a starting point for further small-range motion vector refinement during RD checking of the same CU with integer luminance samples and 4 luminance samples MVD resolution, so that the time-consuming motion estimation process is not repeated three times.

-conditionally invoking RD checking of CUs with 4 luma samples MVD resolution. For a CU, when the RD cost of the integer luminance sample MVD resolution is much greater than the RD cost of the quarter luminance sample MVD resolution, the RD check of the 4 luminance sample MVD resolution of the CU is skipped.

The encoding process is shown in fig. 12. First, 1/4 pixel MVs are tested, and RD costs are calculated and expressed as RDCost0, then integer MVs are tested and RD costs are expressed as RDCost1. If RDCost1< th RDCost0 (where th is a positive threshold), then 4 pixels MV are tested; otherwise skip 4 pixels MV. Basically, when checking integer or 4-pixel MVs, motion information, RD costs, etc. are already known for 1/4-pixel MVs, which can be reused to speed up the encoding process of integer or 4-pixel MVs.

2.3 example of higher motion vector storage precision

In HEVC, motion vector precision is one-quarter pixel (for 4:2:0 video, one-quarter luma samples and one-eighth chroma samples). In JEM, the accuracy of the internal motion vector store and the Merge candidate is increased to 1/16 pixel. Higher motion vector precision (1/16 pixel) is used for motion compensated inter prediction of CU encoded in skip/Merge mode. For CUs encoded in conventional AMVP mode, integer-pixel or quarter-pixel motion is used.

An SHVC upsampling interpolation filter having the same filter length and normalization factor as the HEVC motion compensation interpolation filter is used as the motion compensation interpolation filter for the additional fractional pixel positions. In JEM the chrominance component motion vector accuracy is 1/32 samples, and the additional interpolation filter for the 1/32 pixel fractional position is derived by using the average of the filters for two adjacent 1/16 pixel fractional positions.

2.4 example of Overlapped Block Motion Compensation (OBMC)

In JEM, the syntax at the CU level can be used to turn on and off OBMC. When OBMC is used in JEM, OBMC is performed on all motion compensated (Motion Compensation, MC) block boundaries except for the right and bottom boundaries of the CU. Furthermore, it applies to both luminance and chrominance components. In JEM, MC blocks correspond to coded blocks. When a CU is encoded with sub-CU modes (including sub-CU Merge, affine, and FRUC modes), each sub-block of the CU is a MC block. To process CU boundaries in a unified way, OBMC is performed at a sub-block level for all MC block boundaries, where the sub-block size is set equal to 4 x 4, as shown in fig. 13A and 13B.

Fig. 13A shows the sub-blocks at the CU/PU boundary, and the shadow sub-blocks are the locations of the OBMC application. Similarly, fig. 13B shows child Pus in ATMVP mode.

When OBMC is applied to the current sub-block, the motion vectors of the four connected neighboring sub-blocks (if available and different from the current motion vector) are also used to derive the prediction block of the current sub-block, in addition to the current motion vector. These multiple prediction blocks based on multiple motion vectors are combined to generate a final prediction signal for the current sub-block.

The prediction block based on the motion vector of the neighboring sub-block is denoted as PN, where N indicates indexes of neighboring upper, lower, left, and right sub-blocks, and the prediction block based on the motion vector of the current sub-block is denoted as PC. When the PN is based on the motion information of the neighboring sub-block containing the same motion information as the current sub-block, the OBMC is not performed from the PN. Otherwise, each sample of PN is added to the same sample in the PC, i.e., four rows/columns of PN are added to the PC. The weighting factors {1/4,1/8,1/16,1/32} are used for PN and the weighting factors {3/4,7/8,15/16,31/32} are used for PC. The exception is a small MC block (i.e., when the height or width of the coded block is equal to 4 or the CU is coded with a sub-CU mode) for which only two rows/columns of PN are added to the PC. In this case, the weighting factor {1/4,1/8} is used for P _N And weighting factors 3/4,7/8 are used for PC. For PN generated based on motion vectors of vertically (horizontally) adjacent sub-blocks, samples in the same row (column) of PN are added to PCs with the same weighting factors.

In JEM, for CUs of size less than or equal to 256 luma samples, a CU level flag is signaled to indicate whether OBMC is applied to the current CU. For CUs that are over 256 luma samples in size or are not encoded using AMVP mode, OBMC is applied by default. At the encoder, when OBMC is applied to the CU, its effect is taken into account during the motion estimation phase. The prediction signal formed by the OBMC using the motion information of the top and left neighboring blocks is used to compensate the top and left boundaries of the original signal of the current CU, and then a conventional motion estimation process is applied.

2.5 example of Local Illumination Compensation (LIC)

The LIC is based on a linear model for illumination variation using a scaling factor a and an offset b. And an encoding unit (CU) encoded for each inter mode adaptively enables or disables it.

When LIC is applied to a CU, a least squares error method is employed to derive parameters a and b by using neighboring samples of the current CU and their corresponding reference samples. Fig. 14 shows an example of neighboring samples used to derive parameters of an IC algorithm. More specifically and as shown in fig. 14, sub-sampled (2:1 sub-sampled) neighboring samples of a CU and corresponding samples (identified by motion information of the current CU or sub-CU) in a reference picture are used. IC parameters are derived and applied to each prediction direction separately.

When a CU is encoded with the Merge mode, the LIC flag is copied from the neighboring block in a similar manner to the motion information copy in the Merge mode; otherwise, an LIC flag is signaled to the CU to indicate whether LIC is applied.

When LIC is enabled for pictures, an additional CU level RD check is needed to determine whether LIC is applied to the CU. When LIC is enabled for CU, the sum of the de-Mean absolute differences (Mean-Removed Sum of Absolute Difference, MR-SAD) and the sum of the de-Mean absolute hadamard transform differences (Mean-Removed Sum of Absolute Hadamard-Transformed Difference, MR-SATD) are used instead of SAD and SATD, respectively for integer-pixel motion search and fractional-pixel motion search.

In order to reduce coding complexity, the following coding scheme is applied in JEM: -disabling LIC for the whole picture when there is no significant illumination change between the current picture and its reference picture. To identify this situation, a histogram of the current picture and each reference picture of the current picture is calculated at the encoder. Disabling the LIC for the current picture if the histogram difference between the current picture and each reference picture of the current picture is less than a given threshold; otherwise, LIC is enabled for the current picture.

2.6 example of affine motion compensated prediction

In HEVC, only translational motion models are applied to motion compensated prediction (Motion Compensation Prediction, MCP). However, the camera and object may have a variety of movements, such as zoom in/out, rotation, perspective movement, and/or other irregular movement. JEM, on the other hand, applies reduced affine transformation motion compensated prediction. FIG. 15 shows a block 1500 with two control point motion vectors V ₀ And V ₁ Examples of affine motion fields are described. The Motion Vector Field (MVF) of block 1500 can be described by the following equation:

as shown in fig. 15, (v) _0x ,v _0y ) Is the motion vector of the upper left corner control point, and (v _1x ,v _1y ) Is the motion vector of the upper right corner control point. To simplify motion compensated prediction, sub-block based affine transformation prediction may be applied. The subblock size mxn is derived as follows:

here, mvPre is the motion vector score accuracy (e.g., 1/16 in JEM). (v) _2x ,v _2y ) Is the motion vector of the lower left corner control point, calculated according to equation (1). If desired, M and N can be adjusted downward to be divisors of w and h, respectively.

Fig. 16 shows an example of affine MVF for each sub-block of block 1600. To derive the motion vector for each mxn sub-block, the motion vector for the center sample for each sub-block may be calculated according to equation (1) and rounded to a motion vector score accuracy (e.g., 1/16 in JEM). A motion compensated interpolation filter may then be applied to generate a prediction for each sub-block with the derived motion vector. After MCP, the high precision motion vector of each sub-block is rounded and saved with the same precision as the conventional motion vector.

2.6.1 Examples of AF_INTER modes

In JEM, there are two affine motion modes: af_inter mode and af_merge mode. For CUs with width and height both greater than 8, the af_inter mode may be applied. Affine flags in the CU level are signaled in the bitstream to indicate whether AF INTER mode is used. In the AF INTER mode, neighboring blocks are used to construct a block having a pair of motion vectors { (v) ₀ ,v ₁ )|v ₀ ＝{v _A ,v _B ,v _c },v ₁ ＝{v _D ,v _E Candidate list of }.

Fig. 17 shows an example of Motion Vector Prediction (MVP) of a block 1700 in the AF INTER mode. As shown in fig. 17, v is selected from the motion vector of sub-block A, B or C ₀ . The motion vectors from neighboring blocks may be scaled according to the reference list. The motion vector may also be scaled according to a relationship between a Picture Order Count (POC) of a reference of the neighboring block, a POC of a reference of the current CU, and a POC of the current CU. Selecting v from adjacent sub-blocks D and E ₁ Is similar. If the number of candidate lists is less than 2, the list is populated with motion vector pairs composed by copying each AMVP candidate. When the candidate list is greater than 2, the candidates may first be ordered according to neighboring motion vectors (e.g., based on similarity to two of the candidate (pair candidates)). In some embodiments, the first two candidates are retained. In some embodiments, a Rate Distortion (RD) cost check is used to determine which motion vector pair candidate to select as the control point motion vector prediction (Control Point Motion Vector Prediction, CPMVP) of the current CU. An index indicating the location of the CPMVP in the candidate list may be signaled in the bitstream. After determining the CPMPP of the current affine CU, affine motion estimation is applied and a control point motion vector (Contr) is found ol Point Motion Vector, CPMV). The difference between CPMV and CPMVP is then signaled in the bitstream.

In the AF INTER mode, when a 4/6 parameter affine mode is used, 2/3 control points are required, and thus 2/3 MVDs need to be encoded for these control points, as shown in fig. 18A and 18B. In existing embodiments, MV may be derived, for example, from mvd ₀ Predicting mvd ₁ And mvd ₂ 。

In the present context,mvd _i and mv ₁ Is the predicted motion vector, and the motion vector difference and the motion vector of the upper left pixel (i=0), the upper right pixel (i=1), or the lower left pixel (i=2) are shown in fig. 18B, respectively. In some embodiments, the addition of two motion vectors (e.g., mvA (xA, yA) and mvB (xB, yB)) is equal to the sum of the two components individually. For example, newmv=mva+mvb implies that the two components of newMV are set to (xa+xb) and (ya+yb), respectively.

2.6.2 Examples of fast affine ME algorithm in AF INTER mode

In some embodiments of affine mode, it is desirable to jointly determine MVs for 2 or 3 control points. Direct joint searching of multiple MVs is computationally complex. In an example, a fast affine ME algorithm is proposed and incorporated into a VTM/BMS.

For example, a fast affine ME algorithm is described for a 4-parametric affine model, and the idea can be extended to a 6-parametric affine model.

Replacing (a-1) with a' enables the motion vector to be rewritten as:

if it is assumed that the motion vectors of the two control points (0, 0) and (0,w) are known, the affine parameters can be derived according to equation (5):

the motion vector may be rewritten in vector form as:

here, p= (x, y) is the pixel position.

In some embodiments, and at the encoder, the MVD of af_inter may be iteratively derived. MV to be measured ⁱ (P) is expressed as MV derived in the ith iteration of position P, and dMV _C ⁱ Represented as MV in the ith iteration _C Is updated in the update process. Then at (i+1) th timeIn the course of the iteration of the method,

pic is processed _ref Denoted as reference picture, and Pic _cur Expressed as the current picture, and expressed as q=p+mv ⁱ (P). If MSE is used as the matching criterion, the function that needs to be minimized can be written as:

if it is assumed thatSmall enough, then->Can be rewritten as an approximation based on taylor expansion of order 1 as follows:

in the present context,if the symbol E is adopted ⁱ⁺¹ (P)＝Pic _cur (P)-Pic _ref (Q), then:

itemsBy setting the derivative of the error function to zero and then according to +.>The incremental MVs for control points (0, 0) and (0,w) are calculated to derive as follows: />

In some embodiments, the MVD derivation process may be iterated n times, and the final MVD may be calculated as follows:

In the foregoing embodiment, the first embodiment is described as the second embodiment by mvd ₀ Incremental MV prediction of control point (0, 0) represented by mvd ₁ The increment MV of the control point (0,w) represented causes the error for mvd ₁ Only Is encoded.

2.6.3 Examples of AF_MERGE mode

When a CU is applied in af_merge mode, it gets the first block encoded in affine mode from the valid neighboring reconstructed blocks. Fig. 19A shows an example of a selection order of candidate blocks of the current CU 1900. As shown in fig. 19A, the selection order may be from the left side (1901), the upper side (1902), the upper right side (1903), the lower left corner (1904), to the upper left side (1905) of the current CU 1900. Fig. 19B shows another example of a candidate block for the current CU 1900 in the af_merge mode. If the neighboring lower left corner block 1901 is encoded in affine mode, as shown in FIG. 19B, the motion vectors v of the upper left, upper right and lower left corners of the CU including the sub-block 1901 are derived ₂ 、v ₃ And v ₄ . Based on v ₂ 、v ₃ And v ₄ Calculation of motion vector v for the upper left corner of current CU 1900 ₀ . The motion vector v1 on the upper right side of the current CU may be calculated accordingly.

After calculating the CPMV of the current CUs v0 and v1 according to the affine motion model in equation (1), the MVF of the current CU may be generated. In order to identify whether the current CU is encoded in the af_merge mode, an affine flag may be signaled in the bitstream when there is at least one neighboring block encoded in the affine mode.

2.7 example of motion vector derivation (PMMVD) for pattern matching

The PMMVD mode is a special Merge mode based on the Frame Rate Up Conversion (FRUC) method. With this mode, the motion information of the block is pushed on the decoder side, instead of signaling the motion information of the block.

When the Merge flag of the CU is true, the FRUC flag may be signaled to the CU. When the FRUC flag is false, the Merge index may be signaled and the regular Merge mode may be used. When the FRUC flag is true, an additional FRUC mode flag may be signaled to indicate which method (e.g., bilateral matching or template matching) to use to derive motion information for the block.

At the encoder side, the decision as to whether or not to use FRUC Merge mode for the CU is based on RD cost selection made for the regular Merge candidate. For example, multiple matching patterns (e.g., bilateral matching and template matching) are checked for CUs by using RD cost selection. The matching pattern that results in the least cost is further compared to other CU patterns. If the FRUC match pattern is the most efficient pattern, then the FRUC flag is set to true for the CU and the relevant match pattern is used.

In general, the motion derivation process in FRUC Merge mode has two steps: CU-level motion search is performed first, and then sub-CU-level motion refinement is performed. At the CU level, an initial motion vector is derived for the entire CU based on bilateral matching or template matching. First, a list of MV candidates is generated and the candidate that results in the smallest matching cost is selected as the starting point for further CU level refinement. Then, a local search based on bilateral matching or template matching is performed near the start point. The MV result that gives the smallest matching cost is taken as the MV of the entire CU. Subsequently, the motion information is further refined at the sub-CU level, starting from the derived CU motion vector.

For example, the following derivation procedure is performed for w×h CU motion information derivation. In the first stage, MVs for the entire W×H CU are derived. In the second stage, the CU is further divided into mxm sub-CUs. The value of M is calculated as in equation (3), D being a predefined division depth, default to 3 in JEM. The MV for each sub-CU is then derived.

Fig. 20 shows an example of bilateral matching used in the Frame Rate Up Conversion (FRUC) method. The bilateral matching is used to derive the motion information of the current CU by finding the closest match between the two blocks along the motion trajectory of the current CU (2000) in two different reference pictures (2010, 2011). Under the assumption of a continuous motion trajectory, motion vectors MV0 (2001) and MV1 (2002) pointing to two reference blocks are proportional to temporal distances (e.g., TD0 (2003) and TD1 (2004)) between the current picture and the two reference pictures. In some embodiments, when the current picture 2000 is temporally between two reference pictures (2010, 2011) and the temporal distance from the current picture to the two reference pictures is the same, the bilateral matching becomes a mirror-based bi-directional MV.

Fig. 21 shows an example of template matching used in the Frame Rate Up Conversion (FRUC) method. Template matching may be used to derive motion information for the current CU 2100 by finding the closest match between a template in the current picture (e.g., the top and/or left neighboring block of the current CU) and a block in the reference picture 2110 (e.g., the same size as the template). Template matching may also be applied to AMVP mode in addition to FRUC Merge mode described previously. In both JEM and HEVC, AMVP has two candidates. Using the template matching method, new candidates can be derived. If the template matching newly derived candidate is different from the first existing AMVP candidate, it is inserted at the very beginning of the AMVP candidate list, and then the list size is set to two (e.g., by removing the second existing AMVP candidate). When applied to AMVP mode, only CU level search is applied.

The MV candidate set at CU level may include the following: (1) if the current CU is in AMVP mode, then the original AMVP candidate, (2) all Merge candidates, (3) several MVs in the interpolated MV field (described later), and (4) top and left neighboring motion vectors.

When bilateral matching is used, each valid MV of the Merge candidate may be used as input to generate MV pairs in the case of hypothetical bilateral matching. For example, at reference list a, one valid MV of the Merge candidate is (MVa, ref _a ). Then find the reference picture ref of its paired bilateral MV in another reference list B _b So that ref _a And ref _b Located on different sides of the current picture in the temporal domain. If such ref _b Not available in reference list B, ref _b Is determined to be equal to ref _a Different references, and its temporal distance to the current picture is the minimum in list B. At determination of ref _b Thereafter, by basing on the current picture and ref _a 、ref _b The temporal distance between them scales MVa to derive MVb.

In some embodiments, four MVs from the interpolated MV field may also be added to the CU-level candidate list. More specifically, interpolation MVs at the locations (0, 0), (W/2, 0), (0, H/2) and (W/2, H/2) of the current CU are added. When FRUC is applied in AMVP mode, the original AMVP candidates are also added to the CU-level MV candidate set. In some embodiments, at the CU level, 15 MVs of the AMVP CU and 13 MVs of the Merge CU may be added to the candidate list.

The MV candidate set at the sub-CU level includes:

(1) The determined MV is searched from the CU level,

(2) Top, left side, upper left corner and upper right corner adjacent MVs,

(3) Scaled versions of collocated MVs from reference pictures,

(4) One or more ATMVP candidates (e.g., up to four), and

(5) One or more STMVP candidates (e.g., up to four).

Scaled MVs from reference pictures are derived as follows. The reference pictures in both lists are traversed. The MV at the collocated position of the sub-CU in the reference picture is scaled to the reference of the starting CU level MV. ATMVP and STMVP candidates may be the first four. At the sub-CU level, one or more MVs (e.g., up to 17) are added to the candidate list.

And generating an interpolation MV field.Before encoding a frame, an interpolated motion field is generated for the entire picture based on the single side ME. The motion field may then later be used as a CU level or sub-CU level MV candidate.

In some embodiments, the motion field of each reference picture in the two reference lists is traversed at a 4 x 4 block level. Fig. 22 shows an example of a single-sided motion estimation (Motion Estimation, ME) 2200 in a FRUC method. For each 4 x 4 block, if the motion associated with the block passes through the 4 x 4 block in the current picture and the block is not assigned any interpolation motion, the motion of the reference block is scaled to the current picture according to temporal distances TD0 and TD1 (in the same manner as the MV scaling of TMVP in HEVC), and the scaled motion is assigned to the block in the current frame. If no scaled MVs are assigned to a 4X 4 block, the motion of the block is marked as unusable in the interpolation motion field.

Interpolation and matching costs.When the motion vector points to a fractional sample point, motion compensated interpolation is required. To reduce complexity, bilinear interpolation may be used for bilateral matching and template matching instead of conventional 8-tap HEVC interpolation.

The computation of the matching costs is somewhat different at the different steps. When selecting candidates from the candidate set at the CU level, the matching cost may be the sum-absolute-difference (Absolute Sum Difference, SAD) of bilateral matching or template matching. After determining the starting MV, the matching cost C for bilateral matching at the sub-CU level search is calculated as follows:

here, w is a weighting factor. In some embodiments, w may be empirically set to 4.MV and MV ^s Indicating the current MV and the starting MV, respectively. The SAD can still be used as a matching cost for template matching at the sub-CU level search.

In FRUC mode, MV is derived only by using luminance samples. The derived motion will be used for both luminance and chrominance of the MC inter prediction. After the MV is decided, the final MC is performed using an 8-tap interpolation filter for luminance and a 4-tap interpolation filter for chrominance.

MV refinement is a pattern-based MV search, with criteria of bilateral matching cost or template matching cost. In JEM, two search modes, unrestricted center biased diamond search (Unrestricted Center-Biased Diamond Search, UCBDS) and adaptive cross search, are supported for MV refinement at the CU level and at the sub-CU level, respectively. For both CU and sub-CU level MV refinement, MVs are searched directly with quarter luma sample MV precision, and then eighth luma sample MV refinement. The search range for MV refinement for the CU and sub-CU steps is set equal to 8 luma samples.

In bilateral matching Merge mode, bi-prediction is applied because the motion information of a CU is derived based on the closest match between two blocks along the current CU's motion trajectory in two different reference pictures. In template matching Merge mode, the encoder may select for the CU among unidirectional prediction from list 0, unidirectional prediction from list 1, or bi-prediction. The selection may be based on template matching costs as follows:

if cosbi < = factor min (cost 0, cost 1)

Then bi-directional prediction is used;

otherwise, if cost0< = cost1

Then unidirectional prediction from list 0 is used;

otherwise the first set of parameters is selected,

using unidirectional predictions from list 1;

here, cost0 is the SAD of the list 0 template match, cost1 is the SAD of the list 1 template match, and cost bi is the SAD of the bi-prediction template match. For example, when the factor value is equal to 1.25, this means that the selection process is biased towards bi-prediction. Inter prediction direction selection may be applied to the CU level template matching process.

2.8 examples of bidirectional optical flow (BIO)

The bi-directional optical flow (BIO) method is a sample-wise motion refinement performed on top of block-wise motion compensation for bi-directional prediction. In some embodiments, the sample level motion refinement does not use signaling.

Suppose I ^(k) Is the luminance value from reference k (k=0, 1) after block motion compensation, andandrespectively represent I ^(k) A horizontal component and a vertical component of the gradient. Assuming that the optical flow is valid, the motion vector field (v _x ,v _y ) The equation is given by:

combining the optical flow equation with Hermite interpolation of each sample point motion trail to obtain a unique third-order polynomial, wherein the third-order polynomial finally matches the function value I ^(k) And derivatives thereofAnd->The value of the third order polynomial at t=0 is the BIO prediction:

FIG. 23 shows an example of optical flow trajectories in a bi-directional optical flow (BIO) method. Here, τ ₀ And τ ₁ Representing the distance to the reference frame. POC calculation distance τ based on Ref0 and Ref1 ₀ And τ ₁ ：τ ₀ =poc (current) -POC (Ref 0), τ ₁ =poc (Ref 1) -POC (current). If both predictions are from the same time domain direction (both from the past or both from the future), sign is different (e.g., τ ₀ ·τ ₁ < 0). In this case, if the predictions are not from the same time instant (e.g., τ ₀ ≠τ ₁ ) BIO is applied. Both reference regions have non-zero motion (e.g., MVx ₀ ,MVy ₀ ,MVx ₁ ,MVy ₁ Not equal to 0) and the block motion vector is proportional to the temporal distance (e.g., MVx ₀ /MVx ₁ ＝MVy ₀ /MVy ₁ ＝-τ ₀ /τ ₁ )。

Motion vector field (vx, v) is determined by minimizing the difference delta between the values in points a and B _y ). Fig. 9 shows an example of the intersection of a motion trajectory and a reference frame plane. The model uses only the first linear term of the local taylor expansion of Δ:

all values in the above equation depend on the sample position denoted (i ', j'). Assuming that the motion is uniform in the locally surrounding area, delta can be minimized inside a square window Ω of (2m+1) x (2m+1) centered on the current predicted point (i, j), where M equals 2:

for this optimization problem, JEM uses a simplified method, with minimization first in the vertical direction and then in the horizontal direction. The results are shown below:

/>

wherein, the liquid crystal display device comprises a liquid crystal display device,

to avoid division by zero or very small values, regularization parameters r and m may be introduced in equations (28) and (29), where

r＝500·4 ^d-8 Equation (31)

m＝700·4 ^d-8 Equation (32)

Where d is the bit depth of the video samples.

In order to keep the memory access of BIO the same as conventional bi-predictive motion compensation, all prediction and gradient values I are calculated for positions inside the current block ^(k) 、Fig. 24A shows an example of an access location outside of block 2400. As shown in FIG. 24A, in equation [ ]30 A (2m+1) x (2m+1) square window Ω centered on the current prediction point on the boundary of the prediction block needs to access the location outside the block. In JEM, I is external to the block ^(k) 、The value of (c) is set equal to the nearest available value inside the block. This may be implemented, for example, as filling region 2401, as shown in fig. 24B.

With BIO, the motion field can be refined for each sample. To reduce computational complexity, block-based BIO designs are used in JEM. Motion refinement may be calculated based on 4 x 4 blocks. In block-based BIO, s in equation (30) for all samples in a 4×4 block can be aggregated _n Then s _n Used to derive the BIO motion vector offset for a 4 x 4 block. More specifically, the following formula may be used for block-based BIO derivation:

here b _k Representing the set of samples belonging to the kth 4 x 4 block of the prediction block. S in equation (28) and equation (29) _n Replaced by ((s) _n,bk )>>4) To derive an associated motion vector offset.

In some scenarios, the MV group (MV region) of the BIO may be unreliable due to noise or irregular motion. Thus, in BIO, the amplitude of the MV group is clipped to a threshold value. The threshold is determined based on whether the reference pictures of the current picture are all from one direction. For example, if all the reference pictures of the current picture come from one direction, the value of the threshold is set to 12×2 ^14-d The method comprises the steps of carrying out a first treatment on the surface of the Otherwise, it is set to 12×2 ^13-d 。

The gradient of the BIO may be calculated simultaneously with motion compensated interpolation using operations consistent with the HEVC motion compensation process (e.g., 2D separable finite impulse response (Finite Impulse Response, FIR)). In some embodiments, the input to the 2D separable FIR is a motion compensation process and motion from blockThe fractional positions (fracX, fracY) of the fractional parts of the vector are the same reference frame samples. For horizontal gradientsFirst a bisfiltered s vertical interpolation signal corresponding to the fractional position fracY with the descaled offset d-8 is used. A gradient filter bisafiltg is then applied in the horizontal direction, which bisafiltg corresponds to the fractional position fracX with the descaled offset 18-d. For vertical gradientsThe gradient filter is applied vertically using a BIOfileG corresponding to the fractional position fracY with the descaled offset d-8. Signal shifting is then performed in the horizontal direction using a bisfilter s corresponding to the fractional position fracX with the descaled offset 18-d. To maintain reasonable complexity, the length of the interpolation filter for gradient computation, bisafiltg, and for signal displacement, bisafiltf, may be shorter (e.g., 6 taps). Table 1 shows example filters that may be used for gradient computation for different fractional positions of block motion vectors in BIO. Table 2 shows an example interpolation filter that may be used for prediction signal generation in BIO.

Table 1: exemplary Filter for gradient computation in BIO

Fractional pixel location	Gradient interpolation filter (BIOfilterG)
		0	{8,-39,-3,46,-17,5}
1/16	{8,-32,-13,50,-18,5}
		1/8	{7,-27,-20,54,-19,5}
3/16	{6,-21,-29,57,-18,5}
		1/4	{4,-17,-36,60,-15,4}
5/16	{3,-9,-44,61,-15,4}
		3/8	{1,-4,-48,61,-13,3}
7/16	{0,1,-54,60,-9,2}
		1/2	{-1,4,-57,57,-4,1}

Table 2: exemplary interpolation Filter for prediction Signal Generation in BIO

Fractional pixel location	Interpolation filter of prediction signal (BIOfilter S)
		0	{0,0,64,0,0,0}
1/16	{1,-3,64,4,-2,0}
		1/8	{1,-6,62,9,-3,1}
3/16	{2,-8,60,14,-5,1}
		1/4	{2,-9,57,19,-7,2}
5/16	{3,-10,53,24,-8,2}
		3/8	{3,-11,50,29,-9,2}
7/16	{3,-11,44,35,-10,3}
		1/2	{3,-10,35,44,-11,3}

In JEM, BIO can be applied to all bi-predictive blocks when the two predictions come from different reference pictures. When Local Illumination Compensation (LIC) is enabled for a CU, BIO may be disabled.

In some embodiments, OBMC is applied to the block after a conventional MC process. To reduce computational complexity, no BIO may be applied during the OBMC process. This means that the BIO is applied to the MC process of the block when its own MV is used, and is not applied to the MC process when MVs of neighboring blocks are used during the OBMC process.

2.9 example of decoder-side motion vector refinement (DMVR)

In the bi-prediction operation, for prediction of one block region, two prediction blocks formed using a Motion Vector (MV) of list 0 and a MV of list 1, respectively, are combined to form a single prediction signal. In the Decoder-side motion vector refinement (DMVR) method, two motion vectors of bi-prediction are further refined by a bilateral template matching process. Bilateral template matching is applied in the decoder to perform a distortion-based search between the bilateral template and reconstructed samples in the reference picture in order to obtain refined MVs without sending additional motion information.

In DMVR, the bilateral template is generated as a weighted combination (i.e., average) of two prediction blocks from list 0's initial MV0 and list 1's MV1, respectively, as shown in fig. 25. The template matching operation includes calculating a cost measure (cost measure) between the generated template and a sample region (around the initial prediction block) in the reference picture. For each of the two reference pictures, consider the MV that results in the smallest template cost as the updated MV for the list to replace the original MV. In JEM, nine MV candidates are searched for each list. The nine MV candidates include an original MV and 8 surrounding MVs having one luminance sample offset in the horizontal or vertical direction or both directions from the original MV. Finally, two new MVs, i.e., MV0 'and MV1' as shown in fig. 25, are used to generate the final bi-prediction result. The Sum of Absolute Differences (SAD) is used as the cost measure.

DMVR is applied to the Merge mode of bi-prediction, where one MV comes from a past reference picture and another MV comes from a future reference picture without sending additional syntax elements. In JEM, DMVR is not applied when LIC, affine motion, FRUC, or sub-CU Merge candidates are enabled for the CU.

3. Disadvantages of the prior embodiments

In sub-block based coding tools like ATMVP, STMVP and affine mode, smaller sub-blocks typically bring higher coding gain, albeit at the cost of higher memory bandwidth. On the other hand, if larger sub-blocks are used directly to reduce the storage bandwidth, a small coding performance penalty may result.

In some implementations of affine mode, motion vectors are derived for each 4×4 block and motion compensation is also performed at the 4×4 block level, which may increase storage bandwidth compared to 4×8, 8×4, or 8×8 blocks.

In the existing implementation, restrictions on affine patterns are presented for worst-case bandwidth reduction. To ensure that the worst case bandwidth of an affine block is not worse than 4 x 8 or 8 x 4 blocks, the motion vector difference between affine control points is used to decide whether the sub-block size of the affine block is 4 x 4 or 8 x 8.

The memory bandwidth reduction of affine patterns is controlled by limiting the motion vector differences (also called control point differences) between affine control points. In general, if the control point difference satisfies the following constraint as in (34), the affine motion uses 4×4 sub-blocks (i.e., 4×4 affine patterns). Otherwise, it uses 8×8 sub-blocks (8×8 affine pattern). The 6-parametric model and the 4-parametric model are limited as follows.

6-constraint of parametric affine model:

in the above equation, the left side represents a reduction in span (span) level of the sub-affine block, (7/2) factor indicates 3.5 pixel offset, and the Norm () function is a normalization of the motion vector difference based on (w×h). In some embodiments, norm () is defined as:

/>

4-constraint of parametric affine model:

in the 4-parametric affine model, (v) _2x -v _0x ) Sum (v) _2y -v _0y ) Is set as follows:

in this case, (v) _2x -v _0x ) Sum (v) _2y -v _0y ) Is the Norm:

by applying (36) to (34), constraints of the 4-parametric affine model are established.

In another prior embodiment, each 8 x 8 block within a block is considered a base unit. The MVs of all four 4 x 4 sub-blocks inside an 8 x 8 block are constrained so that the maximum difference between the integer parts of the four 4 x 4 sub-blocks MVs does not exceed 1 pixel. Let the bandwidth be (8+7+1)/(8×8) =4 samples/pixel.

Both implementations can reduce the worst-case bandwidth to less than 4 x 8 or 8 x 4, but can result in a significant coding performance penalty.

4. Exemplary embodiments of affine inter modes

In some embodiments, the storage bandwidth problem in affine models may be solved by analyzing the region of interest in the reference picture of the affine model. As shown in fig. 26, the reference pixels at the four corners of the CU define a minimum rectangular area that can cover all the reference pixels of the CU. Assuming that the sizes of CU and its rectangular reference region are w×h and m×n, respectively, if (m+7) ×n+7)/(w×h), for example, the average storage bandwidth is smaller than a given value, pixels required for once interpolating the reference region in the outer rectangle 2603 shown in fig. 26 may be loaded, but different sub-blocks of the CU may be interpolated with different MVs.

In affine mode, motion vectors of corner pixels are actually derived for sub-blocks comprising corner pixels (corner pixels), as described in detail below.

For the 6-parameter model as shown in equation (38), the motion vectors of the four corner pixels are derived by replacing (x, y) with (SW/2, SH/2), (W-SW/2, SH/2), (SW/2, H-SH/2), (W-SW/2, H-SH/2) into equation (38), where w×h and sw×sh are CU size and sub-block size, respectively. The motion vectors of the four corner pixels are shown in equations (39) to (42), and 0, 1, 2, and 3 refer to the upper left corner, the upper right corner, the lower left corner, and the lower right corner, respectively.

/>

With motion vectors, the position of the reference pixel (upper left corner relative to the current CU) can be calculated as follows:

the horizontal and vertical distances between any two reference pixels of the four corner pixels are listed in table 3. It can be seen that the horizontal/vertical distance between P0 and P1 is the same as the horizontal/vertical distance between P2 and P3, and that the horizontal/vertical distance between P0 and P2 is the same as the horizontal/vertical distance between P1 and P3. The maximum horizontal and vertical distances are expressed in horMax and verMax, and the rectangular reference area is horMax. For any given affine model parameter, a rectangular reference region of the CU can be calculated and the required storage bandwidth determined.

Table 3: exemplary distance between different reference pixels

	Horizontal distance	Vertical distance
			(P0,P1)	\|a*(W-SW)+W-1\|	\|e*(W-SW)\|
(P0,P2)	\|b*(H-SH)\|	\|d*(H-SH)+H-1\|
			(P0,P3)	\|a(W-SW)+b(H-SH)+W-1\|	\|e(W-SW)+d(H-SH)+H-1\|
(P1,P2)	\|a(W-SW)-b(H-SH)+W-1\|	\|-e(W-SW)+d(H-SH)+H-1\|
			(P1,P3)	\|b*(H-SH)\|	\|d*(H-SH)+H-1\|
(P2,P3)	\|a*(W-SW)+W-1\|	\|e*(W-SW)\|

Similarly, for the 4-parameter affine model in equation (47), equations (39) to (46) and e and f in table 3 are replaced with-b and a.

5. Example methods for memory bandwidth reduction in a sub-block coding tool

Embodiments of the disclosed technology overcome the shortcomings of existing implementations, thereby providing higher coding efficiency for video coding. The following examples described for various embodiments illustrate that memory bandwidth reduction in sub-block coding tools based on the disclosed techniques may enhance existing and future video coding standards. The examples of the disclosed technology provided below illustrate general concepts and are not meant to be construed as limiting. In the examples, various features described in these examples may be combined unless explicitly indicated to the contrary.

Although examples are described in the context of Coding Units (CUs), they may also be applied to blocks, prediction units (PDUs), etc. Further, for prediction directions 0 and 1, the sizes of the reference rectangular areas of the CU are denoted as m0×n0 and m1×n1, respectively, and the sizes of the CU are denoted as w×h.

Example 1 It is proposed that the sub-block size used in ATMVP/STMVP may depend on the motion information (e.g. prediction direction, motion vector, reference picture, etc.) of the CU (or current block)。

(a) In one example, a sub-block with a smaller size may be used for unidirectional prediction, and a sub-block with a larger size may be used for bidirectional prediction. For example, if the current CU is uni-directionally predicted, a 4×4, 4×8, or 8×4 sub-block is used. Alternatively, if the current CU is bi-predictive, an 8 x 8 sub-block is used.

(b) In one example, if the current CU is uni-directionally predicted, an mxn sub-block is used when the CU height is greater than the CU width, and an nxm sub-block is used in other cases. N is greater than or equal to M, such as m=4 and n=8.

(i) Alternatively, if the current CU is uni-directionally predicted, an mxn sub-block is used when the CU height is greater than the CU width, and an nxm sub-block is used in other cases. M is greater than or equal to N, such as m=8 and n=4.

(c) In one example, if W < =n and H < =m, e.g., n=m=8, ATMVP/STMVP is disabled for a CU of size w×h.

(i) Alternatively, if W < =n or H < =m, e.g., n=m=8, ATMVP/STMVP is disabled for a CU of size w×h.

(ii) Alternatively, if W > =n and H > =m, e.g., n=m=64, ATMVP/STMVP is disabled for a CU of size w×h.

(iii) Alternatively, if W > =n or H > =m, e.g., n=m=64, ATMVP/STMVP is disabled for a CU of size w×h.

(d) In one example, MVs of different sub-blocks are constrained to be within a given range around one or more selected MVs.

(i) In one example, the selected MV is the MV of the center sub-block.

(ii) In one example, the selected MV is the MV of the upper left-hand sub-block.

Example 2It is proposed that the sub-block size used in affine mode may depend on affine mode parameters, prediction direction, shape of reference pictures and CUs.

(a) In one example, the number of filler pixels for each dimension of the reference rectangular area of the CU is plen 1 and plen 2. For bi-predictive CUs, if (m0+flen1) × (n0+flen2) + (m1+flen1) × (n1+flen2) is less than or equal to TH1×w×h, then subblocks sw1×sh1 are used; otherwise, the sub-block SW2×sh2 is used.

(i) In some embodiments, for a uni-directionally predicted CU, if (m0+flen1) × (n0+flen2) is less than or equal to th2×w×h, then subblocks sw1×sh1 are used; otherwise, the sub-block SW2×sh2 is used.

(ii) In some embodiments, for bi-predictive CUs, if (m0+flen1) × (n0+flen2) is greater than TH3×w×h, or if (m1+flen1) × (n1+flen2) is greater than TH3×w×h, then sub-blocks sw2×sh2 are used; otherwise, the sub-block SW1×sh1 is used.

(iii) In some embodiments, for bi-predicted CUs, different sub-block sizes may be used for different prediction directions. If (m0+flen1) × (n0+flen2) is greater than TH4×w×h, then for prediction direction 0, sub-block sw2×sh2 is used, otherwise sub-block sw1×sh1 is used. If (m1+flen1) × (n1+flen2) is greater than TH4×w×h, then for prediction direction 1, sub-block sw2×sh2 is used, otherwise sub-block sw1×sh1 is used.

(iv) In one example, the size of the sub-block may depend on the location of the current block. For example, the size of the sub-blocks may be different when the current block is at or not at Coding Tree Unit (CTU) boundaries.

(v) In one example, SW1 and SH1 are equal to 4, and SW2 and SH2 are equal to 8. Alternatively, SW1×sh1 is equal to 4×8 or 8×4. Alternatively, SW1 and SH1 may depend on the shape of CU. For example, for a CU having a width greater than or equal to the height, sw1×sh1 is set equal to 8×4, and for other CUs, sw1×sh1 is set equal to 4×8.

(vi) The FLEN1 and FLEN2 may be set equal to the length of the interpolation filter, or alternatively, may be set equal to the length of the interpolation filter plus/minus N, e.g., N is equal to 1. Alternatively, if the horizontal MV component has integer MV precision, then flute 1 is set to zero, and if the vertical MV component has integer MV precision, then flute 2 is set to zero.

(vii) TH1, TH2 and TH3 are positive floating point numbers.

(1) In one example, TH2 is set equal to 10.3125 (165/16), and TH1, TH3, and TH4 are set equal to 5.15625 (165/32).

(2) In one example, TH2 is set equal to 7.03125 (225/32), and TH1, TH3, and TH4 are set equal to 3.515625 (225/64).

(b) In one example, the sub-blocks SW1×sh1 are always used, but MVs of each sub-block may be constrained. As shown in fig. 27, CU is first divided into sw2×sh2 (greater than sw1×sh1) sub-blocks, and then a motion vector (denoted as MV) of the sw2×sh2 sub-block is derived using a center position _C ) Finally deriving the MVs of each SW1 XSH 1 sub-block within the SW2 XSH 2 block and limiting the MVs to MVs _C Within a given range around. For bi-predictive CUs, such processing is performed in each prediction direction.

(i) In one example, SW1×sh1 is equal to 4×4, and SW2×sh2 is equal to 8×8. The motion vectors of the upper left, upper right, lower left and lower right 4 x 4 sub-blocks are denoted as MV, respectively ₀ 、MV ₁ 、MV ₂ 、MV ₃ . For each prediction direction X (x=0 or 1), abs (MV _i LX[0]-MV _C LX[0]) Limited to be smaller than Rhor _i And will abs (MV _i LX[1]-MV _C LX[1]) Limited to less than Rver _i Wherein Rhor is a kind of _i And Rver _i Is a positive floating point number, and for different i (i=0, 1, 2 or 3), they may be different, and MV _i LX[0]And MV (sum MV) _i LX[1]Respectively MV _i The horizontal and vertical components of LX.

(ii) In one example, SW1×sh1 is equal to 4×4, and SW2×sh2 is equal to 4×8. The motion vectors of the top and bottom 4 x 4 sub-blocks are denoted as MV, respectively ₀ And MV (sum MV) ₁ . For each prediction direction X (x=0 or 1), abs (MV _i LX[0]-MV _C LX[0]) Limited to be smaller than Rhor _i And will abs (MV _i LX[1]-MV _C LX[1]) Limited to less than Rver _i Wherein Rhor is a kind of _i And Rver _i Is a positive floating point number, and for different i (i=0 or 1), they may be different, and MV _i LX[0]And MV (sum MV) _i LX[1]Respectively MV _i The horizontal and vertical components of LX.

(iii) In one example, SW1×sh1 is equal to 4×4, and SW2×sh2 is equal to 8×4. The motion vectors of the left and right 4 x 4 sub-blocks are denoted as MV, respectively ₀ And MV (sum MV) ₁ . For each prediction direction X (x=0 or 1), abs (MV _i LX[0]-MV _C LX[0]) Limited to be smaller than Rhor _i And will abs (MV _i LX[1]-MV _C LX[1]) Limited to less than Rver _i Wherein Rhor is a kind of _i And Rver _i Is a positive floating point number, and for different i (i=0 or 1), they may be different, and MV _i LX[0]And MV (sum MV) _i LX[1]Respectively MV _i The horizontal and vertical components of LX.

(iv) In one example, MV _C MV set equal to one of the SW1 XSH 1 sub-blocks, e.g. MV _C Set equal to MV of the upper left corner sw1×sh1 sub-block.

(v)Rhor _i And Rver _i May depend on the prediction direction of the CU and the size of sw2×sh2, e.g., they may be larger for uni-directionally predicted CUs and smaller for bi-directionally predicted CUs; and for larger SW2 x SH2 they may be larger and vice versa.

(vi) In one example, when SW2 XSH 2 is equal to 8X 8/8X 4/4X 8 and SW1 XSH 1 is equal to 4X 4, rhor _i And Rver _i Are set equal to 1/2 pixel.

(vii) In one example, when SW2 XSH 2 is equal to 8X 8/8X 4/4X 8 and SW1 XSH 1 is equal to 4X 4, rhor _i And Rver _i Are set equal to 1/4 pixel.

(viii) In one example, when SW2 XSH 2 is equal to 8X 8/8X 4/4X 8 and SW1 XSH 1 is equal to 4X 4, rhor _i Is set equal to 1/2 pixel and Rver _i Is set equal to 1 pixel.

(ix) In one example, when SW2 XSH 2 is equal to 8X 8/8X 4/4X 8 and SW1 XSH 1 is equal to 4X 4, rhor _i Is set equal to 1/2 pixel and Rver _i Is set equal to 1/4 pixel.

Example 3When CU size is largeAt W _max ×H _max When it is possible to divide it into several smaller areas, and to process each area based on the technique described in example 2.

(a) In one example, when the CU size is 128×128, it is divided into 4 64×64 regions.

(b) In one example, when the CU size is 128×64 or 64×128, it is divided into 2 64×64 regions.

Example 4Whether or how the memory bandwidth reduction is applied on the sub-block coded CU may depend on the size of the CU.

(a) In one example, if the CU size is greater than w×h, the storage bandwidth reduction method is not performed.

(b) In one example, if the CU size is smaller than w×h, the storage bandwidth reduction method is not performed.

The examples described above may be incorporated in the context of a method described below, such as method 2800, which may be implemented at a video decoder or video encoder.

Fig. 28 shows a flow chart of an exemplary method for video encoding. The method 2800 includes, at step 2810, selecting a sub-block size based on motion information associated with the current block for a bitstream representation of the current block.

In some embodiments, and in the context of example 1, selecting the sub-block size comprises: selecting a first size if performing the conversion includes unidirectional prediction; and if performing the conversion includes bi-prediction, selecting a second size, and wherein the first size is smaller than the second size. In one example, the height of the current block is greater than the width of the current block, and the sub-block size is 4×8. In another example, the width of the current block is greater than the height of the current block, and the sub-block size is 8×4.

In some embodiments, wherein the sub-block size is mxn, wherein the current block size is W x H, and wherein performing the conversion does not include optional temporal motion vector prediction (ATMVP) and spatial motion vector prediction (STMVP). In one example, W is N and H is M, and n=m=8. In another example, w+.n or h+.m, and n=m=8. In yet another example, W is N and H is M, and n=m=64. In yet another example, w≡n or h≡m, and n=m=64.

In some embodiments, selecting the sub-block size is further based on one or more dimensions of the current block, and the motion information includes at least one of a prediction direction, a motion vector, or a reference picture. In other embodiments, performing the conversion is based on a motion vector of a center sub-block of the current block. In still other embodiments, performing the conversion is based on a motion vector of an upper left sub-block of the current block.

The method 2800 includes, at step 2820, performing a conversion between the bitstream representation and the current block based on the sub-block size. In some embodiments, the conversion generates the current block from the bitstream representation (e.g., as may be implemented in a video decoder). In other embodiments, the conversion generates a bitstream representation from the current block (e.g., as may be implemented in a video encoder).

In some embodiments, and in the context of example 2, performing the conversion is based on affine patterns, and wherein selecting the sub-block size is further based on one or more affine model parameters.

In some embodiments, and in the context of example 2, performing the conversion is based on an affine mode, wherein the size of the current block is w×h, wherein the size of the reference rectangular region of the current block is m0×n0 for the first prediction direction, wherein the size of the reference rectangular region is m1×n1 for the second prediction direction, and wherein the number of filler pixels per dimension of the reference rectangular region is plen 1 and plen 2, respectively.

In one scenario, performing the conversion includes bi-prediction, wherein selecting the sub-block size includes: selecting a first size if (m0+flen1) × (n0+flen2) + (m1+flen1) × (n1+flen2) is less than or equal to th1×wxh; otherwise, the second size is selected, and where TH1 is a positive score. For example, TH1 is 165/32 or 225/64.

In another scenario, performing the conversion includes unidirectional prediction, wherein selecting the sub-block size includes: selecting a first size if (m0+flen1) × (n0+flen2) is less than or equal to th2×wxh; otherwise, the second size is selected, and where TH2 is a positive score. For example, TH2 is 165/16 or 225/32.

In yet another scenario, performing the conversion includes bi-prediction, wherein selecting the sub-block size includes: selecting a first size if (m0+flen1) × (n0+flen2) or (m1+flen1) × (n1+flen2) is greater than TH3×w×h; otherwise, the second size is selected, and where TH3 is a positive score. For example, TH3 is 165/32 or 225/64.

In yet another scenario, performing the conversion includes bi-prediction, and different sub-block sizes are used for different prediction directions. For example, selecting the sub-block size includes: if (M0+FLEN1) × (N0+FLEN2) is less than or equal to TH4 XWXH, then for the first prediction direction, selecting a first size; otherwise, the second size is selected, and where TH4 is a positive score. Further, selecting the sub-block size further includes: if (M1+FLEN1) × (N1+FLEN2) is less than or equal to TH4 XWXH, then for the second prediction direction, selecting the first size; otherwise, the second size is selected. In an example, TH4 is 165/32 or 225/64.

In the above exemplary scenario, the first size may be 4×4 or 4×8 or 8×4, and the second size may be 8×8.

In some embodiments, performing the conversion is based on a center position of a sub-block of size sw2×sh2 and includes bi-prediction, and wherein the sub-block of size sw2×sh2 includes a plurality of sub-blocks of size sw1×sh1.

In some embodiments, an absolute difference between a first motion vector of a sub-block of size SW2 x SH2 and a second motion vector of one of a plurality of sub-blocks of size SW1 x SH1 is less than a predetermined threshold. In one example, SW2×sh2 is 8×8 or 8×4 or 4×8, sw1×sh1 is 4×4, and the predetermined threshold is 1/2 pixel. In another example, SW2×sh2 is 8×8 or 8×4 or 4×8, sw1×sh1 is 4×4, and the predetermined threshold is 1/4 pixel.

In some embodiments, and in the context of example 3, the current block is 128 x 128 in size, and performing the conversion includes dividing the current block into four 64 x 64 blocks and performing the conversion for each of the four 64 x 64 blocks.

In some embodiments, the current block is 128×64 or 64×128 in size, and performing the conversion includes dividing the current block into two 64×64 blocks and performing the conversion for each of the two 64×64 blocks.

In some embodiments, and for the above embodiments, the current block may be a coding unit or a prediction unit.

6. Example implementations of the disclosed technology

Fig. 29 is a block diagram of a video processing apparatus 2900. The apparatus 2900 may be used to implement one or more of the methods described herein. The apparatus 2900 may be embodied in a smart phone, tablet, computer, internet of things (Internet of Things, ioT) receiver, or the like. The apparatus 2900 may include one or more processors 2902, one or more memories 2904, and video processing hardware 2906. Processor(s) 2902 may be configured to implement one or more methods described in this document, including but not limited to method 2800. Memory(s) 2904 may be used to store data and code for implementing the methods and techniques described herein. Video processing hardware 2906 may be used to implement some of the techniques described in this document in hardware circuitry.

In some embodiments, the video encoding method may be implemented using an apparatus implemented on a hardware platform as described with respect to fig. 29.

Fig. 30 is a block diagram illustrating an example video processing system 3000 in which various techniques disclosed herein may be implemented. Various embodiments may include some or all of the components of system 3000. The system 3000 may include an input 3002 for receiving video content. The video content may be received in an original or uncompressed format, such as 8 or 10 bit multi-component pixel values, or may be in a compressed or encoded format. Input 3002 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as ethernet, passive optical network (Passive Optical Network, PON), etc., and wireless interfaces such as Wi-Fi or cellular interfaces.

The system 3000 can include an encoding component 3004 that can implement various encoding methods described in this document. The encoding component 3004 may reduce the average bit rate of the video from the input 3002 to the output of the encoding component 3004 to produce an encoded representation of the video. Coding techniques are therefore sometimes referred to as video compression or video transcoding techniques. The output of the encoding component 3004 may be stored or transmitted via a communication connection as represented by component 3006. The stored or communicatively transmitted bit stream (or encoded) representation of the video received at input 3002 may be used by component 3008 to generate pixel values or to transmit displayable video to display interface 3010. The process of generating user-viewable video from a bitstream representation is sometimes referred to as video decompression. Further, while certain video processing operations are referred to as "encoding" operations or tools, it will be appreciated that the encoding tools or operations are used at the encoder, and that the corresponding decoding tools or operations that reverse the encoding results will be performed by the decoder.

Examples of the peripheral bus interface or the display interface may include a universal serial bus (Universal Serial Bus, USB), or a high-definition multimedia interface (High Definition Multimedia Interface, HDMI), or a display port (Displayport), or the like. Examples of storage interfaces include SATA (Serial Advanced Technology Attachment ), PCI, IDE interfaces, and the like. The techniques described in this document may be embodied in various electronic devices such as mobile phones, laptops, smartphones, or other devices capable of performing digital data processing and/or video display.

Fig. 31 is a flowchart representation of a method 3100 for video processing in accordance with the present technique. The method 3100 includes, at operation 3110, determining, for a transition between an encoding unit of the video and a bitstream representation of the video, a sub-block size of a sub-block of the encoding unit based on motion information associated with the encoding unit. The method 3100 includes, at operation 3120, performing the conversion based on the determined sub-block size. In some embodiments, the conversion generates the coding unit from the bitstream representation. In some embodiments, the conversion generates a bitstream representation from the encoding unit.

In some embodiments, selecting the sub-block size includes selecting a first size as the sub-block size in the case of converting the coding unit including unidirectional prediction, and selecting a second size in the case of converting the coding unit including bidirectional prediction. The first dimension may be smaller than the second dimension. In some embodiments, in the case of converting a coding unit comprising unidirectional prediction, the first size comprises 4×4, 4×8, or 8×4 as a sub-block size. In some embodiments, in the case of converting a coding unit comprising bi-prediction, the second size comprises 8 x 8 as a sub-block size. In some embodiments, in the case of converting a coding unit including unidirectional prediction, the coding unit includes an mxn sub-block when it is determined that the height of the coding unit is greater than the width of the coding unit. In some embodiments, in the case of converting a coding unit including unidirectional prediction, the coding unit includes an nxm sub-block when it is determined that the height of the coding unit is less than or equal to the width of the coding unit. In some embodiments, N is greater than or equal to M. For example, m=4 and n=8. In some embodiments, M is greater than or equal to N. For example, m=8 and n=4.

In some embodiments, the current block has dimensions W H, with W and H being positive integers. The conversion does not include performing an optional temporal motion vector prediction (ATMVP) or spatial motion vector prediction (STMVP) on the coding unit. In some embodiments, where W.ltoreq.N and H.ltoreq.M, the conversion does not include performing ATMVP or STMVP on the coding unit, N and M being positive integers. For example, n=m=8. In some embodiments, where W.ltoreq.N or H.ltoreq.M, the conversion does not include performing ATMVP or STMVP on the coding unit. For example, n=m=8. In some embodiments, where W.gtoreq.N and H.gtoreq.M, the conversion does not include performing ATMVP or STMVP on the coding unit. For example, n=m=64. In some embodiments, where W.gtoreq.N or H.gtoreq.M, the conversion does not include performing ATMVP or STMVP on the coding unit. For example, n=m=64.

In some embodiments, the motion information includes at least one of a prediction direction, a motion vector, or a reference picture of the coding unit. In some embodiments, the motion vectors of the sub-blocks of the coding unit are constrained to be within a range around the selected motion vector. In some embodiments, the selected motion vector is the motion vector of the center sub-block of the coding unit. In some embodiments, the selected motion vector is the motion vector of the upper left sub-block of the coding unit.

In some embodiments, the sub-block size is selectively adjusted based on the size of the coding unit. In some embodiments, where the coding unit is greater than wmax×hmax, the subblock sizes are not adjusted, wmax and Hmax being positive integers. In some embodiments, in the case where the coding unit is less than wmin×hmin, the subblock size is not adjusted, wmin and Hmin are positive integers.

Fig. 32 is a flowchart representation of a method 3200 for video processing in accordance with the present technique. The method 3200 includes, at operation 3210, for a transition between a coding unit of video comprising a sub-block and a bitstream representation of the video, determining whether to use a storage bandwidth reduction tool for the transition based on a size of the coding unit. The method 3200 includes, at operation 3220, performing the conversion based on the determination. In some embodiments, where the coding unit is greater than wmax×hmax, the memory bandwidth reduction tool is not used, wmax and Hmax being positive integers. In some embodiments, where the coding unit is less than wmin×hmin, the memory bandwidth reduction tool is not used, wmin and Hmin are positive integers. In some embodiments, the conversion generates the coding unit from the bitstream representation. In some embodiments, the conversion generates a bitstream representation from the encoding unit.

Fig. 33 is a flowchart representation of a method 3300 for video processing in accordance with the present technique. The method 3300 includes, at operation 3310, determining a sub-block size based on information associated with the coding unit for a transition between the coding unit of the video and a bitstream representation of the video in the affine mode. Method 3300 includes, at operation 3320, performing conversion in affine mode based on sub-block size. The conversion may generate the coding unit from the bitstream representation. The conversion may generate a bitstream representation from the encoding unit. In some embodiments, the information includes one or more parameters of the affine pattern, a prediction direction of the coding unit, one or more reference pictures of the coding unit, or a shape of the coding unit.

In some embodiments, the size of the coding unit is w×h, wherein for the first prediction direction, the size of the first reference rectangular region of the current block is m0×n0, wherein for the second prediction direction, the size of the second reference rectangular region is m1×n1, and wherein each dimension of the first reference rectangular region and the second reference rectangular region includes fen 1 and fen 2 additional pixels, respectively, for motion compensation. In some embodiments, the converting includes bi-prediction, and wherein in the case of (m0+flen1) × (n0+flen2) + (m1+flen1) × (n1+flen2) is less than or equal to th1×wxh, the sub-block size is a first size, otherwise the sub-block size is a second size, TH1 is a positive fraction. TH1 may be 165/32 or 225/64. In some embodiments, the conversion comprises unidirectional prediction, and wherein in the case of (m0+flen1) × (n0+flen2) less than or equal to th2×wxh, the sub-block size is a first size, otherwise the sub-block size is a second size, TH2 is a positive fraction. TH2 may be 165/16 or 225/32. In some embodiments, the converting includes bi-prediction, and wherein in the case of (m0+flen1) × (n0+flen2) or (m1+flen1) × (n1+flen2) being greater than th3×wxh, the sub-block size is a first size, otherwise the sub-block size is a second size, TH3 is a positive fraction. TH3 may be 165/32 or 225/64.

In some embodiments, the conversion comprises bi-prediction, and wherein different sub-block sizes are used for different prediction directions. In some embodiments, in the case where (m0+flen1) × (n0+flen2) is less than or equal to TH4×w×h, the sub-block size is a first size for the first prediction direction, otherwise the sub-block size is a second size, TH4 is a positive fraction. In some embodiments, in the case where (m1+flen1) × (n1+flen2) is less than or equal to TH4×w×h, the sub-block size is the first size for the second prediction direction, otherwise the sub-block size is the second size. TH may be 165/32 or 225/64.

In some embodiments, the first dimension is 4 x 4, 4 x 8, or 8 x 4, and wherein the second dimension is 8 x 8. In some embodiments, the first size is determined based on a shape of the coding unit. In some embodiments, FLEN1 and FLEN2 are determined based on the length of the interpolation filter. In some embodiments, the length of the interpolation filter is L, and wherein fluen1=l±n and fluen2=l±n, N is a positive integer. For example, n=1. In some embodiments, FLEN1 and FLEN2 are determined based on the accuracy of the motion vectors of the coding units. In some embodiments, in the case where the horizontal component of the motion vector has integer precision, flen1=0. In some embodiments, in the case where the vertical component of the motion vector has integer precision, flen2=0.

In some embodiments, the sub-block size is further determined based on the location of the coding unit. In some embodiments, the sub-block size is determined based on whether the coding unit is located at a boundary of the coding tree unit. In some embodiments, the conversion includes bi-prediction and the sub-block size is fixed to sw1×sh1. The encoding unit is divided into at least one region of a size sw2×sh2, the at least one region including a plurality of sub-blocks each of which has a sub-block size sw1×sh1, a motion vector MVc of the region is derived from a center position of the region, and the motion vector of each of the plurality of sub-blocks is limited to a range around the motion vector MVc of the region. In some embodiments, for one or more prediction directions of each of the plurality of sub-blocks, an absolute difference between a motion vector of each of the plurality of sub-blocks and a motion vector MVc of the region is less than one or more thresholds associated with the prediction directions of the motion vectors. In some embodiments, the one or more thresholds are different for different prediction directions for each of the plurality of sub-blocks. In some embodiments, the one or more thresholds are determined based on a prediction direction of the coding unit or a size of the region. In some embodiments, the motion vector MVc of the region is the same as the first motion vector of one of the plurality of sub-blocks. In some embodiments, the first motion vector corresponds to an upper left sub-block of the plurality of sub-blocks.

In some embodiments, SW2 SH2 is 8×8, 8×4, or 4×8, SW1×SH1 is 4×4, and the one or more predetermined thresholds are all equal to 1/2 pixel. In some embodiments, SW2 SH2 is 8×8, 8×4, or 4×8, SW1×SH1 is 4×4, and the one or more predetermined thresholds are all equal to 1/4 pixel. In some embodiments, SW2×sh2 is 8×8, 8×4, or 4×8, sw1×sh1 is 4×4, a first threshold of the one or more predetermined thresholds is equal to 1/2 pixel, and a second threshold of the one or more predetermined thresholds is equal to 1 pixel. In some embodiments, SW2×sh2 is 8×8, 8×4, or 4×8, sw1×sh1 is 4×4, a first threshold of the one or more predetermined thresholds is equal to 1/2 pixel, and a second threshold of the one or more predetermined thresholds is equal to 1/4 pixel.

In some embodiments, where the size of the coding unit is greater than wmax×hmax, the coding unit is partitioned into at least one region. In some embodiments, the coding unit has a size of 128×128, and the coding unit is divided into four regions, each region having a size of sw2×sh2=64×64. In some embodiments, the coding unit is 128×64 or 64×128 in size, and the coding unit is divided into two regions, each region being sw2×sh2=64×64 in size. In some embodiments, where the coding unit is greater than wmax×hmax or less than wmin×hmin, the sub-block sizes are fixed, wmax, hmax, wmin and Hmin being positive integers. In some embodiments, in the case where the encoding unit is greater than wmax×hmax or less than wmin×hmin, the motion vector of each of the plurality of sub-blocks is not limited to a range around the motion vector MVc of the region, wmax, hmax, wmin and Hmin being positive integers.

From the foregoing, it will be appreciated that specific embodiments of the disclosed technology have been described herein for purposes of illustration, but that various modifications may be made without deviating from the scope of the invention. Accordingly, the disclosed technology is not limited except as by the appended claims.

Implementations of the subject matter and the functional operations described in this patent document may be implemented in various systems, digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible and non-transitory computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a combination of materials affecting a machine-readable propagated signal, or a combination of one or more of them. The term "data processing unit" or "data processing apparatus" encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. In addition to hardware, an apparatus may include code that creates a runtime environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

A computer program (also known as a program, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. The computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (Field Programmable Gate Array ) or an ASIC (Application Specific Integrated Circuit, application-specific integrated circuit).

Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer does not require such a device. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

It is intended that the specification, together with the accompanying drawings, be considered exemplary only, with the exemplary meaning given by way of example. As used herein, the use of "or" is intended to include "and/or" unless the context clearly indicates otherwise.

Although this patent document contains many specifics, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features that are described in this patent document in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

Similarly, although operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. Furthermore, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

Only some embodiments and examples are described and other embodiments, enhancements, and variations may be made based on what is described and shown in this patent document.

Claims

1. A method for video processing, comprising:

for conversion between a coding unit of video and a bitstream of video, determining a subblock size of a subblock of the coding unit based on motion information associated with the coding unit in an optional temporal motion vector prediction, ATMVP, mode or a spatial motion vector prediction, STMVP, mode; and

the conversion is performed based on the determined sub-block size.

2. The method of claim 1, wherein the converting generates a codec unit from a bitstream.

3. The method of claim 1, wherein the converting generates a bitstream from a codec unit.

4. The method of claim 1, wherein selecting a sub-block size comprises:

selecting a first size as a sub-block size in case the conversion includes a uni-directionally predicted codec unit, and

a second size is selected where the conversion includes bi-directionally predicted codec units, wherein the first size is smaller than the second size.

5. The method of claim 4, wherein the first size comprises 4 x 4, 4 x 8, or 8 x 4 as a sub-block size in the case where the conversion comprises a uni-directionally predicted codec unit.

6. The method of claim 4, wherein the second size comprises 8 x 8 as a sub-block size in the case where the conversion comprises bi-predictive codec units.

7. The method of claim 1, wherein, in the case where the conversion includes a unidirectional predicted codec unit, the codec unit includes an mxn sub-block when it is determined that a height of the codec unit is greater than a width of the codec unit.

8. The method of claim 1, wherein, in the case where the conversion includes a unidirectional predicted codec unit, the codec unit includes an nxm sub-block when it is determined that a height of the codec unit is less than or equal to a width of the codec unit.

9. The method of claim 7 or 8, wherein N is greater than or equal to M.

10. The method of claim 9, wherein m=4 and n=8.

11. The method of claim 7 or 8, wherein M is greater than or equal to N.

12. The method of claim 11, wherein m=8 and n=4.

13. The method of any of claims 1 to 8, further comprising:

determining whether to disable the optional temporal motion vector prediction ATMVP or spatial motion vector prediction STMVP for the codec unit based on dimensions of the codec unit, wherein the dimensions of the codec unit are w×h, and W and H are positive integers.

14. The method of claim 13, wherein disabling ATMVP or STMVP for the codec unit is determined with w+.n and h+.m, N and M being positive integers.

15. The method of claim 14, wherein n=m=8.

16. The method of claim 13, wherein it is determined to disable ATMVP or STMVP for the codec unit if w+.n or h+.m.

17. The method of claim 16, wherein n=m=8.

18. The method of claim 13, wherein it is determined to disable ATMVP or STMVP for the codec unit if W is N and H is M.

19. The method of claim 18, wherein n=m=64.

20. The method of claim 13, wherein disabling ATMVP or STMVP for the codec unit is determined if w≡n or h≡m.

21. The method of claim 20, wherein n=m=64.

22. The method of any of claims 1 to 8, wherein the motion information comprises at least one of a prediction direction, a motion vector, or a reference picture of the codec unit.

23. The method according to any of claims 1 to 8, wherein the motion vectors of the sub-blocks of the codec unit are constrained within a range around the selected motion vector.

24. The method of claim 23, wherein the selected motion vector is a motion vector of a center sub-block of a codec unit.

25. The method of claim 23, wherein the selected motion vector is a motion vector of an upper left sub-block of a codec unit.

26. The method of any of claims 1 to 8, wherein the sub-block size is selectively adjusted based on the size of the codec unit.

27. The method of claim 26, wherein when the codec unit is greater than W _max ×H _max In the case of (a), the subblock size is not adjusted, W _max And H _max Is a positive integer.

28. The method of claim 26, wherein when the codec unit is less than W _min ×H _min In the case of (a), the subblock size is not adjusted, W _min And H _min Is a positive integer.

29. A method for video processing, comprising:

for a transition between a codec unit of a video including sub-blocks and a bitstream of the video, determining whether to use a storage bandwidth reduction tool for the transition based on a size of the codec unit, and

performing the conversion based on the determination;

wherein in an optional temporal motion vector prediction, ATMVP, mode or spatial motion vector prediction, STMVP, mode, a sub-block size of a sub-block of the coding unit is determined based on motion information associated with the coding unit.

30. The method of claim 29, wherein when the codec unit is greater than W _max ×H _max In the case of (a), a storage bandwidth reduction tool is not used, W _max And H _max Is a positive integer.

31. The method of claim 29, wherein when the codec unit is less than W _min ×H _min In the case of (a), a storage bandwidth reduction tool is not used, W _min And H _min Is a positive integer.

32. The method of any of claims 29 to 31, wherein the converting generates a codec unit from a bitstream.

33. The method of any of claims 29 to 31, wherein the converting generates a bitstream from a codec unit.

34. An apparatus in a video system comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement the method of any of claims 1-33.