TWI846727B - Two-step inter prediction - Google Patents

Abstract

Embodiments of the present disclosure relates to two-step inter-prediction. A method for video processing is provided, comprising: generating, for a current block, intermediate predictions based on a first motion information associated with the current block; updating the first motion information to a second motion information based on the intermediate predictions; and generating a final prediction for the current block based on the second motion information.

Description

Two-step frame prediction

This patent document relates to video encoding and decoding technology, equipment and system.

[Cross references to related applications]

This application promptly claims the priority and benefits of International Patent Application No. PCT/CN2018/104301 filed on September 6, 2018 and International Patent Application No. PCT/CN2018/109250 filed on October 6, 2018, in accordance with applicable patent laws and/or the Paris Convention. The entire disclosure of International Patent Application No. PCT/CN2018/104301 and PCT/CN2018/109250 is incorporated herein by reference as part of the disclosure of this application.

Despite advances in video compression, digital video still accounts for the largest bandwidth usage on the Internet and other digital communications networks. The bandwidth required for digital video usage is expected to continue to grow as the number of connected user devices capable of receiving and displaying video increases.

Signaling Notification describes apparatus, systems and methods related to digital video codecs, and in particular, describes motion refinement based on updated motion vectors generated according to two-step inter-frame prediction. The described methods can be applied to existing video codec standards (e.g., High Efficiency Video Codec (HEVC)) and future video codec standards or video codecs.

In one representative aspect, a video processing method is provided, comprising: determining raw motion information of a current block; scaling raw motion vectors of the raw motion information and derived motion vectors derived based on the raw motion vectors to the same target accuracy; generating updated motion vectors from the scaled raw and derived motion vectors; and performing a conversion between the current block and a bitstream representation of a video including the current block based on the updated motion vectors.

In another representative aspect, a video processing method is provided, including: determining raw motion information of a current block; updating a raw motion vector of the raw motion information of the current block based on a refinement method; clipping the updated motion vector to a range; and performing a conversion between a bitstream representation of the current block and a video including the current block based on the clipped updated motion vector.

In another representative aspect, a video processing method is provided, comprising: determining original motion information associated with a current block; generating updated motion information based on a specific prediction mode; and performing a conversion between the current block and a bitstream representation of video data including the current block based on the updated motion information, wherein the specific prediction mode comprises one or more of bidirectional optical flow (BIO) refinement, decoder-side motion vector refinement (DMVR), frame rate up-conversion (FRUC) technology, or template matching technology.

In yet another representative aspect, a video processing method is provided, comprising: Determining a motion vector difference (MVD) precision of a current block processed using an affine mode from a set of MVD precisions; performing a conversion between the current block and a bitstream representation of a video including the current block based on the determined MVD precision.

In yet another representative aspect, a video processing method is provided, comprising: determining unupdated motion information associated with a current block; updating the unupdated motion information based on multiple decoder-side motion vector derivation (DMVD) methods to generate updated motion information for the current block; and performing a conversion between the current block and a bitstream representation of a video including the current block based on the updated motion information.

In another representative aspect, a video processing method is provided, comprising: generating an intermediate prediction for the current block based on first motion information associated with the current block; updating the first motion information to second motion information based on the intermediate prediction; and generating a final prediction for the current block based on the second motion information.

In another representative aspect, the disclosed technology can be used to provide a method for video encoding and decoding. The method includes receiving a bitstream representation of a current block of video data, generating updated first and second reference motion vectors based on a weighted sum of a first scaled motion vector and first and second scaled reference motion vectors, respectively, wherein a first motion vector is derived based on a first reference motion vector from a first reference block and a second reference motion vector from a second reference block, wherein the current block is associated with the first and second reference blocks, wherein the first scaled motion vector is generated by scaling the first motion vector to a target precision, and wherein the first and second scaled reference motion vectors are generated by scaling the first and second reference motion vectors to a target precision, respectively, and processing the bitstream representation based on the updated first and second reference motion vectors to generate the current block.

In another representative aspect, the disclosed technology can be used to provide a method for video encoding and decoding. The method includes generating an intermediate prediction for the current block based on first motion information associated with the current block, updating the first motion information to second motion information, and generating a final prediction for the current block based on the intermediate prediction or the second motion information.

In another representative aspect, the disclosed technology can be used to provide a method for video encoding and decoding. The method includes receiving a bitstream representation of a current block of video data, generating intermediate motion information based on motion information associated with the current block, generating updated first and second reference motion vectors based on first and second reference motion vectors, respectively, wherein the current block is associated with the first and second reference blocks, and wherein the first and second reference motion vectors are associated with the first and second reference blocks, respectively, and processing the bitstream representation based on the intermediate motion information or the updated first and second reference motion vectors to generate the current block.

In another representative aspect, the disclosed technology can be used to provide a method for video encoding and decoding. The method includes generating an intermediate prediction for the current block based on first motion information associated with the current block, updating the first motion information to second motion information, and generating a final prediction for the current block based on the intermediate prediction or the second motion information.

In another representative aspect, the disclosed technology can be used to provide a method for video encoding and decoding. The method includes receiving a bitstream representation of a current block of video data, generating intermediate motion information based on motion information associated with the current block, generating updated first and second reference motion vectors based on first and second reference motion vectors, respectively, wherein the current block is associated with the first and second reference blocks, and wherein the first and second reference motion vectors are associated with the first and second reference blocks, respectively, and processing the bitstream representation based on the intermediate motion information or the updated first and second reference motion vectors to generate the current block.

In another representative aspect, the disclosed technology can be used to provide a method for video encoding and decoding. The method includes generating an intermediate prediction for the current block based on first motion information associated with the current block, updating the first motion information to second motion information, and generating a final prediction for the current block based on the intermediate prediction or the second motion information.

In another representative aspect, the disclosed technology can be used to provide a method for video encoding and decoding. The method includes receiving a bitstream representation of a current block of video data, generating intermediate motion information based on motion information associated with the current block, generating updated first and second reference motion vectors based on first and second reference motion vectors, respectively, wherein the current block is associated with the first and second reference blocks, and wherein the first and second reference motion vectors are associated with the first and second reference blocks, respectively, and processing the bitstream representation based on the intermediate motion information or the updated first and second reference motion vectors to generate the current block.

In another representative aspect, the disclosed technology can be used to provide a method for video encoding and decoding, the method comprising generating an updated reference block for a bitstream representation of a current block by modifying a reference block associated with the current block; calculating a temporal gradient for bidirectional optical flow (BIO) motion refinement based on the updated reference block; and performing a transformation including BIO motion refinement between the bitstream representation and the current block based on the temporal gradient.

In another representative aspect, the disclosed technology can be used to provide a method for video encoding and decoding, the method comprising generating a temporal gradient for bidirectional optical flow (BIO) motion refinement for a bitstream representation of a current block; generating an updated temporal gradient by subtracting a difference between a first mean and a second mean from the temporal gradient, wherein the first mean is a mean of a first reference block, wherein the second mean is a mean of a second reference block, and wherein the first and second reference blocks are associated with the current block; and performing a transformation including BIO motion refinement between the bitstream representation and the current block based on the updated temporal gradient.

In another representative aspect, the above method is embodied in the form of processor executable code and stored in a computer readable program medium.

In yet another representative aspect, a device configured or operable to perform the above method is disclosed. The device may include a processor programmed to implement the method.

In yet another representative aspect, a video decoder device is disclosed that can implement the method as described herein.

In yet another representative aspect, a video encoder device is disclosed that can implement the method as described herein.

The above and other aspects and features of the disclosed technology are described in more detail in the accompanying drawings, the specification and the patent application scope.

1000:CU

1001: Sub-CU

1050, 1810, 1811: Reference pictures

1051: Corresponding block

1100:CU

1101~1104: Sub-CU

1111~1114, 1400, 1500, 1600, :block

1700, 1800: Current CU

1701~1705、2200、A ₀ 、A ₁ 、B ₀ 、B ₁ 、B ₂ : Block

1801, 1802, V ₀ , V ₁ , MV ₀ , MV0', MV1, MV1': motion vector

1803, 1804: Time distance

1900: Current CU

1910: Current picture

2000: Unilateral motion estimation

2201: Filling area

2700, 2800, 2900, 3000, 3100, 3300, 3400, 3500, 3600, 3800, 3900, 4000, 4100, 4200, 4300: Methods

2710~2730, 2810~2830, 2910~2940, 3010~3030, 3110~3140, 3310~3330, 3410~3440, 3510~3530, 3610~3630, 3810~3840, 3910~3940, 4010~4030, 4110~4120, 4210~4230, 4310~4330: Steps

3700: Video processing device

3702:Processor

3704: Storage

3706: Video processing circuit

A, B, C, D, E: sub-blocks

C ₀ , C ₁ : Position

tb, td: POC distance

Figure 1 shows an example of constructing a Merge candidate list.

Figure 2 shows examples of the locations of spatial candidates.

Figure 3 shows an example of candidate pairs that undergo redundancy verification for spatial merge candidates.

Figures 4A and 4B show examples of the location of the second prediction unit (PU) based on the size and shape of the current block.

Figure 5 shows an example of motion vector scaling for temporal merge candidate selection.

Figure 6 shows an example of candidate positions for temporal merge candidates.

Figure 7 shows an example of generating a combined bidirectional prediction Merge candidate.

Figure 8 shows an example of constructing motion vector prediction candidates.

Figure 9 shows an example of motion vector scaling for spatial motion vector candidates.

Figure 10 shows an example of motion prediction using the optional temporal motion vector prediction (ATMVP) algorithm for coding units (CUs).

Figure 11 shows an example of a coding unit (CU) with sub-blocks and neighboring blocks used by the spatial-temporal motion vector prediction (STMVP) algorithm.

Figures 12A and 12B show example snapshots of sub-blocks when using the Overlapping Block Motion Compensation (OBMC) algorithm.

Figure 13 shows an example of neighboring samples used to derive parameters for the Local Illumination Compensation (LIC) algorithm.

Figure 14 shows an example of a simplified affine motion model.

Figure 15 shows an example of the affine motion vector field (MVF) for each sub-block.

Figure 16 shows an example of motion vector prediction (MVP) for the AF_INTER affine motion mode.

Figures 17A and 17B show example candidates for the AF_MERGE affine motion mode.

Figure 18 shows an example of bilateral matching in the Pattern Matching Motion Vector Derivation (PMMVD) mode, which is a special Merge mode based on the Frame Rate Up-Conversion (FRUC) algorithm.

Figure 19 shows an example of template matching in the FRUC algorithm.

Figure 20 shows an example of unilateral motion estimation in the FRUC algorithm.

Figure 21 shows an example of an optical flow trajectory used by the Bidirectional Optical Flow (BIO) algorithm.

Figures 22A and 22B show example snapshots using the Bidirectional Optical Flow (BIO) algorithm without block expansion.

Figure 23 shows an example of a decoder-side motion vector refinement (DMVR) algorithm based on bilateral template matching.

Figure 24 shows an example of a template definition used in transformation coefficient context modeling.

Figure 25 shows different examples of motion vector scaling.

Figures 26A and 26B show examples of internal and boundary sub-blocks in a PU/CU.

FIG27 shows a flow chart of an example method for video encoding and decoding according to the currently disclosed technology.

FIG28 shows a flow chart of another example method for video encoding and decoding according to the currently disclosed technology.

FIG29 shows a flow chart of another example method for video encoding and decoding according to the currently disclosed technology.

FIG30 shows a flow chart of another example method for video encoding and decoding according to the currently disclosed technology.

FIG31 shows a flow chart of another example method for video encoding and decoding according to the currently disclosed technology.

Figure 32 shows an example of inferring motion vectors in bidirectional optical flow based video coding.

FIG33 shows a flow chart of another example method for video encoding and decoding according to the currently disclosed technology.

FIG34 shows a flow chart of another example method for video encoding and decoding according to the currently disclosed technology.

FIG35 shows a flow chart of another example method for video encoding and decoding according to the currently disclosed technology.

FIG36 shows a flow chart of another example method for video encoding and decoding according to the currently disclosed technology.

Figure 37 is a block diagram of an example of a hardware platform for implementing the visual media decoding or visual media encoding and decoding techniques described in this document.

FIG38 shows a flow chart of another example method for video processing according to the presently disclosed technology.

FIG39 shows a flow chart of another example method for video processing according to the presently disclosed technology.

FIG40 shows a flow chart of another example method for video processing according to the presently disclosed technology.

FIG41 shows a flow chart of another example method for video processing according to the presently disclosed technology.

FIG42 shows a flow chart of another example method for video processing according to the presently disclosed technology.

FIG43 shows a flow chart of another example method for video processing according to the presently disclosed technology.

Video codec methods and techniques are ubiquitous in modern technology due to the increasing demand for higher resolution video. Video codecs typically include electronic circuits or software that compress or decompress digital video and are constantly being improved to provide higher codec efficiency. Video codecs convert uncompressed video to compressed formats and vice versa. There is a complex relationship between video quality, the amount of data used to represent the video (determined by the bit rate), the complexity of the encoding and decoding algorithms, sensitivity to data loss and errors, ease of editing, random access, and end-to-end delay (latency). The compression format typically complies with a standard video compression specification, such as the High Efficiency Video Codec (HEVC) standard (also known as H.265 or MPEG-H Part 2), the to-be-completed Universal Video Codec standard, or other current and/or future video codec standards.

Embodiments of the disclosed technology can be applied to existing video codec standards (e.g., HEVC, H.265) and future standards to improve compression performance. Section headings are used in this document to improve the readability of the description and do not in any way limit the discussion or embodiments (and/or implementations) to only the corresponding section.

1. Example of inter-frame prediction in HEVC/H.265

Video codec standards have improved significantly over the years and now partially offer high codec efficiency and support for higher resolutions. The latest standards such as HEVC and H.265 are based on a hybrid video codec architecture that utilizes temporal prediction plus transform coding.

1.1. Example of prediction model

Each inter-frame predicted PU (prediction unit) has motion parameters for one or two reference picture lists. In some embodiments, the motion parameters include motion vectors and reference picture indices. In other embodiments, inter_pred_idc may also be used to signal the use of one of the two reference picture lists. In yet other embodiments, the motion vectors may be explicitly encoded and decoded as deltas relative to the predictor.

When a CU is encoded or decoded in skip mode, a PU is associated with the CU and there are no significant residual coefficients, no coded motion vector increments or reference picture indices. Specify Merge mode so that the motion parameters of the current PU are obtained from neighboring PUs, including spatial and temporal candidates. Merge mode can be applied to any inter-frame predicted PU, not just skip mode. An alternative to Merge mode is explicit transmission of motion parameters, where, for each PU, the motion vectors, the corresponding reference picture index for each reference picture list, and the reference picture list usage are explicitly signaled.

When the signaling indicates that one of the two reference picture lists will be used, a PU is generated from a sample block. This is called "uni-prediction". Uni-prediction can be used for both P and B slices.

When the signaling indicates that two reference picture lists will be used, the PU is generated from two sample blocks. This is called "bi-prediction". Bi-prediction is only applicable to B slices.

1.1.1.1 Constructing candidate implementations for Merge mode

When predicting PUs using Merge mode, the index pointing to the entry in the Merge candidate list is parsed from the bitstream and used to retrieve motion information. The construction of this list can be summarized according to the following sequence of steps:

Step 1: Original candidate derivation

A. Step 1.1: Spatial candidate derivation

B. Step 1.2: Redundant verification of spatial candidates

C. Step 1.3: Time candidate derivation

Step 2: Insert additional candidates

A. Step 2.1: Create two-way prediction candidates

B. Step 2.2: Insert zero motion candidates

Figure 1 shows an example of constructing a Merge candidate list based on the sequence of steps summarized above. For spatial Merge candidate derivation, up to four Merge candidates are selected from candidates located at five different positions. For temporal Merge candidate derivation, up to one Merge candidate is selected from two candidates. Since a constant number of candidates is assumed at the decoder for each PU, additional candidates are generated when the number of candidates does not reach the maximum number of Merge candidates (MaxNumMergeCand) signaled in the slice header. Since the number of candidates is constant, truncated unary binarization (TU) is used to encode the index of the best Merge candidate. If the size of the CU is equal to 8, all PUs of the current CU share a single Merge candidate list, which is the same as the Merge candidate list of the 2N×2N prediction unit.

1.1.2 Constructing spatial merge candidates

In the derivation of spatial Merge candidates, up to four Merge candidates are selected from the candidates located at the positions depicted in Figure 2. The order of derivation is _A1 , _B1 , _B0 , _A0 , and _B2 . Position B2 is considered only when any PU at positions _A1 , _B1 , _B0 , _A0 is unavailable (for example, because it belongs to another slice or _block ) or is intra-frame coded. After adding the candidate at position _A1 , a redundancy check is performed on the addition of the remaining candidates, which ensures that candidates with the same motion information are excluded from the list, thereby improving the coding efficiency. In order to reduce computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. Instead, only the pairs connected by arrows in FIG. 3 are considered, and a candidate is added to the list only if the corresponding candidate for redundancy checking has different motion information. Another source of duplicate motion information is a "second PU" associated with a partition different from 2N×2N. As an example, FIGS. 4A and 4B depict the second PU for the cases of N×2N and 2N×N, respectively. When the current PU is partitioned as N×2N, the candidate at position A ₁ is not considered for list construction. In some embodiments, adding this candidate may result in two prediction units with the same motion information, which is redundant for having only one PU in the coding unit. Similarly, position B ₁ is not considered when the current PU is partitioned as 2N×N.

1.1.1.3 Build time Merge candidates

In this step, only one candidate is added to the list. Specifically, in the derivation of the temporal merge candidate, the scaled motion vector is derived based on the co-located PU that belongs to the picture with the smallest POC difference with the current picture in a given reference picture list. The reference picture list to be used for the derivation of the co-located PU is explicitly signaled in the slice header.

Figure 5 shows an example of the derivation of a scaled motion vector for a temporal Merge candidate (as dashed line), which is scaled from the motion vector of the co-located PU using the POC distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the co-located picture and the co-located picture. The reference picture index of the temporal Merge candidate is set equal to zero. For B slices, two motion vectors are obtained, one for reference picture list 0 and the other for reference picture list 1, and the two motion vectors are combined to obtain a bidirectional prediction Merge candidate.

In the co-located PU (Y) belonging to the reference frame, the position of the temporal candidate is selected between candidates C ₀ and C ₁ , as shown in Figure 6. If the PU at position C ₀ is not available, is intra-coded, or is outside the current CTU, position C ₁ is used. Otherwise, position C ₀ is used for the derivation of the temporal Merge candidate.

1.1.4 Constructing additional types of Merge candidates

In addition to the space-time merge candidates, there are two additional types of merge candidates: combined bidirectional prediction merge candidates and zero merge candidates. Combined bidirectional prediction merge candidates are generated by utilizing space-time merge candidates. Combined bidirectional prediction merge candidates are only used for B slices. Combined bidirectional prediction candidates are generated by combining the first reference picture list motion parameters of the original candidate with the second reference picture list motion parameters of another candidate. If the two tuples provide different motion hypotheses, they will form a new bidirectional prediction candidate.

Figure 7 shows an example of this process, where two candidates in the original list (710 on the left) have either mvL0 and refIdxL0 or mvL1 and refIdxL1, which are used to create a combined bidirectional prediction Merge candidate that is added to the final list (right).

Zero-motion candidates are inserted to fill the remaining entries in the Merge candidate list up to the MaxNumMergeCand capacity. These candidates have zero spatial displacement and a reference picture index that starts at zero and increases each time a new zero-motion candidate is added to the list. The number of reference frames used by these candidates is 1 and 2, for unidirectional and bidirectional prediction, respectively. In some embodiments, redundancy checks are not performed on these candidates.

1.1.5 Example of motion estimation regions for parallel processing

To speed up the encoding and decoding process, motion estimation can be performed in parallel, thereby deriving motion vectors for all prediction units within a given region at the same time. Deriving Merge candidates from spatial neighbors may interfere with parallel processing, because a prediction unit cannot derive motion parameters from neighboring PUs until its associated motion estimation is completed. To reduce the trade-off between encoding and decoding efficiency and processing latency, a Motion Estimation Region (MER) can be defined, and the size of the MER is signaled in the Picture Parameter Set (PPS) using the "log2_parallel_merge_level_minus2" syntax element. When a MER is defined, Merge candidates that fall into the same region are marked as unavailable and are therefore not considered in list construction.

1.2 Implementation of Advanced Motion Vector Prediction (AMVP)

AMVP exploits the spatiotemporal correlation of motion vectors with neighboring PUs for explicit transmission of motion parameters. It constructs a motion vector candidate list by first checking the availability of the top and left sides of the temporally neighboring PU positions, removing redundant candidates and adding zero vectors to make the candidate list a constant length. The encoder can then select the best predictor from the candidate list and send a corresponding index indicating the selected candidate. Similar to Merge index signaling, the index of the best motion vector candidate is encoded using truncated unary. The maximum value to be encoded in this case is 2 (see Figure 8). In the following section, details on the derivation process of motion vector prediction candidates are provided.

1.2.1 Example of constructing motion vector prediction candidates

Figure 8 summarizes the derivation process of motion vector prediction candidates and can be implemented for each reference picture list with an index as input.

In motion vector prediction, two types of motion vector candidates are considered: spatial motion vector candidates and temporal motion vector candidates. For spatial motion vector candidate derivation, two motion vector candidates are ultimately derived based on the motion vectors of each PU located at the five different positions previously shown in FIG. 2.

For temporal motion vector candidate derivation, a motion vector candidate is selected from two candidates that are derived based on two different co-located positions. After generating the first list of spatio-temporal candidates, duplicate motion vector candidates in the list are removed. If the number of potential candidates is greater than 2, motion vector candidates whose reference picture index within the associated reference picture list is greater than 1 are removed from the list. If the number of spatio-temporal motion vector candidates is less than 2, additional zero motion vector candidates are added to the list.

1.2.2 Constructing spatial motion vector candidates

In the derivation of the spatial motion vector candidate, a maximum of two candidates are considered among five potential candidates, which are derived from the PU located at the positions shown previously in Figure 2, which are the same as the positions of the motion merge. The derivation order for the left side of the current PU is defined as A ₀ , A ₁ , and scaled A ₀ , scaled A _1. The derivation order for the top side of the current PU is defined as B ₀ , B ₁ , B ₂ , scaled B ₀ , scaled B ₁ , scaled B _2. Therefore, for each side, there are four cases that can be used as motion vector candidates, two of which do not require the use of spatial scaling, and two use spatial scaling. The four different cases are summarized as follows:

No room to zoom

- (1) Same reference image list, and same reference image index (same POC)

- (2) Different reference image lists, but same reference images (same POC)

Spatial zoom

- (3) Same reference image list, but different reference images (different POC)

- (4) Different reference image lists, and different reference images (different POCs)

First check the case without spatial scaling, then check spatial scaling. Spatial scaling is considered when the POC differs between the reference picture of the neighboring PU and the reference picture of the current PU regardless of the reference picture list. If all PUs of the left candidate are unavailable or intra-coded, scaling of the above motion vectors is allowed to help the parallel derivation of the left and top MV candidates. Otherwise, spatial scaling of the above motion vectors is not allowed.

As shown in the example of Figure 9, in the case of spatial scaling, the motion vectors of neighboring PUs are scaled in a similar way to temporal scaling. One difference is that the reference picture list and the index of the current PU are given as input; the actual scaling process is the same as the temporal scaling process.

1.2.3 Constructing temporal motion vector candidates

Except for the reference picture index derivation, all processes used to derive the temporal merge candidate are the same as those used to derive the spatial motion vector candidate (as shown in the example of FIG6 ). In some embodiments, the reference picture index is signaled to the decoder.

2. Example of inter-frame prediction method in Joint Exploration Model (JEM)

In some embodiments, a reference software called the Joint Exploration Model (JEM) is used to explore future video codec techniques. In JEM, sub-block based prediction is adopted in several codec tools, such as affine prediction, optional temporal motion vector prediction (ATMVP), spatio-temporal motion vector prediction (STMVP), bidirectional optical flow (BIO), frame rate up-conversion (FRUC, local adaptive motion vector resolution (LAMVR), overlapping block motion compensation (OBMC), local illumination compensation (LIC), and decoder-side motion vector refinement (DMVR).

2.1 Example of sub-CU-based motion vector prediction

In JEM with quadtree plus binary tree (QTBT), each CU can have at most one set of motion parameters for each prediction direction. In some embodiments, two sub-CU level motion vector prediction methods are considered in the encoder by dividing a large CU into sub-CUs and deriving motion information for all sub-CUs of the large CU. The Alternative Temporal Motion Vector Prediction (ATMVP) method allows each CU to extract multiple sets of motion information from multiple blocks smaller than the current CU in the collocated reference picture. In the Spatial-Temporal Motion Vector Prediction (STMVP) method, the motion vectors of the sub-CUs are recursively derived by using the temporal motion vector prediction values and spatially neighboring motion vectors. In some embodiments, in order to preserve a more accurate motion field for sub-CU motion prediction, motion compression of the reference frame may be disabled.

2.1.1 Example of Optional Temporal Motion Vector Prediction (ATMVP)

In the ATMVP method, the temporal motion vector prediction (TMVP) method is modified by extracting multiple sets of motion information (including motion vectors and reference indices) from blocks smaller than the current CU.

FIG10 shows an example of the ATMVP motion prediction process for CU 1000. The ATMVP 1000 method predicts the motion vector of a sub-CU 1001 within CU 1000 in two steps. The first step is to identify the corresponding block 1051 in the reference picture 1050 using the time vector. The reference picture 1050 is also called a motion source picture. The second step is to divide the current CU 1000 into sub-CUs 1001 and obtain the motion vector and the reference index of each sub-CU from the block corresponding to each sub-CU.

In the first step, the reference picture 1050 and the corresponding block are determined by the motion information of the spatial neighboring blocks of the current CU 1000. In order to avoid repeated scanning of neighboring blocks, the first Merge candidate in the Merge candidate list of the current CU 1000 is used. The first available motion vector and its associated reference index are set to the index of the temporal vector and the motion source picture. In this way, the corresponding block can be identified more accurately compared to TMVP, where the corresponding block (sometimes called a collocated block) is always located at the lower right or center position relative to the current CU.

In the second step, the corresponding block of the sub-CU 1051 is identified by the time vector in the motion source picture 1050 by adding the time vector to the coordinates of the current CU. For each sub-CU, the motion information of its corresponding block (e.g., the minimum motion grid covering the center sample) is used to derive the motion information of the sub-CU. After the motion information of the corresponding N×N block is identified, it is converted into a reference index and motion vector of the current sub-CU in the same way as the TMVP of HEVC, where motion scaling and other processes are also applicable. For example, the decoder checks whether the low latency condition is met (e.g., the POC of all reference pictures of the current picture is less than the POC of the current picture) and may use the motion vector MV _x (e.g., the motion vector corresponding to reference picture list X) to predict the motion vector MV _y of each sub-CU (e.g., where X is equal to 0 or 1 and Y is equal to 1-X).

2.1.2 Example of Space-Temporal Motion Vector Prediction (STMVP)

In the STMVP method, the motion vectors of the sub-CU are recursively derived in the raster scanning order. Figure 11 shows an example of a CU with four sub-blocks and neighboring blocks. Consider an 8×8 CU1100 containing four 4×4 sub-CUs A(1101), B(1102), C(1103) and D(1104). The neighboring 4×4 blocks in the current frame are labeled as a(1111), b(1112), c(1113) and d(1114).

The motion derivation of sub-CU A starts by identifying its two spatial neighbors. The first neighbor is the N×N block above sub-CU A 1101 (block c 1103). If block c (1113) is not available or is intra-frame coded, then (starting from block c 1113, from left to right) check the other N×N blocks above sub-CU A (1101). The second neighbor is the block to the left of sub-CU A 1101 (block b 1112). If block b (1112) is not available or is intra-frame coded, then (starting from block b 1112, from top to bottom) check the other blocks to the left of sub-CU A 1101. The motion information obtained from the neighboring blocks of each list is scaled to the first reference frame of the given list. Next, the temporal motion vector predictor (TMVP) of sub-block A 1101 is derived by following the same process as the TMVP derivation specified in HEVC. The motion information of the collocated block at D 1104 is extracted and scaled accordingly. Finally, after retrieving and scaling the motion information, all available motion vectors are averaged separately for each reference list. The average motion vector is designated as the motion vector of the current sub-CU.

2.1.3 Example of sub-CU motion prediction mode signaling

In some embodiments, sub-CU mode is enabled as an additional Merge candidate, and no additional syntax elements are required to signal the mode. Two additional Merge candidates are added to the Merge candidate list of each CU to represent the ATMVP mode and the STMVP mode. In some embodiments, if the sequence parameter set indicates that ATMVP and STMVP are enabled, up to seven Merge candidates can be used. The coding logic of the additional Merge candidates is the same as that of the Merge candidates in HM, which means that for each CU in a P or B slice, two additional Merge candidates may require an additional two RD checks. In some embodiments, for example, in JEM, all bins of the Merge index are context-coded by CABAC (context-based adaptive binary arithmetic coding and decoding). In other embodiments, for example, in HEVC, only the first binary is context coded, while the remaining bins are context bypass coded.

2.2 Adaptive motion vector difference resolution

In some embodiments, when use_integer_mv_flag in the slice header is equal to 0, the motion vector difference (MVD) (between the PU's motion vector and the predicted motion vector) is signaled in units of quarter luma samples. In JEM, Locally Adaptive Motion Vector Resolution (LAMVR) is introduced. In JEM, MVD can be coded and decoded in units of quarter luma samples, integer luma samples, or four luma samples. The MVD resolution is controlled at the codec unit (CU) level, and the MVD resolution flag is conditionally signaled for each CU with at least one non-zero MVD component.

For a CU with at least one non-zero MVD component, a first flag is signaled to indicate whether quarter luma sample MV precision is used in the CU. When the first flag (equal to 1) indicates that quarter luma sample MV precision is not used, another flag is signaled to indicate whether integer luma sample MV precision or four luma sample MV precision is used.

When the first MVD resolution flag of a CU is zero or not encoded for the CU (meaning all MVDs in the CU are zero), a quarter luma sample MV resolution is used for the CU. When the CU uses integer luma sample MV precision or quad luma sample MV precision, the MVP in the CU's AMVP candidate list is rounded to the corresponding precision.

In the encoder, CU-level RD check is used to determine which MVD resolution is used for a CU. That is, for each MVD resolution, CU-level RD check is performed three times. To speed up the encoder, the following encoding scheme is applied in JEM.

During RD verification of a CU with normal quarter luma sample MVD resolution, the motion information (integer luma sample accuracy) of the current CU is stored. The stored motion information (after rounding) is used as a starting point for further small-range motion vector refinement during RD verification for the same CU with integer luma samples and 4 luma sample MVD resolution, so that the time-consuming motion estimation process is not repeated three times.

Conditionally call RD check for CUs with 4 luma samples MVD resolution. For a CU, when the RD cost integer luma sample MVD resolution is much larger than quarter luma sample MVD resolution, skip RD check for CU's 4 luma sample MVD resolution.

2.3 Example of higher motion vector storage accuracy

In HEVC, the motion vector accuracy is quarter-pixel (one-quarter luma sample and one-eighth chroma sample for 4:2:0 video). In JEM, the accuracy of internal motion vector storage and Merge candidate selection is increased to 1/16 pixel. The higher motion vector accuracy (1/16 pixel) is used for motion compensated inter-frame prediction for CUs encoded or decoded in Skip/Merge mode. For CUs encoded or decoded using normal AMVP mode, integer pixel or quarter-pixel motion is used.

The SHVC upsampled interpolation filter with the same filter length and normalization factor as the HEVC motion compensation interpolation filter is used as the motion compensation interpolation filter for the additional fractional pixel positions. The chroma component motion vector accuracy in JEM is 1/32 samples, and the additional interpolation filter for the 1/32 pixel fractional position is derived by using the average of two adjacent 1/16 pixel fractional position filters.

2.4 Example of Overlapped Block Motion Compensation OBMC (Overlapped Block Motion Compensation)

In JEM, OBMC can be turned on and off using CU-level syntax. When OBMC is used in JEM, OBMC is performed on all motion compensation (MC) block boundaries except the right and bottom boundaries of the CU. In addition, it is also applied to luma and chroma components. In JEM, MC blocks correspond to codec blocks. When a CU is encoded and decoded with sub-CU mode (including sub-CU Merge, affine, and FRUC modes), each sub-block of the CU is an MC block. In order to handle CU boundaries in a unified manner, OBMC is performed at the sub-block level for all MC block boundaries, where the sub-block size is set to be equal to 4×4, as shown in Figures 12A and 12B.

Figure 12A shows sub-blocks at the CU/PU boundary, and the shaded sub-blocks are where OBMC is applied. Similarly, Figure 12B shows sub-blocks in ATMVP mode.

When OBMC is applied to the current sub-block, in addition to the current motion vector, the motion vectors of the four connected neighboring sub-blocks (if available and different from the current motion vector) are also used to derive the prediction block for the current sub-block. These multiple prediction blocks based on multiple motion vectors are combined to generate the final prediction signal for the current sub-block.

The prediction block based on the motion vector of the neighboring sub-block is denoted as _PN , where N indicates the index of the neighboring upper, lower, left and right sub-blocks, and the prediction block based on the motion vector of the current sub-block is denoted as _PC . When _PN is based on the motion information of the neighboring sub-block containing the same motion information as the current sub-block, OBMC is not performed from _PN . Otherwise, each _PN sample is added to the same sample in _PC , that is, four rows/columns of _PN are added to _PC . Weighting factors {1/4, 1/8, 1/16, 1/32} are used for _PN , and weighting factors {3/4, 7/8, 15/16, 31/32} are used for PC. The exception is small MC blocks (i.e., when the height or width of the coded block is equal to 4 or the CU is coded in sub-CU mode), for which only two rows/columns of _PN are added to _PC . In this case, weighting factors {1/4, 1/8} are used for _PN , and weighting factors {3/4, 7/8} are used for _PC . For _PN generated based on motion vectors of vertically (horizontally) adjacent sub-blocks, samples in the same row (column) of _PN are added to _PC with the same weighting factor.

In JEM, for CUs with size less than or equal to 256 luma samples, a CU-level flag is signaled to indicate whether OBMC is applied to the current CU. For CUs with size greater than 256 luma samples or not encoded or decoded using AMVP mode, OBMC is applied by default. At the encoder, when OBMC is applied to a CU, its impact is considered during the motion estimation stage. The prediction signal formed by OBMC using the motion information of the upper and left neighboring blocks is used to compensate the upper and left boundaries of the original signal of the current CU, and then normal motion estimation processing is applied.

2.5 Example of Local Lighting Compensation (LIC)

Illumination compensation LIC is based on a linear model for illumination variations, using a scaling factor a and an offset b. It is adaptively enabled or disabled for each inter-mode coded codec unit (CU).

When LIC is applied to a CU, the least square error method is used to derive parameters a and b by using neighboring samples of the current CU and their corresponding reference samples. FIG13 shows an example of neighboring samples used to derive parameters of the IC algorithm. More specifically, as shown in FIG13, neighboring samples of subsamples (2:1 subsamples) of the CU and corresponding samples in the reference picture (identified by motion information of the current CU or sub-CU) are used. IC parameters are derived and applied to each prediction direction separately.

When encoding or decoding a CU in Merge mode, the LIC flag is copied from the neighboring block in a manner similar to the motion information copying in Merge mode; otherwise, the LIC flag is signaled to the CU to indicate whether LIC is applied.

When LIC is enabled for a picture, an additional CU-level RD check is required to determine whether LIC is applied to the CU. When LIC is enabled for a CU, the Mean-Removed Sum Of Absolute Difference (MR-SAD) and Mean-Removed Sum Of Absolute Hadamard-Transformed Difference (MR-SATD) are used instead of SAD and SATD for integer pixel motion search and fractional pixel motion search, respectively.

In order to reduce the coding complexity, the following coding scheme is applied in JEM.

When there is no significant illumination change between the current picture and its reference pictures, LIC is disabled for the entire picture. To identify this situation, the histogram of the current picture and each of the reference pictures of the current picture is calculated at the encoder. If the histogram difference between the current picture and each of the reference pictures of the current picture is less than a given threshold, LIC is disabled for the current picture; otherwise, LIC is enabled for the current picture.

2.6 Example of affine motion compensation prediction

In HEVC, only the translational motion model is applied to motion compensation prediction (MCP). However, there may be multiple motions of the camera and the object, such as zooming in/out, rotation, perspective motion, and/or other irregular motions. On the other hand, in JEM, a simplified affine transformation motion compensation prediction is applied. FIG. 14 shows an example where the affine motion field of a block 1400 is described by two control point motion vectors V ₀ and V _1. The motion vector field (MVF) of the block 1400 is described by the following equation:

As shown in Figure 14, (v _0x ,v _0y ) is the motion vector of the upper left control point, and (v _1x ,v _1y ) is the motion vector of the upper right control point.

To further simplify the motion compensation prediction, a sub-block based affine transformation prediction can be applied. The sub-block size M × N is derived as follows:

Here, MvPre is the motion vector fractional accuracy (e.g., 1/16 in JEM) and (v _2x ,v _2y ) is the motion vector of the lower left control point calculated according to Equation 1. If necessary, M and N can be adjusted down to make them divisors of w and h, respectively.

FIG15 shows an example of an affine MVF for each sub-block of block 1500. To derive the motion vector for each M×N sub-block, the motion vector of the center sample of each sub-block is calculated according to Equation 1 and rounded to the motion vector fractional accuracy (e.g., 1/16 in JEM). Then, a motion compensation interpolation filter is applied to generate a prediction for each sub-block using the derived motion vector.

After MCP, the high-precision motion vectors of each sub-block are rounded and saved with the same accuracy as the normal motion vectors.

In JEM, there are two affine motion modes: AF_INTER mode and AF_MERGE mode. For CUs with width and height greater than 8, AF_INTER mode can be applied. A CU-level affine flag is signaled in the bitstream to indicate whether AF_INTER mode is used. In AF_INTER mode, a candidate list with motion vector pairs {(v ₀ ,v ₁ ) | v ₀ ={v _A ,v _B ,v _c },v ₁ ={v _D ,v _E }} is constructed using neighboring blocks.

Figure 16 shows an example of motion vector prediction (MVP) for block 1600 in AF_INTER mode. As shown in Figure 16, _v0 is selected from the motion vectors of sub-blocks A, B, or C. The motion vectors from the neighboring blocks can be scaled according to the reference list. The motion vectors from the neighboring blocks can also be scaled according to the relationship between the picture order count (POC) used for the reference of the neighboring blocks, the POC used for the reference of the current CU, and the POC of the current CU. The method of selecting _v1 from the neighboring sub-blocks D and E is similar. If the number of candidate lists is less than 2, the list can be filled by motion vector pairs composed by copying each AMVP candidate. When the candidate list is greater than 2, the candidates can be first sorted according to adjacent motion vectors (for example, based on the similarity of the two motion vectors in the candidate pair). In some embodiments, the first two candidates are retained. In some embodiments, a rate-distortion (RD) cost check is used to determine which motion vector pair candidate is selected as the control point motion vector prediction (CPMVP) of the current CU. An index indicating the position of the CPMVP in the candidate list can be signaled in the bitstream. After determining the CPMVP of the current affine CU, affine motion estimation is applied and the control point motion vector (CPMV) is found. The difference between the CPMV and the CPMVP is then signaled in the bitstream.

When a CU is applied in AF_MERGE mode, it obtains the first block encoded or decoded in affine mode from a valid adjacent reconstructed block. FIG17A shows an example of a selection order of candidate blocks for the current CU 1700. As shown in FIG17A, the selection order may be from the left (1701), top (1702), top right (1703), bottom left (1704) to top left (1705) of the current CU 1700. FIG17B shows another example of candidate blocks for the current CU 1700 in AF_MERGE mode. If the adjacent bottom left block 1701 is encoded or decoded in affine mode, as shown in FIG17B, motion vectors _v2 , _v3 , and _v4 of the top left, top right, and bottom left corners of the CU containing block A are derived. And the motion vector v ₀ of the upper left corner of the current CU 1700 is calculated according to v ₂ , v ₃ and v _4. The motion vector v1 of the upper right corner of the current CU can be calculated accordingly.

After calculating the CPMV _v0 and _v1 of the current CU according to the affine motion model in equation (1), the MVF of the current CU can be generated. In order to identify whether the current CU is coded or decoded in AF_MERGE mode, an affine flag can be signaled in the bitstream when at least one neighboring block is coded or decoded in affine mode.

2.7 Example of Pattern Matched Motion Vector Derivation (PMMVD)

PMMVD mode is a special Merge mode based on the Frame-Rate Up Conversion (FRUC) method. In this mode, the motion information of the block is derived on the decoder side instead of signaling the motion information of the block.

When the Merge flag of a CU is true, the FRUC flag can be signaled to the CU. When the FRUC flag is false, the Merge index can be signaled and the normal Merge mode can be used. When the FRUC flag is true, an additional FRUC mode flag can be signaled to indicate which method (e.g., bilateral matching or template matching) will be used to derive the motion information of the block.

On the encoder side, the decision on whether to use the FRUC Merge mode for a CU is based on the RD cost selection made for normal Merge candidates. For example, multiple matching modes (e.g., bilateral matching and template matching) for a CU are checked by using the RD cost selection. The matching mode that causes the minimum cost is further compared with other CU modes. If the FRUC matching mode is the most effective mode, the FRUC flag is set to true for the CU and the relevant matching mode is used.

Typically, the motion inference process in FRUC Merge mode has two steps: first, a CU-level motion search is performed, and then a sub-CU-level motion refinement is performed. At the CU level, the original motion vector of the entire CU is derived based on bilateral matching or template matching. First, a list of MV candidates is generated, and the candidate that causes the minimum matching cost is selected as the starting point for further CU-level refinement. Then, a local search based on bilateral matching or template matching is performed near the starting point. The MV result with the minimum matching cost is used as the MV of the entire CU. Subsequently, the motion information is further refined at the sub-CU level with the derived CU motion vector as the starting point.

For example, the following derivation process is performed for W × H CU motion information derivation. In the first stage, the MV of the entire W × H CU is derived. In the second stage, the CU is further divided into M × M sub-CUs. The value of M is calculated as shown in (3), and D is a predefined partition depth, which is set to 3 by default in JEM. Then the MV of each sub-CU is derived.

FIG18 shows an example of bilateral matching used in the frame rate up conversion (FRUC) method. Bilateral matching is used to derive the motion information of the current CU by finding the closest match between two blocks along the motion trajectory of the current CU (1800) in two different reference pictures (1810, 1811). Under the assumption of continuous motion trajectories, the motion vectors MV0 (1801) and MV1 (1802) pointing to the two reference blocks are proportional to the temporal distance between the current picture and the two reference pictures (e.g., TD0 (1803) and TD1 (1804)). In some embodiments, when the current picture 1800 is temporally between two reference pictures (1810, 1811) and the temporal distances from the current picture to the two reference pictures are the same, the bilateral matching becomes a mirror-based bidirectional MV.

FIG. 19 shows an example of template matching used in the frame rate up conversion (FRUC) method. Template matching is used to derive motion information of the current CU 1900 by finding the closest match between the template in the current picture 1910 (the top and/or left neighboring blocks of the current CU) and the blocks in the reference picture (e.g., the same size as the template). In addition to the above-mentioned FRUC Merge mode, template matching can also be applied to the AMVP mode. As done in both JEM and HEVC, there are two candidates for AMVP. Through the template matching method, a new candidate is derived. If the candidate newly derived by template matching is different from the first existing AMVP candidate, it is inserted at the very beginning of the AMVP candidate list, and then the list size is set to 2 (e.g., by removing the second existing AMVP candidate). When applied to the AMVP mode, only CU-level search is applied.

The MV candidates set at the CU level can include the following: (1) the original AMVP candidates if the current CU is in AMVP mode, (2) all Merge candidates, (3) several MVs in the interpolated MV field (described later), and the top and left neighboring motion vectors.

When bilateral matching is used, each valid MV of the Merge candidate can be used as input to generate MV pairs assuming bilateral matching. For example, in reference list A, one valid MV of the Merge candidate is (MVa, refa). Then, the reference image refb of its paired bilateral MV is found in other reference list B, so that refa and refb are on different sides of the current image in time. If such refb is not available in reference list B, refb is determined to be a reference different from refa, and its temporal distance to the current image is the minimum in list B. After refb is determined, MVb is derived by scaling MVa based on the temporal distance between the current image refa and refb.

In some embodiments, four MVs from the interpolated MV field can also be added to the CU-level candidate list. More specifically, the interpolated MVs at positions (0,0), (W/2,0), (0, H/2), and (W/2, H/2) of the current CU are added. When FRUC is applied to AMVP mode, the original AMVP candidates are also added to the CU-level MV candidate set. In some embodiments, at the CU level, 15 MVs are added to the candidate list for AMVP CU and 13 MVs are added to the candidate list for MergeCU.

The MV candidates set at the sub-CU level include: (1) MVs determined from CU-level searches, (2) neighboring MVs at the top, left, upper left, and upper right corners, (3) scaled versions of collocated MVs from reference pictures, (4) one or more 4 ATMVP candidates (up to 4), (5) one or more STMVP candidates (e.g., up to 4). The scaled MVs from the reference picture are derived as follows. Traverse the reference pictures in the two lists. The MVs at the collocated positions of the sub-CUs in the reference picture are scaled to the reference of the starting CU-level MVs. ATMVP and STMVP candidates may be limited to the first four. At the sub-CU level, one or more MVs (e.g., up to 17) are added to the candidate list.

Generation of interpolated MV fields. Before encoding or decoding the frame, an interpolated motion field is generated for the entire picture based on one-sided ME. The motion field can then be used later as a CU-level or sub-CU-level MV candidate.

In some embodiments, the motion field of each reference picture in the two reference lists is traversed at the 4×4 block level. FIG. 20 shows an example of unilateral motion estimation (ME) 2000 in the FRUC method. For each 4×4 block, if the motion associated with the block passes through a 4×4 block in the current picture and the block is not assigned any interpolated motion, the motion of the reference block is scaled to the current picture according to the temporal distances TD0 and TD1 (in the same way as the MV scaling of TMVP in HEVC), and the scaled motion is assigned to the block in the current frame. If no scaled MV is assigned to the 4×4 block, the motion of the block is marked as unavailable in the interpolated motion field.

Interpolation and matching costs. When motion vectors point to fractional sample locations, motion compensation interpolation is required. To reduce complexity, bilinear interpolation can be used for bilateral matching and template matching instead of conventional 8-tap HEVC interpolation.

The calculation of the matching cost is slightly different at different steps. When selecting candidates from the candidate set at the CU level, the matching cost can be the absolute sum difference (SAD) of bilateral matching or template matching. After determining the starting MV, the matching cost C of the bilateral matching of the sub-CU level search is calculated as follows:

Here, w is a weighting factor. In some embodiments, w can be set to 4. MV and MVs indicate the current MV and the starting MV ^, respectively. SAD can still be used as the matching cost of template matching for sub-CU level search.

In FRUC mode, MV is derived by using luma samples only. The derived motion will be used for MC frame-to-frame prediction for both luma and chroma. After the MV is determined, the final MC is performed using an 8-tap interpolation filter for luma and a 4-tap interpolation filter for chroma.

MV refinement is a pattern-based MV search with bilateral matching cost or template matching cost as the criterion. In JEM, two search modes are supported - Unrestricted Center-Biased Diamond Search (UCBDS) and Adaptive Cross Search, which perform MV refinement at CU level and sub-CU level respectively. For both CU and sub-CU level MV refinement, MV is directly searched with quarter luma sample MV precision, and followed by one-eighth luma sample MV refinement. The search range for MV refinement for CU and sub-CU steps is set equal to 8 luma samples.

In bilateral matching Merge mode, bidirectional prediction is applied because the motion information of the CU is derived based on the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures. In template matching Merge mode, the encoder can choose from unidirectional prediction in list 0, unidirectional prediction in list 1, or bidirectional prediction for the CU. The choice can be based on the template matching cost as follows:

If costBi<=factor*min(cost0,cost1) then use bidirectional prediction; otherwise, if cost0<=cost1, use the unidirectional prediction in list 0; otherwise, use the unidirectional prediction in list 1;

Where cost0 is the SAD of template matching in list 0, cost1 is the SAD of template matching in list 1, and costBi is the SAD of template matching with bidirectional prediction. For example, when the value of factor is equal to 1.25, it means that the selection process is biased towards bidirectional prediction. Inter-frame prediction direction selection can be applied to the CU-level template matching process.

2.8 Example of Bidirectional Optical Flow (BIO)

In BIO, motion compensation is first performed to generate a first prediction of the current block (in each prediction direction). The first prediction is used to derive the spatial gradient, temporal gradient, and optical flow of each sub-block/pixel within the block, which is then used to generate a second prediction, e.g., the final prediction of the sub-block/pixel. The details are described below.

The bidirectional optical flow (BIO) method is a sample-wise motion refinement that is performed on top of block-wise motion compensation for bidirectional prediction. In some embodiments, sample-level motion refinement does not use signaling.

/

,

/

分別為I ^(k)梯度的水平分量和垂直分量。假設光流是有效的，則運動向量場(v _x ,v _y )由下式給出：

Let I ^{( k )} be the brightness value of reference k (k = 0, 1) after block motion compensation, and

/

,

/

They are the horizontal and vertical components of the gradient of I ^{( k )} respectively. Assuming that the optical flow is valid, the motion vector field ( v _x , _vy ) is given by:

/

,

/

兩者。該三階多項式在t=0時的值是BIO預測：

Combining this optical flow equation with the Hermitian interpolation of each sample's motion trajectory yields a unique third-order polynomial that ultimately matches the function value I ^{( k )} and its derivative

/

,

/

Both. The value of this third-order polynomial at t=0 is the BIO prediction:

FIG21 shows an example optical flow trajectory in the bidirectional optical flow (BIO) method. Here, τ ₀ and τ ₁ represent the distance to the reference frame, as shown in FIG21 . The distances τ ₀ and τ ₁ are calculated based on the POC of Ref0 and Ref1: τ ₀ =POC(current)-POC(Ref0), τ ₁ =POC(Ref1)-POC(current). If both predictions are from the same time direction (both from the past or both from the future), the signs are different, i.e., τ ₀ · τ ₁ <0. In this case, BIO is applied only when the predictions are not from the same time instant (i.e., τ ₀ ≠τ ₁ ), both reference regions have non-zero motion ( MVx ₀ , MVy ₀ , MVx ₁ , MVy ₁ ≠0) and the block motion vector is proportional to the temporal distance ( MVx ₀ /MVx ₁ = MVy ₀ /MVy ₁ = -τ ₀ /τ ₁ ).

The motion vector field ( vx _, vy ) is determined by minimizing the difference Δ between the values in points A and B. Figure 9 shows an example of the intersection of the motion trajectory and the reference frame plane. The model uses only the first linear term of the local Taylor expansion of _Δ :

All values in the above equation depend on the sample location, denoted as ( i',j' ). Assuming the motion is consistent in the local surrounding area, △ is minimized within a square window Ω of size (2M+1) x (2M+1) centered at the current prediction point, where M is equal to 2:

For this optimization problem, JEM uses a simplified approach, first minimizing in the vertical direction and then minimizing in the horizontal direction. This yields the following:

in,

To avoid division by zero or very small values, regularization parameters r and m can be introduced in equations 9 and 10.

r =500.4 ^{d -8} (12)

m = 700.4 ^{d -8} (13)

Here d is the bit depth of the video samples.

/

,

/

。圖22A示出了塊2200外部的存取位置的示例。如圖22A所示，在等式(9)中，以在預測塊的邊界上的當前預測點為中心的(2M+1) x (2M+1)方形窗口Ω需要存取塊外部的位置。在JEM中，將塊外部的I ^(k) ,

/

,

/

的值設置為等於塊內最近的可用值。例如，這可以實施為填充區域2201，如圖22B所示。 To keep the BIO memory accesses the same as for conventional bidirectional prediction motion compensation, all predictions and gradient values I ^{( k )} are calculated only for the position within the current block .

/

,

/

FIG. 22A shows an example of accessing locations outside of a block 2200. As shown in FIG. 22A, in equation (9), a (2M+1) x (2M+1) square window Ω centered at the current prediction point on the boundary of the prediction block needs to access locations outside the block. In JEM, I ^{( k )} outside the block is converted to

/

,

/

The value of is set equal to the nearest available value within the block. For example, this can be implemented as filling area 2201, as shown in Figure 22B.

With BIO, the motion field can be refined for each sample. To reduce the computational complexity, a block-based BIO design is used in JEM. Motion refinement can be calculated based on 4x4 blocks. In the block-based BIO, the values of _sn in Equation 9 for all samples in the 4x4 block can be aggregated, and then the aggregated value of _sn is used to derive the BIO motion vector offset for the 4x4 block. More specifically, the following formula can be used for block-based BIO derivation:

Here _bk denotes the set of samples belonging to the kth 4x4 block of the prediction block. Replace _sn in Equations 9 and 10 with (( _sn,bk )>>4) to derive the associated motion vector offset.

In some scenarios, the MV regiment of BIO may be unreliable due to noise or irregular motion. Therefore, in BIO, the size of the MV regiment is clipped by a threshold. The threshold is determined based on whether the reference images of the current image are all from one direction. For example, if all the reference images of the current image are from one direction, the value of the threshold is set to 12×2 ^{14- d} ; otherwise, it is set to 12×2 ^{13- d} .

/

，首先使用與具有去縮放偏移d-8的分數位置fracY相對應的BIOfilterS垂直插值信號。然後在水平方向上應用梯度濾波器BIOfilterG，該BIOfilterG與具有去縮放偏移18-d的分數位置fracX相對應。對於垂直梯度

/

，首先使用與具有去縮放偏移d-8的分數位置fracY相對應的BIOfilterG垂直應用梯度濾波器。然後在水平方向上使用BIOfilterS執行信號位移，該BIOfilterS與具有去縮放偏移18-d的分數位置fracX相對應。用於梯度計算的插值濾波器BIOfilterG和用於信號位移的插值濾波器BIOfilterS的長度可以較短(例如，6抽頭)，以保持合理的複雜度。表1示出了可以用於BIO中塊運動向量的不同分數位置的梯度計算的濾波器的示例。表2示出了可以用於BIO中預測信號生成的插值濾波器的示例。 The gradient of the BIO may be simultaneously calculated using motion compensated interpolation using operations consistent with the HEVC motion compensation process (e.g., 2D separable finite impulse response (FIR)). In some embodiments, the input to the 2D separable FIR is the same reference frame samples as the motion compensation process and the fractional position (fracX, fracY) according to the fractional part of the block motion vector. For the horizontal gradient

/

, the signal is first interpolated vertically using BIOfilterS corresponding to fractional position fracY with descaling offset d-8. Then a gradient filter BIOfilterG is applied horizontally, which corresponds to fractional position fracX with descaling offset 18-d. For the vertical gradient

/

, first a gradient filter is applied vertically using BIOfilterG corresponding to fractional position fracY with descaling offset d-8. Then a signal shift is performed in the horizontal direction using BIOfilterS corresponding to fractional position fracX with descaling offset 18-d. The length of the interpolation filter BIOfilterG used for gradient calculation and the interpolation filter BIOfilterS used for signal shift can be short (e.g., 6 taps) to keep the complexity reasonable. Table 1 shows examples of filters that can be used for gradient calculation of different fractional positions of block motion vectors in BIO. Table 2 shows examples of interpolation filters that can be used for prediction signal generation in BIO.

Table 1: Example filters for gradient calculation in BIO

In JEM, BIO can be applied to all dual prediction blocks when the two predictions come from different reference images. BIO can be disabled when Local Illumination Compensation (LIC) is enabled for a CU.

In some embodiments, OBMC is applied to a block after the normal MC process. To reduce computational complexity, BIO may not be applied during the OBMC process. This means that BIO is applied to the MC process of a block only when its own MV is used, and is not applied to the MC process when the MV of a neighboring block is used during the OBMC process.

2.9 Example of Decoder-side Motion Vector Refinement (DMVR)

In the bidirectional prediction operation, for the prediction of a block area, two prediction blocks formed by the motion vector (MV) of list0 and the MV of list1 are combined to form a single prediction signal. In the decoder-side motion vector refinement (DMVR) method, the two motion vectors of the bidirectional prediction are further refined by the bilateral template matching process. Bilateral template matching is applied in the decoder to perform a distortion-based search between the bilateral template and the reconstructed samples in the reference picture in order to obtain the refined MV without transmitting additional motion information.

In DMVR, a bilateral template is generated as a weighted combination (i.e., average) of two prediction blocks, from the original MV0 of list 0 and MV1 of list 1, respectively, as shown in Figure 23. The template matching operation includes calculating a cost metric between the generated template and the sample area (around the original prediction block) in the reference image. For each of the two reference images, the MV that produces the minimum template cost is considered as the updated MV of the list to replace the original MV. In JEM, nine MV candidates are searched for each list. The nine MV candidates include the original MV and 8 surrounding MVs that have a brightness sample offset in the horizontal or vertical direction or in both directions from the original MV. Finally, two new MVs, namely MV0' and MV1' as shown in Figure 23, are used to generate the final bilateral prediction result. The sum of absolute differences (SAD) is used as a cost metric.

Apply DMVR to the Merge mode of bidirectional prediction, where one MV comes from the past reference picture and the other MV comes from the future reference picture, without transmitting additional syntax elements. In JEM, DMVR is not applied when LIC, affine motion, FRUC, or sub-CU Merge candidate is enabled for a CU.

3 Examples of CABAC modifications

In JEM, CABAC includes the following three major changes compared to the design in HEVC: context modeling for the modification of transform coefficients; multi-hypothesis probability estimation with context-dependent update speed; adaptive initialization for the context model.

3.1 Example of context modeling for transformation coefficients

In HEVC, the transform coefficients of a codec block are encoded and decoded using non-overlapping coefficient groups (CGs), and each CG contains a 4×4 block of coefficients of the codec block. The CGs within a codec block and the transform coefficients within the CGs are encoded and decoded according to a predefined scanning order. The encoding and decoding of the transform coefficient level of a CG with at least one non-zero transform coefficient can be divided into multiple scanning channels. In the first channel, the first binary symbol (represented by bin0, also called significant_coeff_flag, which indicates that the size of the coefficient is greater than 0) is encoded and decoded. Next, two passes for context encoding/decoding the second/third binary symbol (bin) can be applied (represented by bin1 and bin2, also called coeff_abs_greater1_flag and coeff_abs_greater2_flag, respectively). Finally, if necessary, two more passes for encoding/decoding symbol information and the residual value of the coefficient level (also called coeff_abs_level_remaining) are called. Only the binary symbols in the first three passes are encoded/decoded in the normal mode, and these binary symbols are called normal binary symbols in the following description.

In JEM, the context modeling of conventional binary symbols is changed. When binary symbol i is encoded and decoded in the i-th scanning channel (i is 0, 1, 2), the context index depends on the value of the i-th binary symbol of the previous encoded and decoded coefficients in the neighborhood covered by the local template. Specifically, the context index is determined based on the sum of the i-th binary symbols of the neighboring coefficients.

As shown in Figure 24, the local template contains up to five spatially neighboring transform coefficients, where x represents the position of the current transform coefficient and xi (i is 0 to 4) indicates its five neighbors. In order to capture the characteristics of transform coefficients at different frequencies, a codec block can be divided into up to three regions, and the division method is fixed regardless of the codec block size. For example, when bin0 of the luma transform coefficient is encoded and decoded, as shown in Figure 24, a codec block is divided into three regions marked with different colors, and the context index assigned to each region is listed. Luma and chroma components are processed in a similar manner, but with separate context model sets. In addition, the context model selection for bin0 of the luma component (e.g., valid mark) also depends on the transform size.

3.2 Example of multiple hypothesis probability estimation

The binary arithmetic codec applies a "multi-hypothesis" probability update model based on the two probability estimates _P0 and _P1 associated with each context model, and updates independently at different adaptive rates as follows:

和

(j=0,1)分別表示解碼二元位之前和之後的概率。變量M_i(為4、5、6、7)是控制索引等於i的上下文模型的概率更新速度的參數；並且k表示概率的精度(這裡等於15)。 in

and

(j=0,1) represent the probabilities before and after the decoded binary bit, respectively. The variable _Mi (4, 5, 6, 7) is a parameter that controls the probability update speed of the context model with index equal to i; and k represents the precision of the probability (here equal to 15).

The probability estimate P for the interval refinement used in the binary arithmetic codec is the mean of the estimates from two assumptions:

In JEM, the value of the parameter _Mi used in equation (15) that controls the probability update speed of each context model is assigned as follows.

On the encoder side, the codec binary associated with each context model is recorded. After encoding and decoding a slice, for each context model with index equal to i, the rate cost using different _Mi values (4, 5, 6, 7) is calculated and the one that provides the minimum rate cost is selected. For simplicity, this selection process is performed only when a new combination of slice type and slice-level quantization parameter is encountered.

A 1-bit flag is signaled for each context model i to indicate whether _Mi is different from the default value of 4. When the flag is 1, two bits are used to indicate whether _Mi is equal to 5, 6 or 7.

3.3 Initialization of context model

Instead of using a fixed table for context model initialization in HEVC, the initial probability states of the context models for inter-coded slices can be initialized by copying the states from the previously coded pictures. More specifically, after coding the center-located CTU of each picture, the probability states of all context models are stored to be used as the initial states of the corresponding context models on subsequent pictures. In JEM, the initial set of states for each inter-coded slice is copied from the stored states of the previously coded picture with the same slice type and the same slice level QP as the current slice. This lacks loss robustness but is used in the current JEM scheme for codec efficiency experimentation purposes.

4 Examples of related embodiments and methods

Methods related to the disclosed technology include extended LAMVR in which the supported motion vector resolution ranges from 1/4 pixel to 4 pixels (1/4 pixel, 1/2 pixel, 1-pixel, 2-pixel, and 4 pixels). When MVD information is signaled, information about the motion vector resolution is signaled at the CU level.

Depending on the resolution of the CU, the motion vector (MV) and motion vector prediction (MVP) of the CU are adjusted. If the applied motion vector resolution is denoted as R (R can be ¼, ½, 1, 2, 4), the MV (MV _x , MV _y ) and MVP (MVP _x , MVP _y ) are expressed as follows: (MV _x , MV _y ) = (Round (MV _x / (R * 4)) * (R * 4), Round (MV _y / (R * 4)) * (R * 4)) (17) (MVP _x , MVP _y ) = (Round (MVP _x / (R * 4)) * (R * 4), Round (MVP _y / (R * 4)) * (R * 4)) (18)

Since both the motion vector prediction and the MV are adjusted by adaptive resolution, the MVD ( _MVDx , _MVDy ) is also aligned with the resolution and is signaled according to the resolution as follows: ( _MVDx , _MVDy ) = (( _MVx - _MVPx ) / (R*4), (MVy _{- MVPy} ) / R*4)) (19)

In this proposal, the motion vector resolution index (MVR index) indicates the MVP index as well as the motion vector resolution. As a result, the proposed method has no MVP index signaling. The following table shows what each value of the MVR index represents.

In case of bidirectional prediction, AMVR has 3 modes for each resolution. AMVR Bi-Index indicates whether to signal MVDx, MVDy for each reference list (list 0 or list 1). An example definition of AMVR Bi-Index is shown in the following table.

5. Disadvantages of existing implementation methods

In one prior implementation using BIO, the MV calculated between the reference block/subblock in list 0 (denoted by refblk0) and the reference block/subblock in list 1 (refblk1), denoted by (v _x , _vy ), is only used for motion compensation of the current block/subblock, but not for motion prediction, deblocking, OBMC, etc. of future coded blocks, which may be inefficient. For example, (v _x , _vy ) may be generated for each subblock/pixel of the block, and formula (7) may be used to generate a second prediction of the subblock/pixel. However, (v _x , _vy ) is not used for motion compensation of the subblock/pixel, which may also be inefficient.

In another existing implementation method of using DMVR and BIO for bidirectional prediction of PU, first, DMVR is executed. After that, the motion information of PU is updated. Then, BIO is executed using the updated motion information. That is, the input of BIO depends on the output of DMVR.

In another existing implementation using OBMC, for AMVP mode, for small blocks (width*height<=256), whether to enable OBMC is determined at the encoder and the decoder is notified by signaling. This increases the complexity of the encoder. At the same time, for a given block/subblock, when OBMC is enabled, it is always applied to both luminance and chrominance, which may lead to decreased encoding and decoding efficiency.

In another existing implementation using AF_INTER mode, MVD needs to be encoded and decoded, however, it can only be encoded and decoded with 1/4 pixel accuracy, which may be inefficient.

6. Example method for two-step inter-frame prediction for visual media encoding and decoding

Embodiments of the presently disclosed technology overcome the shortcomings of existing implementations, and provide additional solutions to provide video codecs with higher codec efficiency. Based on the disclosed technology, two-step inter-frame prediction can enhance existing and future video codec standards, as illustrated in the examples described below for various implementations. The examples of the disclosed technology provided below illustrate the general concepts and are not meant to be interpreted as limiting. In the examples, the various features described in these examples can be combined unless explicitly indicated to the contrary.

Regarding terminology, the reference pictures of the current picture from list 0 and list 1 are denoted as Ref0 and Ref1, respectively. Denote τ ₀ =POC(current)-POC(Ref0), τ ₁ =POC(Ref1)-POC(current), and denote the reference blocks of the current block from Ref0 and Ref1 as refblk0 and refblk1, respectively. For a subblock in the current block, the original MV of its corresponding subblock in refblk0 pointing to refblk1 is denoted as (v _x ,v _y ). The MVs of the subblocks in Ref0 and Ref1 are denoted by (mvL0 _x ,mvL0 _y ) and (mvL1 _x ,mvL1 _y ), respectively. The derived MV derived from the original MV in BIO is denoted by (v _x ,v _y ). As described in this patent document, the updated motion vector based method for motion prediction can be extended to existing and future video coding standards.

Example 1. MV(v _x ,v _y ) and MV(mvLX _x ,mvLX _y ), where X = 0 or 1, should be scaled to the same precision before the addition operation, such as before performing the techniques in Example 1(e) and/or Example 2.

(a) In one example, the target precision (to be scaled to) is set to the higher (for better performance)/lower (for lower complexity) precision between MV(v _x , _vy ) and MV(mvLX _x ,mvLX _y ). Alternatively, the target precision (to be scaled to) is set to a fixed value (e.g., 1/32 pixel precision) regardless of the precision of the two MVs.

(b) In one example, the original MV (mvLX _x , mvLX _y ) can be scaled to a higher precision before the addition operation, for example, it can be scaled from 1/4 pixel precision to 1/16 pixel precision. In this case, mvLX _x = sign(mvLX _x ) * (abs(mvLX _x )<<N), mvLX _y = sign(mvLX _y ) * (abs(mvLX _y )<<N), where the function sign(.) returns the sign of the input parameter (as shown below), and the function abs(.) returns the absolute value of the input parameter, and N=log2(curr_mv_precision/targ_mv_precision), and curr_mv_precision and targ_mv_precision are the current MV precision and the target MV precision, respectively. For example, if the MV is scaled from 1/4 pixel precision to 1/16 pixel precision, then N=log2((1/4)/(1/16))=2.

(i) Alternatively, mvLX _x =mvLX _x <<N, mvLX _y =mvLX _y <<N.

(ii) Alternatively, mvLX _x =mvLX _x <<(N+K), mvLX _y =mvLX _y <<(N+K).

(iii) Alternatively, mvLX _x =sign(mvLX _x ) * (abs(mvLX _x )<<(N+K)), mvLX _y =sign(mvLX _y ) * (abs(mvLX _y )<<(N+K)).

(iv) Similarly, if MV( _vx , _vy ) needs to be scaled to lower precision, the scaling process specified in Example 1(d) can be applied.

(c) In one example, if the precision of MV(v _x ,v _y ) is lower/higher than the precision of MV(mvLX _x ,mvLX _y ), then MV(v _x ,v _y ) should be scaled to a finer/coarser precision. For example, if MV(mvLX _x ,mvLX _y ) has 1/16 pixel precision, then MV(v _x ,v _y ) should also be scaled to 1/16 pixel precision.

(d) If (v _x ,v _y ) needs to be right shifted (i.e., scaled to lower precision) by N places to obtain the same precision as (mvLX _x ,mvLX _y ), then v _x =(v _x +offset)>>N, v _y =(v _y +offset)>>N, where, for example, offset=1<<(N-1).

(i) Alternatively, v _x =sign(v _x ) * ((abs(v _x )+offset)>>N), _vy =sign( _vy ) * ((abs( _vy )+offset)>>N).

(ii) Similarly, if MV(mvLX _x ,mvLX _y ) needs to be scaled to higher precision, the above scaling process as specified in Example 1(b) can be applied.

(e) In one example, it is proposed to scale the MV ( _vx , _vy ) derived in the BIO and add it to the original MV ( _mvLXx , _mvLXy ) (X = 0 or 1) of the current block/subblock. The updated MVs are calculated as: _mvL0'x = _-vx * ( _τ0 /( _τ0 + _τ1 ))+ _mvL0x , mvL0'y = _-vy * ( _τ0 /( _τ0 + _τ1 ))+ _mvL0y , and _mvL1'x = _vx * ( _τ1 /( _τ0 + _τ1 ) ₎ + _mvL1x , _mvL1'y = _vy * ( _τ1 /( _τ0 + _τ1 ))+ _mvL1y .

(i) In one example, the updated MV is used for future motion prediction (such as in AMVP, Merge, and Affine modes), deblocking, OBMC, etc.

(ii) Alternatively, the updated MV can only be used for motion prediction of its non-immediately following CU/PU in decoding order.

(iii) Alternatively, the updated MV can only be used as TMVP in AMVP, Merge or Affine mode.

(f) If (v _x ,v _y ) needs to be right shifted (i.e., scaled to lower precision) by N places to obtain the same precision as (mvLX _x ,mvLX _y ), then v _x =(v _x +offset)>>(N+ K), v _y =(v _y +offset)>>(N+K), where, for example, offset=1<<(N+K-1). K is an integer, for example, K is equal to 1, 2, 3, -2, -1, or 0.

(i) Alternatively, v _x =sign(v _x ) * ((abs(v _x )+offset)>>(N+K)), _vy =sign( _vy ) * ((abs( _vy )+offset)>>(N+K)), where, for example, offset=1<<(N+K-1).

Example 2. Instead of considering the POC distance (e.g., in the calculation of τ ₀ and τ ₁ as described above), the scaling method of the MV called in the BIO process can be simplified.

(a) mvL0' _x = -v _x /S ₀ +mvL0 _x , mvL0' _y = -v _y /S ₀ +mvL0 _y , and/or mvL1' _x = v _x /S ₁ +mvL1 _x , mvL1' _y = v _y /S ₁ +mvL1 _y . In one example, S ₀ and/or S ₁ are set to 2. In one example, it is called under certain conditions (such as τ ₀ >0 and τ ₁ >0).

(i) Alternatively, an offset may be added during the segmentation process. For example, mvL0' _x =(-v _x +offset0)/S ₀ +mvL0 _x , mvL0' _y =-(v _y +offset0)/S ₀ +mvL0 _y , and/or mvL1' _x =(v _x +offset1)/S ₁ +mvL1 _x , mvL1' _y =(v _y +offset1)/S ₁ +mvL1 _y . In one example, offset0 is set to S ₀ /2 and offset1 is set to S ₁ /2.

(ii) In one example, mvL0' _x =((-v _x +1)>>1)+mvL0 _x , mvL0' _y =(-(v _y +1)>>1)+mvL0 _y , and/or mvL1' _x =((v _x +1)>>1)+mvL1 _x , mvL1' _y =((v _y +1)>>1)+mvL1 _y .

(b)mvL0' _x =-SF ₀ * v _x +mvL0 _x , mvL0' _y =-v _y * SF ₀ +mvL0 _y , and/or mvL1' _x =-SF ₁ * v _x +mvL1 _x , mvL1' _y =-SF ₁ *v _y +mvL1 _y . In one example, SF ₀ is set to 2, and/or SF ₁ is set to 1. In one example, it is called under certain conditions (such as τ ₀ >0 and τ ₁ <0 and τ ₀ >|τ ₁ |), as shown in Figure 25(b).

(c)mvL0' _x =SFACT ₀ *v _x +mvL0 _x , mvL0' _y =SFACT ₀ *v _y +mvL0 _y , and/or mvL1' _x =SFACT ₁ *v _x +mvL1 _x , mvL1' _y =SFACT ₁ *v _y +mvL1 _y . In one example, SFACT ₀ is set to 1, and/or SFACT ₁ is set to 2. In one example, it is called under certain conditions (such as τ ₀ >0 and τ ₁ <0 and τ ₀ < |τ ₁ |), as shown in Figure 25(c).

Example 3. When τ ₀ >0 and τ ₁ >0, the derivation of (v _x ,v _y ) and the update of (mvLX _x ,mvLX _y ) can be done together to maintain high accuracy.

(a) In one example, if (v _x ,v _y ) needs to be right-shifted (i.e., scaled to lower precision) by N places to obtain the same precision as (mvLX _x ,mvLX _y ), then mvL0' _x =((-v _x +offset)>>(N+1))+mvL0 _x , mvL0' _y =((-v _y +offset)>>(N+1))+mvL0 _y , mvL1' _x =((v _x +offset)>>(N+1))+mvL1 _x , mvL1' _y =((v _y +offset)>>(N+1))+mvL1 _y , where, for example, offset=1<<N.

(b) In an example, if (v _x ,v _y ) needs to be right shifted (i.e., scaled to lower precision) by N bits to obtain the same precision as (mvLX _x ,mvLX _y ), then mvL0' _x =((-v _x +offset)>>(N+K+1))+mvL0 _x , mvL0' _y =((-v _y +offset)>>(N+K+1))+mvL0 _y , mvL1' _x =((v _x +offset)>>(N+K+1))+mvL1 _x , mvL1' _y =((v _y +offset)>>(N+K+1))+mvL1 _y , where, for example, offset=1<<(N+K). K is an integer, for example, K is equal to 1, 2, 3, -2, -1, or 0.

(c) Alternatively, mvL0' _x = -sign(v _x ) * ((abs(v _x )+offset) >> (N+1))+mvL0 _x , mvL0' _y = -sign(v _y ) * ((abs(v _y )+offset) >> (N+1))+mvL0 _y , mvL1' _x = sign(v _x ) * ((abs(v _x )+offset) >> (N+1))+mvL1 _x , mvL1' _y = sign(v _y ) * ((abs(v _y )+offset) >> (N+1))+mvL1 _y .

(d) Alternatively, mvL0' _x =-sign(v _x ) * ((abs(v _x )+offset)>>(N+K+1))+mvL0 _x , mvL0' _y =-sign(v _y ) * ((abs(v _y )+offset)>>(N+K+1))+mvL0 _y , mvL1' _x =sign(v _x ) * ((abs(v _x )+offset)>>(N+K+1))+mvL1 _x , mvL1' _y =sign(v _y ) * ((abs(v _y )+offset)>>(N+K+1))+mvL1 _y , where, for example, offset=1<<(N+K). K is an integer, for example, K is equal to 1, 2, 3, -2, -1 or 0.

Example 4. The cropping operation can be further applied to updated MVs adopted in BIO and/or DMVR or other types of coding and decoding methods that may require updating MVs.

(a) In one example, the updated MV is cropped in the same way as other traditional MVs, such as cropping to a specific range compared to the image boundaries.

(b) Alternatively, the updated MV is clipped within a certain range (or multiple ranges for different sub-blocks) compared to the MV used in the MC process. That is, the difference between the MV used in the MC and the updated MV is clipped within a certain range (or multiple ranges for different sub-blocks).

Example 5. The use of updated MVs called in BIOs and/or other types of codec methods that may require updating MVs can be constrained.

(a) In one example, the updated MV is used for future motion prediction (such as in AMVP, Merge and/or Affine modes), deblocking, OBMC, etc. Alternatively, the updated MV can be used in a first module and the original MV can be used in a second module. For example, the first module is motion prediction and the second module is deblocking.

(i) In one example, future motion prediction refers to motion prediction in blocks to be encoded/decoded after the current block in the current picture or slice.

(ii) Alternatively, future motion prediction refers to the prediction of motion in pictures or slices to be encoded/decoded after the current picture or slice.

(b) Alternatively, the updated MV can only be used for motion prediction of its non-immediately following CU/PU in decoding order.

(c) The updated MV should not be used for motion prediction of the next CU/PU in decoding order.

(d) Alternatively, the updated MV may be used only as a predictor for encoding and decoding subsequent pictures/slices/strips, such as TMVP in AMVP, and/or Merge and/or Affine modes.

(e) Alternatively, the updated MV may only be used as a predictor for encoding and decoding subsequent pictures/slices/strips, such as ATMVP and/or STMVP, etc.

Example 6. In one example, a two-step inter-frame prediction process is proposed, in which a first step is performed to generate some intermediate predictions (first predictions) based on signaled/derived motion information associated with the current block, and a second step is performed to derive a final prediction (second prediction) for the current block based on updated motion information that may depend on the intermediate predictions.

(a) In one example, the BIO process (i.e., using the signaled/derived motion information used to generate the first prediction and the spatial gradient, temporal gradient, and optical flow for each sub-block/pixel within the block) is only used to derive the updated MV as specified in Example 1 (and formula (7) is not applied to generate the second prediction), and then the updated MV is used to perform motion compensation and generate the second prediction (i.e., the final prediction) for each sub-block/pixel within the block.

(b) In one example, an interpolation filter different from the interpolation filter used for inter-frame coded blocks that are not coded in this way may be used in the first or/and second steps to reduce the memory bandwidth.

(i) In one example, a shorter tap filter (such as a 6-tap filter, a 4-tap filter, or a bilinear filter) may be used.

(ii) Alternatively, the filters utilized in the first/second step may be predefined (e.g. filter taps, filter coefficients).

(iii) Alternatively, in addition, the filter taps selected for the first and/or second step may depend on codec information such as block size/block shape (square, non-square, etc.)/strip type/prediction direction (unidirectional or bidirectional prediction or multiple hypotheses, forward or backward).

(iv) Alternatively, furthermore, different blocks may select different filters for the first/second step. In one example, one or more candidate sets of multiple filters may be predefined or signaled. The block may select from the candidate set. The selected filter may be indicated by a signaled index or may be derived on the fly without signaling.

(c) In one example, only integer MVs are used when generating the first prediction, and no interpolation filtering process is applied in the first step.

(i) In one example, a fractional MV is rounded to the nearest integer MV.

(1) If there is more than one nearest integer MV, the fractional MV is rounded to the smaller nearest integer MV.

(2) If there is more than one nearest integer MV, the fractional MV is rounded to the larger nearest integer MV.

(3) If there is more than one nearest integer MV, the fractional MV is rounded to the nearest MV that is closer to zero.

(ii) In one example, a fractional MV is rounded to the nearest integer MV that is not less than the fractional MV.

(iii) In one example, a fractional MV is rounded to the nearest integer MV that is not greater than the fractional MV.

(d) The use of this method can be signaled in the SPS, PPS, slice header, CTU or CU or CTU group.

(e) The use of this method may also depend on codec information such as block size/block shape (square, non-square, etc.)/strip type/prediction direction (unidirectional or bidirectional prediction or multiple hypotheses, forward or backward).

(i) In one example, this method can be automatically disabled under certain conditions, for example, when the current block is encoded or decoded in affine mode, this method can be disabled.

(ii) In one example, this method can be automatically applied under certain conditions, such as when a block is encoded or decoded with bi-prediction and the block size is larger than a threshold (e.g., more than 16 samples).

Example 7. In one example, it is proposed that before calculating the time gradient in BIO, the reference block (or prediction block) can be modified first, and the calculation of the time gradient is based on the modified reference block.

(a) In one example, the mean is removed for all reference bins.

(i) For example, for a reference block X (X=0 or 1), first, the mean (denoted by MeanX) is calculated for the block, and then MeanX is subtracted from each pixel in the reference block.

(ii) Alternatively, for different reference image lists, one can decide whether to remove the mean or not. For example, for one reference block/subblock, the mean is removed before computing the temporal gradient, while for another reference block/subblock, the mean is not removed.

(iii) Alternatively, different reference blocks (e.g., 3 or 4 reference blocks utilized in multi-hypothesis prediction) can choose whether to be modified first.

(b) In one example, the mean is defined as the average of the samples selected in the reference block.

(c) In one example, all pixels in reference block X or a sub-block of reference block X are used to calculate MeanX.

(d) In one example, only some of the pixels in the reference block X or a sub-block of the reference block are used to calculate MeanX. For example, only every second row/column of pixels are used.

(i) Alternatively, in the example, only every fourth row/column of pixels is used to calculate MeanX.

(ii) Alternatively, use only the four corner pixels to compute MeanX.

(iii) Alternatively, only the four corner pixels and the center pixel, e.g., the pixel at position (W/2, H/2) (where W×H is the reference block size), are used to compute MeanX.

(e) In one example, the reference block can be first filtered before being used to derive the temporal gradient.

(i) In one example, a smoothing filtering method may be first applied to the reference block.

(ii) In one example, pixels at block boundaries are first filtered.

(iii) In one example, overlapping block motion compensation (OBMC) is first applied before deriving the temporal gradient.

(iv) In one example, illumination compensation (IC) is first applied before deriving the temporal gradient.

(v) In one example, weighted predictions are first applied before deriving temporal gradients.

(f) In one example, the temporal gradient is first calculated and then modified. For example, the temporal gradient is further subtracted by the difference between Mean0 and Mean1.

Example 8. In one example, signaling can be sent from an encoder to a decoder, such as in a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a slice, a CTU, or a CU, to inform whether to update the MV used for a BIO codec block and/or to use the updated MV for future motion prediction and/or how to use the updated MV for future motion prediction.

Example 9. In one example, it is proposed to add constraints to the motion vectors used in the BIO process.

(a) In one example, (v _x , _vy ) is constrained to a given range, -M _x <v _x <N _x , and/or -M _y < _vy <N _y , where M _x ,N _x ,My _y ,N _y are non-negative integers and may be equal to 32, for example.

(b) In one example, the updated MV of the BIO codec sub-block/BIO codec block is constrained to a given range, such as -M _L0x <mvL0' _x <N _L0x and/or -M _L1x <mvL1' _x <N _L1x ,-M _L0y <mvL0' _y <N _L0y and/or -M _L1y <mvL1' _y <N _L1y , where M _L0x ,N _L0x ,M L1x ,N _L1x ,M _L0y ,N _L0y ,M _L1y ,N _L1y are non-negative _integers and may be, for example, equal to 1024, 2048, etc.

Example 10. It is proposed that for BIO, DMVR, FRUC, template matching, or other methods that require content update MV (or motion information including MV and/or reference picture) derived from the bitstream, the use of updated motion information may be subject to constraints.

(a) In one example, even if the motion information is updated at the block level, the updated and non-updated motion information may be stored differently for different sub-blocks. In one example, the updated motion information of some sub-blocks may be stored, and for the other remaining sub-blocks, the non-updated motion information may be stored.

(b) In one example, if MV (or motion information) is updated at the sub-block/block level, the updated MV is stored only for the internal sub-blocks (i.e., the sub-blocks not at the PU/CU/CTU boundary), and then, as shown in Figures 26A and 26B, used for motion prediction, deblocking, OBMC, etc. Alternatively, the updated MV is stored only for the boundary sub-blocks.

(c) In one example, if the neighboring block and the current block are not in the same CTU or the same region with a size such as 64×64 or 32×32, the updated motion information from the neighboring block is not used.

(i) In one example, if the neighboring block and the current block are not in the same CTU or the same region with a size such as 64×64 or 32×32, the neighboring block is considered "unavailable".

(ii) Alternatively, if the neighboring block and the current block are not in the same CTU or the same region with a size such as 64×64 or 32×32, the current block uses motion information without an update process.

(d) In one example, if the neighboring block and the current block are not in the same CTU column or the same column of a region having a size such as 64×64 or 32×32, the updated MV from the neighboring block is not used.

(i) In one example, if the neighboring block and the current block are not in the same CTU column or the same column of a region having a size such as 64×64 or 32×32, the neighboring block is considered "unavailable".

(ii) Alternatively, if the neighboring block and the current block are not in the same CTU column or the same column of a region having a size such as 64×64 or 32×32, the current block uses motion information without an update process.

(e) In one example, if the bottommost column of a block is a CTU or a bottommost column of a region having a size such as 64×64 or 32×32, the motion information of the block is not updated.

(f) In one example, if the rightmost row of a block is a CTU or the rightmost row of a region having a size such as 64×64 or 32×32, the motion information of the block is not updated.

(g) In one example, refined motion information from some neighboring CTUs or regions is used for the current CTU, and unrefined motion information from other neighboring CTUs or regions is used for the current CTU.

(i) In one example, refined motion information from the left CTU or left region is used for the current CTU.

(ii) Alternatively, additionally, refined motion information from the upper left CTU or upper left region is used for the current CTU.

(iii) Alternatively, additionally, refined motion information from the upper CTU or upper region is used for the current CTU.

(iv) Alternatively, in addition, refined motion information from the upper right CTU or upper right region is used for the current CTU.

(v) In one example, the region has a size such as 64×64 or 32×32.

Example 11. In one example, it is proposed that different MVD precisions can be used in AF_INTER mode, and syntax elements can be signaled to indicate the MVD precision for each block/CU/PU. Precision sets including multiple different MVD precisions forming a geometric sequence are allowed.

(a) In one example, {1/4,1,4} pixel MVD accuracy is allowed.

(b) In one example, {1/4,1/2,1,2,4} pixel MVD accuracy is allowed.

(c) In one example, {1/16, 1/8, 1/4} pixel MVD accuracy is allowed.

(d) The syntax element is present under further conditions such as when there is a non-zero MVD component of the block/CU/PU.

(e) In one example, MVD accuracy information is always signaled regardless of whether there are any non-zero MVD components.

(f) Alternatively, for 4/6 parameter AF_INTER mode, where 2/3MVD is encoded or decoded, different MVD precisions may be used for 2/3 MVD (1 MVD per control point in unidirectional prediction, 2 MVDs per control point in bidirectional prediction, i.e. 1 MVD per control point in each prediction direction), and the 2/3 control points are associated with different MVD precisions. In this case, in addition, the 2/3 syntax element may be signaled to indicate the MVD precision.

(g) In one example, the method described in PCT/CN2018/091792 can be used to encode and decode MVD precision in AF_INTER mode.

Example 12. In one example, it is proposed that if more than one DMVD method is executed on a block (e.g., PU), different decoder-side motion vector derivation (DMVD) methods such as BIO, DMVR, FRUC, and template matching work independently, i.e., the input of a DMVD method does not depend on the output of another DMVD method.

(a) In one example, in addition, a prediction block and/or a set of updated motion information (e.g., motion vectors and reference pictures for each prediction direction) is generated from the set motion information derived by multiple DMVD methods.

(b) In one example, motion compensation is performed using the derived motion information from each DMVD method, and they are averaged or weighted averaged or filtered (e.g., through a median filter) to generate a final prediction.

(c) In one example, the motion information derived by all DMVD methods is averaged or weighted averaged or filtered (such as through a median filter) to generate the final motion information. Alternatively, different priorities are assigned to different DMVD methods, and the motion information derived by the method with the highest priority is selected as the final motion information. For example, when BIO and DMVR are performed on a PU, the motion information generated by DMVR is used as the final motion information.

(d) In one example, for a PU, no more than N DMVD methods are allowed, where N>=1.

(i) Assign different priorities to different DMVD methods, and execute the effective methods with the highest N priorities.

(e) The DMVD methods are executed in a simultaneous manner. The updated MV of one DMVD method is not input as the starting point of the next DMVD method. For all DMVD methods, the non-updated MV is input as the search starting point. Alternatively, the DMVD methods are executed in a cascaded manner. The updated MV of one DMVD method is input as the search starting point of the next DMVD method.

Other implementation options

This section describes a method for refining MVs and storing them for further use in BIO-encoded blocks. The refined MVs can be used for motion vector prediction of subsequent blocks within the current slice/CTU row/slice, and/or for filtering (e.g., deblocking filter processing) and/or motion vector prediction of blocks located at different pictures.

As shown in FIG. 32 , the derived motion vector pointing from the sub-block in reference block 0 to the sub-block in reference block 1 (denoted by (DMV _x ,DMV _y )) is used to further improve the prediction of the current sub-block.

It is proposed to further refine the motion vector of each sub-block by using the derived motion vector in BIO. Denote the POC distance (e.g., absolute POC difference) between the LX reference picture and the current picture as deltaPOCX, and denote (MVLX _x ,MVLX _y ) and (MVLX _x ',MVLX _y ') as the signaled and refined motion vectors of the current sub-block, where X=0 or 1. Then (MVLX _x ',MVLX _y ') is calculated as follows:

However, multiplication and division are required in the above equation. To solve this problem, the derivation of the refined motion vector is simplified as follows:

In some embodiments, this method is only used when the current CU is predicted from the previous and next pictures, so it only operates in a random access (RA) configuration.

Example 13. The proposed method can be applied under certain conditions, such as block size, stripe/picture/slice type.

(a) In one example, when the block size contains samples smaller than M*H, such as 16 or 32 or 64 luma samples, the above method is not allowed.

(b) Alternatively, when the minimum dimension of the width or height of the block is less than or not greater than X, the above method is not allowed. In one example, X is set to 8.

(c) Alternatively, when the width of the block is >th1 or >=th1 and/or the height of the block is >th2 or >=th2, the above method is not allowed. In one example, X is set to 8.

(d) Alternatively, when the width of the block is <th1 or <=th1 and/or the height of the block is <th2 or <=th2, the above method is not allowed. In one example, X is set to 8.

Example 14. The above method can be applied at the sub-block level.

(a) In one example, the BIO update process, or the two-step frame-to-frame prediction process, or the temporal gradient derivation method described in Example 7 may be called for each sub-block.

(b) In one example, when the width or height or both the width and height of a block are greater than (or equal to) a threshold value L, the block may be split into multiple sub-blocks. Each sub-block is processed in the same manner as a normal codec block with sub-block size.

Example 15. The threshold can be predefined or signaled at SPS/PPS/picture/slice/slice level.

(a) Alternatively, the threshold may depend on some codec information, such as block size, picture type, temporal layer index, etc.

The examples described above may be combined in the context of the methods described below, e.g., methods 2700-3100, 3300-3600, and 3800-4200, which may be implemented at a video decoder.

FIG27 illustrates a flow chart of an example method for video decoding. Method 2700 includes, at step 2710, receiving a bitstream representation of a current block of video data.

The method 2700 includes, at step 2720, generating updated first and second reference motion vectors based on a weighted sum of the first scaled motion vector and the first and second scaled reference motion vectors, respectively. In some embodiments, the first scaled motion vector is generated by scaling the first motion vector to a target precision, and wherein the first and second scaled reference motion vectors are generated by scaling the first and second reference motion vectors to the target precision, respectively. In some embodiments, the first motion vector is derived based on a first reference motion vector from a first reference block and a second reference motion vector from a second reference block, and wherein the current block is associated with the first and second reference blocks.

In some embodiments, an indication of the target accuracy is signaled in a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, a codec tree unit (CTU), or a codec unit (CU).

In some embodiments, the first motion vector has a first precision, and the first and second reference motion vectors have a reference precision. In other embodiments, the first precision may be higher or lower than the reference precision. In other embodiments, the target precision may be set to the first precision, the reference precision, or a fixed (or predetermined) precision that is independent of the first precision and the reference precision.

In some embodiments, the first motion vector is derived based on bidirectional optical flow (BIO) refinement using the first and second reference motion vectors.

The method 2700 includes, at step 2730, processing the bitstream representation based on the updated first and second reference motion vectors to generate the current block. In some embodiments, the processing is based on bidirectional optical flow (BIO) refinement or decoder-side motion vector refinement (DMVR), and wherein the updated first and second reference motion vectors are clipped prior to processing.

In some embodiments, processing is based on bidirectional optical flow (BIO) refinement, and the updated first and second reference motion vectors are constrained to a predetermined value range prior to processing.

In some embodiments, the processing is based on bidirectional optical flow (BIO) refinement, decoder-side motion vector refinement (DMVR), frame rate up-conversion (FRUC) techniques, or template matching techniques. In one example, updated first and second reference motion vectors are generated for internal sub-blocks that are not on the boundary of the current block. In another example, updated first and second reference motion vectors are generated for a subset of sub-blocks of the current block.

In some embodiments, the processing is based on at least two techniques, which may include bidirectional optical flow (BIO) refinement, decoder-side motion vector refinement (DMVR), frame rate up-conversion (FRUC) techniques, or template matching techniques. In one example, processing is performed for each of the at least two techniques to generate multiple result sets, which may be averaged or filtered to generate the current block. In another example, processing is performed in a cascaded manner for each of the at least two techniques to generate the current block.

FIG28 shows a flow chart of an example method for video decoding. The method 2800 includes, at step 2810, generating an intermediate prediction for the current block based on first motion information associated with the current block. In some embodiments, generating the intermediate prediction includes a first interpolation filtering process. In some embodiments, generating the intermediate prediction is also based on signaling in a sequence parameter set (SPS), a picture parameter set (PPS), a codec tree unit (CTU), a slice header, a codec unit (CU), or a CTU group.

Method 2800 includes, at step 2820, updating the first motion information to the second motion information. In some embodiments, updating the first motion information includes using bidirectional optical flow (BIO) refinement.

Method 2800 includes, at step 2830, generating a final prediction for the current block based on the intermediate prediction or the second motion information. In some embodiments, generating the final prediction includes a second interpolation filtering process.

In some embodiments, the first interpolation filtering process uses a first filter set that is different from a second filter set used by the second interpolation filtering process. In some embodiments, at least one filter tap of the first or second interpolation filtering process is based on the dimension, prediction direction, or prediction type of the current block.

FIG29 shows a flow chart of another example method for video decoding. The example includes some features and/or steps similar to those shown in FIG28 and described above. At least some of these features and/or components may not be described separately in this section.

Method 2900 includes, at step 2910, receiving a bitstream representation of a current block of video data. In some embodiments, step 2910 includes receiving the bitstream representation from a memory location or buffer in a video encoder or decoder. In other embodiments, step 2910 includes receiving the bitstream representation at the video decoder via a wireless or wired channel. In other embodiments, step 2910 includes receiving the bitstream representation from a different module, unit, or processor, which may implement one or more methods as described in the embodiments in this document, but is not limited thereto.

Method 2900 includes, at step 2920, generating intermediate motion information based on motion information associated with the current block.

Method 2900 includes, at step 2930, generating updated first and second reference motion vectors based on the first and second reference motion vectors, respectively. In some embodiments, the current block is associated with the first and second reference blocks. In some embodiments, the first and second reference motion vectors are associated with the first and second reference blocks, respectively.

Method 2900 includes, at step 2940, processing the bitstream representation based on the intermediate motion information or the updated first and second reference motion vectors to generate a current block.

In some embodiments of method 2900, generating updated first and second reference motion vectors is based on the weighted sum of the first scaled motion vector and the first and second scaled reference motion vectors, respectively. In some embodiments, the first motion vector is derived based on the first reference motion vector and the second reference motion vector, the first scaled motion vector is generated by scaling the first motion vector to a target precision, and the first and second scaled reference motion vectors are generated by scaling the first and second reference motion vectors to the target precision, respectively.

In some embodiments, an indication of the target accuracy is signaled in a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, a codec tree unit (CTU), or a codec unit (CU).

In some embodiments, the first motion vector has a first precision, and the first and second reference motion vectors have a reference precision. In other embodiments, the first precision can be higher or lower than the reference precision. In other embodiments, the target precision can be set to the first precision, the reference precision, or a fixed (or predetermined) precision that is independent of the first precision and the reference precision.

In some embodiments, the first motion vector is derived based on bidirectional optical flow (BIO) refinement using the first and second reference motion vectors.

In some embodiments, processing is based on bidirectional optical flow (BIO) refinement, and the updated first and second reference motion vectors are constrained to a predetermined value range prior to processing.

In some embodiments, the processing is based on bidirectional optical flow (BIO) refinement or decoder-side motion vector refinement (DMVR), and wherein the updated first and second reference motion vectors are clipped prior to processing.

In some embodiments, the processing is based on bidirectional optical flow (BIO) refinement, decoder-side motion vector refinement (DMVR), frame rate up-conversion (FRUC) techniques, or template matching techniques. In one example, updated first and second reference motion vectors are generated for internal sub-blocks that are not on the boundary of the current block. In another example, updated first and second reference motion vectors are generated for a subset of sub-blocks of the current block.

In some embodiments, the processing is based on at least two techniques, which may include bidirectional optical flow (BIO) refinement, decoder-side motion vector refinement (DMVR), frame rate up-conversion (FRUC) techniques, or template matching techniques. In one example, processing is performed for each of the at least two techniques to generate multiple result sets, which may be averaged or filtered to generate the current block. In another example, processing is performed in a cascaded manner for each of the at least two techniques to generate the current block.

FIG30 shows a flow chart of an example method for video decoding. The method 3000 includes, at step 3010, based on first motion information associated with the current block, generating an intermediate prediction for the current block. In some embodiments, generating the intermediate prediction includes a first interpolation filtering process. In some embodiments, generating the intermediate prediction is also based on signaling in a sequence parameter set (SPS), a picture parameter set (PPS), a codec tree unit (CTU), a slice header, a codec unit (CU), or a CTU group.

Method 3000 includes, at step 3020, updating the first motion information to the second motion information. In some embodiments, updating the first motion information includes using bidirectional optical flow (BIO) refinement.

Method 3000 includes, at step 3030, generating a final prediction of the current block based on the intermediate prediction or the second motion information. In some embodiments, generating the final prediction includes a second interpolation filtering process.

In some embodiments, the first interpolation filtering process uses a first filter set that is different from a second filter set used by the second interpolation filtering process. In some embodiments, at least one filter tap of the first or second interpolation filtering process is based on the dimension, prediction direction, or prediction type of the current block.

FIG31 shows a flow chart of another example method for video decoding. The example includes some features and/or steps similar to those shown in FIG30 above. At least some of these features and/or components may not be described separately in this section.

Method 3100 includes, at step 3110, receiving a bitstream representation of a current block of video data. In some embodiments, step 3110 includes receiving the bitstream representation from a memory location or buffer in a video encoder or decoder. In other embodiments, step 3110 includes receiving the bitstream representation at the video decoder via a wireless or wired channel. In other embodiments, step 3110 includes receiving the bitstream representation from a different module, unit, or processor, which module, unit, or processor may implement one or more methods as described in the embodiments herein, but is not limited thereto.

Method 3100 includes, at step 3120, generating intermediate motion information based on motion information associated with the current block.

The method 3100 includes, at step 3130, generating updated first and second reference motion vectors based on the first and second reference motion vectors, respectively. In some embodiments, the current block is associated with the first and second reference blocks. In some embodiments, the first and second reference motion vectors are associated with the first and second reference blocks, respectively.

The method 3100 includes, at step 3140, processing the bitstream representation based on the intermediate motion information or the updated first and second reference motion vectors to generate a current block.

In some embodiments of method 3100, generating updated first and second reference motion vectors is based on a weighted sum of the first scaled motion vector and the first and second scaled reference motion vectors, respectively. In some embodiments, the first motion vector is derived based on the first reference motion vector and the second reference motion vector, the first scaled motion vector is generated by scaling the first motion vector to a target precision, and the first and second scaled reference motion vectors are generated by scaling the first and second reference motion vectors to the target precision, respectively.

In some embodiments, an indication of the target accuracy is signaled in a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, a codec tree unit (CTU), or a codec unit (CU).

In some embodiments, the first motion vector has a first precision, and the first and second reference motion vectors have a reference precision. In other embodiments, the first precision may be higher or lower than the reference precision. In other embodiments, the target precision may be set to the first precision, the reference precision, or a fixed (or predetermined) precision that is independent of the first precision and the reference precision.

In some embodiments, the first motion vector is derived based on bidirectional optical flow (BIO) refinement using the first and second reference motion vectors.

In some embodiments, processing is based on bidirectional optical flow (BIO) refinement, and the updated first and second reference motion vectors are constrained to a predetermined value range prior to processing.

In some embodiments, the processing is based on bidirectional optical flow (BIO) refinement or decoder-side motion vector refinement (DMVR), and wherein the updated first and second reference motion vectors are clipped prior to processing.

In some embodiments, the processing is based on bidirectional optical flow (BIO) refinement, decoder-side motion vector refinement (DMVR), frame rate up-conversion (FRUC) techniques, or template matching techniques. In one example, updated first and second reference motion vectors are generated for internal sub-blocks that are not on the boundary of the current block. In another example, updated first and second reference motion vectors are generated for a subset of sub-blocks of the current block.

In some embodiments, the processing is based on at least two techniques, which may include bidirectional optical flow (BIO) refinement, decoder-side motion vector refinement (DMVR), frame rate up-conversion (FRUC) techniques, or template matching techniques. In one example, processing is performed for each of the at least two techniques to generate multiple result sets, which may be averaged or filtered to generate the current block. In another example, processing is performed in a cascaded manner for each of the at least two techniques to generate the current block.

FIG33 shows a flow chart of an example method for video decoding. The method 3300 includes, at step 3310, generating an intermediate prediction for the current block based on first motion information associated with the current block. In some embodiments, generating the intermediate prediction includes a first interpolation filtering process. In some embodiments, generating the intermediate prediction is also based on signaling in a sequence parameter set (SPS), a picture parameter set (PPS), a codec tree unit (CTU), a slice header, a codec unit (CU), or a CTU group.

Method 3300 includes, at step 3320, updating the first motion information to the second motion information. In some embodiments, updating the first motion information includes using bidirectional optical flow (BIO) refinement.

Method 3300 includes, at step 3330, generating a final prediction of the current block based on the intermediate prediction or the second motion information. In some embodiments, generating the final prediction includes a second interpolation filtering process.

In some embodiments, the first interpolation filtering process uses a first filter set that is different from a second filter set used by the second interpolation filtering process. In some embodiments, at least one filter tap of the first or second interpolation filtering process is based on the dimension, prediction direction, or prediction type of the current block.

FIG34 shows a flow chart of another example method for video decoding. The example includes some features and/or steps similar to those shown in FIG33 as described above. At least some of these features and/or components may not be described separately in this section.

Method 3400 includes, at step 3410, receiving a bitstream representation of a current block of video data. In some embodiments, step 3410 includes receiving the bitstream representation from a memory location or buffer in a video encoder or decoder. In other embodiments, step 3410 includes receiving the bitstream representation at the video decoder via a wireless or wired channel. In other embodiments, step 3410 includes receiving the bitstream representation from a different module, unit, or processor, which may implement one or more methods as described in the embodiments in this document, but is not limited thereto.

Method 3400 includes, at step 3420, generating intermediate motion information based on motion information associated with the current block.

Method 3400 includes, at step 3430, generating updated first and second reference motion vectors based on the first and second reference motion vectors, respectively. In some embodiments, the current block is associated with the first and second reference blocks. In some embodiments, the first and second reference motion vectors are associated with the first and second reference blocks, respectively.

Method 3400 includes, at step 3440, processing the bitstream representation based on the intermediate motion information or the updated first and second reference motion vectors to generate a current block.

In some embodiments of method 3400, generating updated first and second reference motion vectors is based on the weighted sum of the first scaled motion vector and the first and second scaled reference motion vectors, respectively. In some embodiments, the first motion vector is derived based on the first reference motion vector and the second reference motion vector, the first scaled motion vector is generated by scaling the first motion vector to a target precision, and the first and second scaled reference motion vectors are generated by scaling the first and second reference motion vectors to the target precision, respectively.

In some embodiments, an indication of the target accuracy is signaled in a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, a codec tree unit (CTU), or a codec unit (CU).

In some embodiments, the first motion vector has a first precision, and the first and second reference motion vectors have a reference precision. In other embodiments, the first precision may be higher or lower than the reference precision. In other embodiments, the target precision may be set to the first precision, the reference precision, or a fixed (or predetermined) precision that is independent of the first precision and the reference precision.

In some embodiments, the first motion vector is derived based on bidirectional optical flow (BIO) refinement using the first and second reference motion vectors.

In some embodiments, processing is based on bidirectional optical flow (BIO) refinement, and the updated first and second reference motion vectors are constrained to a predetermined value range prior to processing.

In some embodiments, the processing is based on bidirectional optical flow (BIO) refinement or decoder-side motion vector refinement (DMVR), and wherein the updated first and second reference motion vectors are clipped prior to processing.

In some embodiments, the processing is based on bidirectional optical flow (BIO) refinement, decoder-side motion vector refinement (DMVR), frame rate up-conversion (FRUC) techniques, or template matching techniques. In one example, updated first and second reference motion vectors are generated for internal sub-blocks that are not on the boundary of the current block. In another example, updated first and second reference motion vectors are generated for a subset of sub-blocks of the current block.

In some embodiments, the processing is based on at least two techniques, which may include bidirectional optical flow (BIO) refinement, decoder-side motion vector refinement (DMVR), frame rate up-conversion (FRUC) techniques, or template matching techniques. In one example, processing is performed for each of the at least two techniques to generate multiple result sets, which may be averaged or filtered to generate the current block. In another example, processing is performed in a cascaded manner for each of the at least two techniques to generate the current block.

FIG35 shows a flow chart of an example method for video decoding. Method 3500 includes, at step 3510, generating an updated reference block for a bitstream representation of a current block by modifying a reference block associated with the current block.

In some embodiments, method 3500 further includes the step of filtering the reference block using a smoothing filter.

In some embodiments, method 3500 further includes the step of filtering pixels at block boundaries of the reference block.

In some embodiments, method 3500 further includes the step of applying overlapping block motion compensation (OBMC) to the reference block.

In some embodiments, method 3500 further includes the step of applying illumination compensation (IC) to the reference block.

In some embodiments, method 3500 further includes the step of applying a weighted prediction to the reference block.

Method 3500 includes, at step 3520, computing temporal gradients for bidirectional optical flow (BIO) motion refinement based on the updated reference block.

Method 3500 includes, at step 3530, performing a transformation including BIO motion refinement between a bitstream representation and a current block based on a temporal gradient. In some embodiments, the transformation generates the current block from the bitstream representation (e.g., as may be implemented in a video decoder). In other embodiments, the transformation generates the bitstream representation from the current block (e.g., as may be implemented in a video encoder).

In some embodiments, method 3500 further includes calculating a mean of the reference block; and subtracting the mean from each pixel of the reference block. In one example, calculating the mean is based on all pixels of the reference block. In another example, calculating the mean is based on all pixels in a sub-block of the reference block.

In some embodiments, the mean is calculated based on a subset of pixels (in other words, not all pixels) of the reference block. In one example, the subset of pixels includes pixels in every fourth row or column of the reference block. In another example, the subset of pixels includes four corner pixels. In yet another example, the subset of pixels includes four corner pixels and a center pixel.

FIG36 shows a flow chart of another example method for video decoding. The example includes some features and/or steps similar to those shown in FIG35 as described above. At least some of these features and/or components may not be described separately in this section.

Method 3600 includes, at step 3610, generating a temporal gradient for bidirectional optical flow (BIO) motion refinement for a bitstream representation of the current block.

Method 3600 includes, at step 3620, generating an updated temporal gradient by subtracting a difference between a first mean and a second mean from the temporal gradient, wherein the first mean is a mean of a first reference block, the second mean is a mean of a second reference block, and the first and second reference blocks are associated with the current block.

In some embodiments, the mean is based on all pixels of the corresponding reference block (e.g., the first mean is calculated as the mean of all pixels of the first reference block). In another example, the mean is calculated based on all pixels in a sub-block of the corresponding reference block.

In some embodiments, the mean is based on a subset of pixels (in other words, not all pixels) of the corresponding reference block. In one example, the subset of pixels includes pixels in every fourth row or column of the corresponding reference block. In another example, the subset of pixels includes four corner pixels. In yet another example, the subset of pixels includes four corner pixels and a center pixel.

Method 3600 includes, at step 3630, performing a transformation including BIO motion refinement between the bitstream representation and the current block based on the updated temporal gradient. In some embodiments, the transformation generates the current block from the bitstream representation (e.g., as may be implemented in a video decoder). In other embodiments, the transformation generates the bitstream representation from the current block (e.g., as may be implemented in a video encoder).

FIG38 shows a flow chart of an example method for video processing. The method 3800 includes: determining raw motion information of a current block at step 3810; scaling raw motion vectors of the raw motion information and derived motion vectors derived based on the raw motion vectors to the same target accuracy at step 3820; generating updated motion vectors from the scaled raw and derived motion vectors at step 3830; and performing conversion between the current block and a bitstream representation of a video including the current block based on the updated motion vectors at step 3840.

FIG39 shows a flow chart of an example method for video processing. The method 3900 includes: at step 3910, determining raw motion information of a current block; at step 3920, updating a raw motion vector of the raw motion information of the current block based on a refinement method; at step 3930, clipping the updated motion vector to a range; and at step 3940, performing a conversion between the current block and a bitstream representation of a video including the current block based on the clipped updated motion vector.

FIG40 shows a flow chart of an example method for video processing. The method 4000 includes: at step 4010, determining original motion information associated with a current block; at step 4020, generating updated motion information based on a specific prediction mode; and at step 4030, performing a conversion between the current block and a bitstream representation of video data including the current block based on the updated motion information, wherein the specific prediction mode includes one or more of bidirectional optical flow (BIO) refinement, decoder-side motion vector refinement (DMVR), frame rate up-conversion (FRUC) technology, or template matching technology.

FIG41 shows a flow chart of an example method for video processing. The method 4100 includes: at step 4110, determining a motion vector difference (MVD) precision of a current block processed using an affine mode from a set of MVD precisions; at step 4120, performing a conversion between the current block and a bitstream representation of a video including the current block based on the determined MVD precision.

FIG42 shows a flow chart of an example method for video processing. The method 4200 includes: at step 4210, determining unupdated motion information associated with a current block; at step 4220, updating the unupdated motion information based on multiple decoder-side motion vector derivation (DMVD) methods to generate updated motion information for the current block; and at step 4230, performing a conversion between the current block and a bitstream representation of a video including the current block based on the updated motion information.

FIG43 shows a flow chart of an example method for video processing. The method 4300 includes: at step 4310, based on first motion information associated with the current block, generating an intermediate prediction for the current block; at step 4320, updating the first motion information to second motion information based on the intermediate prediction; and at step 4330, generating a final prediction for the current block based on the second motion information.

7. Example implementation of the disclosed technology

FIG. 37 is a block diagram of a video processing device 3700. Device 3700 may be used to implement one or more methods described herein. Device 3700 may be embodied in a smartphone, a tablet, a computer, an Internet of Things (IoT) receiver, etc. Device 3700 may include one or more processors 3702, one or more memories 3704, and video processing hardware 3706. Processor 3702 may be configured to implement one or more methods described herein (including but not limited to methods 27003100, 3300-3600, and 3800-4200). (One or more) memories 3704 may be used to store data and code used to implement the methods and techniques described herein. Video Processing Hardware 3706 can be used to implement some of the techniques described in this document in hardware circuits.

In some embodiments, the video encoding and decoding method can be implemented using a device implemented on a hardware platform as described with respect to FIG. 37 .

The following clause-based format may be used to describe various embodiments and techniques described throughout this document.

1.1 A video processing method, comprising: determining raw motion information of a current block; scaling raw motion vectors of the raw motion information and derived motion vectors derived based on the raw motion vectors to the same target accuracy; generating updated motion vectors from the scaled raw and derived motion vectors; and performing a conversion between a bitstream representation of the current block and a video including the current block based on the updated motion vectors.

1.2. A method as in Example 1.1, wherein the original motion vector has a first precision, the derived motion vector has a second precision different from the first precision, and the target precision is set to a higher precision or a lower precision between the first precision and the second precision.

1.3. The method of Example 1.1, wherein the target accuracy is set to a fixed accuracy.

1.4. The method of Example 1.1, wherein the target accuracy is higher than the accuracy of the original motion vector.

1.5. The method of Example 1.4, wherein the original motion vector is scaled to: mvLX' _x =sign(mvLX _x ) * (abs(mvLX _x )<<N), mvLX' _y =sign(mvLX _y ) * (abs(mvLX _y )<<N), where (mvLX _x ,mvLX _y ) is the original motion vector, (mvLX' _x ,mvLX' _y ) is the scaled original motion vector, the function sign(.) returns the sign of the input parameter, the function abs(.) returns the absolute value of the input parameter, N=log2(curr_mv_precision/targ_mv_precision), and where curr_mv_precision is the precision of the original motion vector, and targ_mv_precision is the precision of the derived motion vector, which is taken as the target precision.

1.6. The method of Example 1.1, wherein the target precision is the same as the precision of the original motion vector.

1.7. A method as in Example 1.1, wherein the original motion vector has a first precision, the derived motion vector has a second precision different from the first precision, and the target precision is set to the first precision.

1.8. The method of Example 1.7, wherein when the derived motion vector is right-shifted by N to achieve the target accuracy, the derived motion vector is scaled as: v' _x =(v _x +offset)>>N,v' _y =(v _y +offset)>>N; or v' _x =sign(v _x ) * ((abs(v _x )+offset)>>N),v' _y =sign(v _y ) * ((abs(v _y )+offset)>>N)

Where (v _x ,v _y ) is the derived motion vector, (v' _x ,v' _y ) is the scaled derived motion vector, offset is the offset applied to the derived motion vector to achieve the target precision, the function sign(.) returns the sign of the input parameter, the function abs(.) returns the absolute value of the input parameter, N=log2(curr_mv_precision/targ_mv_precision), where curr_mv_precision is the first precision and targ_mv_precision is the second precision.

1.9. The method of Example 1.1, wherein the generation of the scaled and updated motion vectors is performed as follows: mvL0' _x = -v _x /S ₀ +mvL0 _x , mvL0' _y = -v _y /S ₀ +mvL0 _y ; and/or mvL1' _x = v _x /S ₁ +mvL1 _x , mvL1' _y = v _y /S ₁ +mvL1 _y

Where (mvL0 _x ,mvL0 _y ) and (mvL1 _x ,mvL1 _y ) are the original motion vectors, (mvL0' _x ,mvL0' _y ) and (mvL1' _x ,mvL1' _y ) are the updated motion vectors, (v _x ,v _y ) are the derived motion vectors, and S ₀ and S ₁ are the scaling factors.

1.10. The method of Example 1.1, wherein the generation of the scaled and updated motion vectors is performed as follows: mvL0' _x = (-v _x + offset0) / S ₀ + mvL0 _x , mvL0' _y = - (v _y + offset0) / S ₀ + mvL0 _y , and/or mvL1'x = (vx + offset1) / S1 + mvL1x , mvL1'y = (vy + offset1) / S1 + mvL1y

Where (mvL0 _x ,mvL0 _y ) and (mvL1 _x ,mvL1 _y ) are the original motion vectors, (mvL0' _x ,mvL0' _y ) and (mvL1' _x ,mvL1' _y ) are the updated motion vectors, (v _x ,v _y ) are the derived motion vectors, offset0 and offset1 are the offsets, and S ₀ and S ₁ are the scaling factors.

1.11. The method of Example 1.1, wherein the generation of the scaled and updated motion vectors is performed as follows: mvL0' _x =((-v _x +1)>>1)+mvL0 _x , mvL0' _y =(-(v _y +1)>>1)+mvL0 _y ; and/or mvL1' _x =((v _x +1)>>1)+mvL1 _x , mvL1' _y =((v _y +1)>>1)+mvL1 _y

Among them, (mvL0 _x ,mvL0 _y ) and (mvL1 _x ,mvL1 _y ) are the original motion vectors, (mvL0' _x ,mvL0' _y ) and (mvL1' _x ,mvL1' _y ) are the updated motion vectors, and (v _x ,v _y ) is the derived motion vector.

1.12. A method as in any of Examples 1.9-1.11, wherein the generation of the scaled and updated motion vectors is performed when τ ₀ >0 and τ ₁ >0, wherein τ ₀ =POC(current)-POC(Ref ₀ ), τ ₁ =POC(Ref ₁ )-POC(current), and wherein POC(current), POC(Ref ₀ ) and POC(Ref ₁ ) are the picture order counts of the current block, the first reference block and the second reference block, respectively.

1.13. The method of Example 1.1, wherein the generation of the scaled and updated motion vectors is performed as follows: mvL0' _x = _-SF0 * _vx + _mvL0x , mvL0' _y = _-vy * _SF0 + _mvL0y ; and/or mvL1'x = -SF1*vx + mvL1x, mvL1'y = -SF1*vy + mvL1y

Wherein, (mvL0 _x ,mvL0 _y ) and (mvL1 _x ,mvL1 _y ) are the original motion vectors, (mvL0' _x ,mvL0' _y ) and (mvL1' _x ,mvL1' _y ) are the updated motion vectors, (v _x ,v _y ) are the derived motion vectors, and SF ₀ and SF ₁ are the scaling factors.

1.14. A method as in Example 1.13, wherein the generation of the scaled and updated motion vectors is performed when τ ₀ >0, τ ₁ <0 and τ ₀ > |τ ₁ |, wherein τ ₀ =POC(current)-POC(Ref ₀ ), τ ₁ =POC(Ref ₁ )-POC(current), and wherein POC(current), POC(Ref ₀ ) and POC(Ref ₁ ) are the picture order counts of the current block, the first reference block and the second reference block, respectively.

1.15. The method of Example 1.1, wherein the generation of the scaled and updated motion vectors is performed as follows: mvL0' _x = SFACT ₀ * v _x + mvL0 _x , mvL0' _y = SFACT ₀ * v _y + mvL0 _y , and/or mvL1' _x = SFACT ₁ * v _x + mvL1 _x , mvL1' _y = SFACT ₁ * v _y + mvL1 _y

Where (mvL0 _x ,mvL0 _y ) and (mvL1 _x ,mvL1 _y ) are the original motion vectors, (mvL0' _x ,mvL0' _y ) and (mvL1' _x ,mvL1' _y ) are the updated motion vectors, (v _x ,v _y ) are the derived motion vectors, and SFACT ₀ and SFACT ₁ are the scaling factors.

1.16. A method as in Example 1.15, wherein the scaling and updating of the motion vector generation is performed when τ ₀ >0, τ ₁ <0 and τ ₀ <|τ ₁ |, wherein τ ₀ =POC(current)-POC(Ref ₀ ), τ ₁ =POC(Ref ₁ )-POC(current), and wherein POC(current), POC(Ref ₀ ) and POC(Ref ₁ ) are the picture order counts of the current block, the first reference block and the second reference block, respectively.

1.17. A method as in Example 1.1, wherein, when τ ₀ >0, τ ₁ >0 and τ ₀ > |τ ₁ |, the derivation of the derived motion vector and the generation of the updated motion vector are performed together, wherein τ ₀ =POC(current)-POC(Ref ₀ ), τ ₁ =POC(Ref ₁ )-POC(current), and wherein POC(current), POC(Ref ₀ ) and POC(Ref ₁ ) are the picture order counts of the current block, the first reference block and the second reference block, respectively.

1.18. The method of Example 1.17, wherein when the derived motion vector is right-shifted by N to achieve a target accuracy, the generation of the scaled and updated motion vector is performed as follows: mvL0' _x =((-v _x +offset)>>(N+1))+mvL0 _x ,mvL0' _y =((-v _y +offset1)>>(N+1))+mvL0 _y ,mvL1' _x =((v _x +offset)>>(N+1))+mvL1 _x ,mvL1' _y =((v _y +offset2)>>(N+1))+mvL1 _y , wherein (mvL0 _x ,mvL0 _y ) and (mvL1 _x ,mvL1 _y ) are the original motion vectors, and (mvL0' _x ,mvL0' _y ) and (mvL1' _x ,mvL1' _y ) are the updated motion vectors, (v _x ,v _y ) are the derived motion vectors, offset1 and offset2 are offsets, N=log2(curr_mv_precision/targ_mv_precision), and where curr_mv_precision is the precision of the original motion vector and targ_mv_precision is the precision of the derived motion vector.

1.19. A method as in Example 1.17, wherein the original motion vector has a first precision, the derived motion vector has a second precision different from the first precision, and the original motion vector is left-shifted by N to achieve a target precision as the second precision.

1.20. A method as in Example 1.17, wherein the original motion vector is shifted left by K and the derived motion vector is shifted right by N-K to achieve the target accuracy.

1.21. The method of Example 1.17, wherein the generation of the scaled and updated motion vectors is performed as follows: mvL0' _x = -sign(v _x ) * ((abs(v _x )+offset0)>>(N+1))+mvL0 _x , mvL0' _y = -sign(v _y ) * ((abs(v _y )+offset0)>>(N+1))+mvL0 _y , mvL1' _x = sign(v _x ) * ((abs(v _x )+offset1)>>(N+1))+mvL1 _x , mvL1' _y = sign(v _y ) * ((abs(v _y )+offset1)>>(N+1))+mvL1 _y

Where, (mvL0 _x ,mvL0 _y ) and (mvL1 _x ,mvL1 _y ) are the original motion vectors, (mvL0' _x ,mvL0' _y ) and (mvL1' _x ,mvL1' _y ) are the updated motion vectors, (v _x ,v _y ) are the derived motion vectors, offset0 and offset1 are offsets, the function sign(.) returns the sign of the input parameter, the function abs(.) returns the absolute value of the input parameter, N=log2(curr_mv_precision/targ_mv_precision), curr_mv_precision is the precision of the original motion vector, and targ_mv_precision is the precision of the derived motion vector.

1.22. A method as in Example 1.1, wherein updating the first and second reference motion vectors includes using bidirectional optical flow (BIO) refinement.

1.23. A method as in any of Examples 1.1-1.22, wherein the method is not applied if the current block satisfies a certain condition.

1.24. The method of Example 1.23, wherein the specific condition specifies at least one of the following: the size of the current block, the slice type of the current block, the picture type of the current block, and the slice type of the current block.

1.25. A method as in Example 1.23, wherein the specific condition stipulates that the number of samples contained in the current block is less than a first threshold.

1.26. The method of Example 1.23, wherein the specific condition stipulates that the minimum dimension of the width and height of the current block is less than or not greater than the second threshold.

1.27. A method as in Example 1.23, wherein the specific condition stipulates that the width of the current block is less than or not greater than a third threshold, and/or the height of the current block is less than or not greater than a fourth threshold.

1.28. A method as in Example 1.23, wherein the specific condition stipulates that the width of the current block is greater than or not less than a third threshold, and/or the height of the current block is greater than or not less than a fourth threshold.

1.29. A method as in Example 1.23, wherein the method is applied at the sub-block level when the width and/or height of the block to which the sub-block belongs is equal to or greater than the fifth threshold.

1.30. A method as in Example 1.29, wherein the current block is split into a plurality of sub-blocks, and each of the plurality of sub-blocks is further subjected to a bidirectional optical flow (BIO) process in the same manner as a normal encoding/decoding block having a size equal to the sub-block size.

1.31. A method as in any of Examples 1.25-1.29, wherein each of the first to fifth thresholds is predefined or signaled at a sequence parameter set (SPS) level, or a picture parameter set (PPS) level, or a picture level, or a slice level, or a slice level.

1.32. A method as in Example 1.31, wherein each of the first to fifth thresholds is defined based on codec information including at least one of a block size, a picture type, and a temporal layer index.

1.33. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement the method of any one of Examples 1.1 to 1.32.

1.34. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for performing a method as described in any one of Examples 1.1 to 1.32.

2.1. A video processing method, comprising: determining original motion information of a current block; updating an original motion vector of the original motion information of the current block based on a refinement method; clipping the updated motion vector to a range; and performing a conversion between the current block and the bitstream representation of a video including the current block based on the clipped updated motion vector.

2.2. The method as described in Example 2.1, wherein the refinement method includes bidirectional optical flow BIO refinement, decoder-side motion vector refinement DMVR, frame rate up-conversion FRUC or template matching.

2.3. The method of Example 2.1, wherein the updated motion vector is clipped to the same range as the range allowed by the original motion vector.

2.4. The method as described in Example 2.1, wherein the difference between the updated motion vector and the original motion vector is clipped to the same range or to different ranges for different sub-blocks.

2.5. A method as described in Example 2.1, wherein the refinement method includes bidirectional optical flow BIO refinement, and the motion vector derived from the original motion vector in the BIO refinement is constrained to a first range as follows: -M _x <v _x <N _x , and/or -M _y < _vy <N _y , where (v _x , _vy ) is the derived motion vector, and M _x , N _x , _My , N _y are non-negative integers.

2.6. The method of Example 2.1, wherein the refinement method comprises bidirectional optical flow (BIO) refinement, and the updated motion vector is constrained to a second range as follows: -M _L0x <mvL0' _x <N _L0x , and/or -M _L2.1x <mvL2.1' _x <N _L2.1x , and/or -M _L0x <mvL0' _x <N _L0x , and/or -M _L2.1y <mvL2.1' _y <N _L2.1y

Wherein (mvL0' _x ,mvL0' _y ) and (mvL2.1' _x ,mvL2.1' _y ) are updated motion vectors of different reference lists, and M _L0x , N _L0x , M _L2.1x , N _L2.1x , M _L0y , N _L0y , M _L2.1y , N _L2.1y are non-negative integers.

2.7. A method as described in any of Examples 2.1-2.6, wherein the method is not applied if the current block satisfies a specific condition.

2.8. The method of Example 2.7, wherein the specific condition specifies at least one of the following: the size of the current block, the slice type of the current block, the picture type of the current block, and the slice type of the current block.

2.9. The method as described in Example 2.7, wherein the specific condition stipulates that the number of samples contained in the current block is less than a first threshold.

2.10. The method as described in Example 2.7, wherein the specific condition stipulates that the minimum size of the width and height of the current block is less than or not greater than the second threshold.

2.11. The method as described in Example 2.7, wherein the specific condition stipulates that the width of the current block is less than or not greater than a third threshold, and/or the height of the current block is less than or not greater than a fourth threshold.

2.12. The method as described in Example 2.7, wherein the specific condition specifies that the width of the current block is greater than or not less than a third threshold, and/or the height of the current block is greater than or not less than a fourth threshold.

2.13. The method as described in Example 2.7, wherein the method is applied at the sub-block level when the width and/or height of the block to which the sub-block belongs is equal to or greater than the fifth threshold.

2.14. The method of Example 2.13, wherein the current block is divided into a plurality of sub-blocks, and each of the plurality of sub-blocks is further subjected to a bidirectional optical flow BIO process in the same manner as a normal codec block having a size equal to the size of the sub-block.

2.15. A method as described in any of Examples 2.9-2.13, wherein each of the first to fifth thresholds is predefined or signaled at a sequence parameter set (SPS) level, or a picture parameter set (PPS) level, or a picture level, or a slice level, or a slice level.

2.16. The method of Example 2.15, wherein each of the first to fifth thresholds is defined based on codec information including at least one of a block size, a picture type, and a temporal layer index.

2.17. A device in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement the method described in any one of Examples 2.1 to 2.16.

2.18. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for performing a method as described in any one of Examples 2.1 to 2.16.

3.1. A video processing method, comprising: determining original motion information associated with a current block; generating updated motion information based on a specific prediction mode; and performing conversion between the current block and a bit stream representation of video data including the current block based on the updated motion information, wherein the specific prediction mode comprises one or more of bidirectional optical flow (BIO) refinement, decoder-side motion vector refinement (DMVR), frame rate up-conversion (FRUC) technology, or template matching technology.

3.2. The method of Example 3.1, wherein the updated motion information includes an updated motion vector.

3.3. The method of Example 3.1, wherein the updated motion vector is used for motion prediction for encoding and decoding subsequent video blocks; or the updated motion vector is used for filtering or overlapping block motion compensation (OBMC).

3.4. The method of Example 3.3, wherein the updated motion vector is used for motion prediction in Advanced Motion Vector Prediction (AMVP) mode, Merge mode and/or Affine mode.

3.5. The method of Example 3.3, wherein the filtering includes deblocking filtering.

3.6. A method as described in any one of Examples 3.1-3.5, wherein the updated motion information is used for the first module and the original motion information is used for the second module.

3.7. The method of Example 3.6, wherein the first module is a motion prediction module and the second module is a deblocking module.

3.8. A method as described in any of Examples 3.2-3.7, wherein the motion prediction is used to process blocks following the current block in the current picture or strip.

3.9. A method as described in any of Examples 3.2-3.7, wherein the motion prediction is used to process a picture or strip to be processed after the current picture or strip including the current block.

3.10. A method as described in any one of Examples 3.1-3.9, wherein the updated motion vector is used only for motion information prediction of a coding unit (CU) or prediction unit (PU) that does not immediately follow the current block in processing order.

3.11. A method as described in any one of Examples 3.1 to 3.10, wherein the updated motion vector is not used for motion prediction of a CU/PU that follows the current block in processing order.

3.12. A method as described in any one of Examples 3.1-3.11, wherein the updated motion vector is used only as a predictor for processing subsequent pictures/slices/strips.

3.13. The method of Example 3.12, wherein the updated motion vector is used as a temporal motion vector prediction (TMVP) in an advanced motion vector prediction (AMVP) mode, a merge mode, or an affine mode.

3.14. A method as described in Example 3.12, wherein the updated motion vector is used only for predictors that process subsequent pictures/slices/strips in an optional temporal motion vector prediction (ATMVP) mode and/or a spatio-temporal motion vector prediction (STMVP) mode.

3.15. A method as described in any one of Examples 3.1-3.14, wherein the encoder notifies the decoder with a signal including information including whether to update the MV used for the BIO codec block and/or whether to use the updated MV for motion prediction and/or how to use the updated MV for motion prediction.

3.16. The method as described in Example 3.15 further includes: signaling the information in a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, a coding tree unit (CTU) or a CU.

3.17. The method as described in Example 3.1 further includes: updating motion information, which includes updating the motion vector and reference picture of each predicted direction at the block level.

3.18. The method as described in Example 3.1 or 3.17, wherein, within a block, for some sub-blocks, the updated motion information is stored, and for the remaining sub-blocks, the unupdated motion information is stored.

3.19. As described in Example 3.1 or 3.17, the updated motion vector is stored only for internal sub-blocks that are not at the PU/CU/CTU boundary

3.20. The method as described in Example 3.19 further includes: using the updated motion vector of the internal sub-block for motion prediction, deblocking or OBMC.

3.21. As described in Example 3.1 or 3.17, the updated motion vector is stored only for the boundary sub-block at the boundary of PU/CU/CTU.

3.22. A method as described in Example 3.1 or 3.17, wherein if the neighboring block and the current block are not in the same CTU or in the same region with a size of 64×64 or 32×32, the updated motion information from the neighboring block is not used.

3.23. The method of Example 3.22, wherein if the neighboring block and the current block are not in the same CTU or in the same region having a size of 64×64 or 32×32, the neighboring block is marked as unavailable.

3.24. The method of Example 3.22, wherein if the neighboring block and the current block are not in the same CTU or in the same region having a size of 64×64 or 32×32, the current block uses unupdated motion information.

3.25. The method of Example 3.17, wherein if the neighboring block and the current block are not in the same CTU column or in the same column of a region having a size of 64×64 or 32×32, the updated motion vector from the neighboring block is not used.

3.26. The method of Example 3.25, wherein if the neighboring block and the current block are not in the same CTU column or in the same column of a region having a size of 64×64 or 32×32, the neighboring block is marked as unavailable.

3.27. The method of Example 3.25, wherein if the neighboring block and the current block are not in the same CTU column or in the same column of a region having a size of 64×64 or 32×32, the current block uses the unupdated motion information from the neighboring block.

3.28. The method of Example 3.17, wherein if the bottommost column of the block is a CTU or a bottommost column of a region having a size of 64×64 or 32×32, the motion information of the block is not updated.

3.29. The method of Example 3.17, wherein if the rightmost column of the block is a CTU or the rightmost column of a region having a size of 64×64 or 32×32, the motion information of the block is not updated.

3.30. The method as described in Example 3.1 or 3.17 further includes predicting the motion information of the block/CU in the current CTU based on the updated motion information or the unupdated motion information of the adjacent CTU or region.

3.31. A method as described in Example 3.30, wherein the updated motion information from the left CTU or left region is used for the current CTU.

3.32. A method as described in Example 3.30 or 3.31, wherein the updated motion information from the upper left CTU or upper left region is used for the current CTU.

3.33. A method as described in any one of Examples 3.30-3.32, wherein the updated motion information from the upper CTU or upper region is used for the current CTU.

3.34. A method as described in any of Examples 3.30-3.33, wherein the updated motion information from the upper right CTU or upper right region is used for the current CTU.

3.35. A method as described in any of Examples 3.30-3.34, wherein each of the one or more regions has a size of 64×64 or 32×32.

3.36. A method as described in any of Examples 3.1-3.35, wherein the method is not applied if the current block satisfies a specific condition.

3.37. The method of Example 3.36, wherein the specific condition specifies at least one of the following: the size of the current block, the slice type of the current block, the picture type of the current block, and the slice type of the current block.

3.38. The method as described in Example 3.36, wherein the specific condition stipulates that the number of samples contained in the current block is less than a first threshold.

3.39. The method as described in Example 3.36, wherein the specific condition stipulates that the minimum dimension of the width and height of the current block is less than or not greater than the second threshold.

3.40. The method of Example 3.36, wherein the specific condition stipulates that the width of the current block is less than or not greater than a third threshold, and/or the height of the current block is less than or not greater than a fourth threshold.

3.41. The method as described in Example 3.36, wherein the specific condition stipulates that the width of the current block is greater than or not less than a third threshold, and/or the height of the current block is greater than or not less than a fourth threshold.

3.42. The method of Example 3.36, wherein the method is applied to the sub-block level when the width and/or height of the block to which the sub-block belongs is equal to or greater than the fifth threshold.

3.43. The method of Example 3.42, wherein the current block is divided into a plurality of sub-blocks, and each of the plurality of sub-blocks is further subjected to bidirectional optical flow (BIO) processing in the same manner as a normal codec block having a size equal to the size of the sub-block.

3.44. A method as described in any of Examples 3.38-3.42, wherein each of the first to fifth thresholds is predefined or signaled at a sequence parameter set (SPS) level, or a picture parameter set (PPS) level, or a picture level, or a slice level, or a slice level.

3.45. The method of Example 3.44, wherein each of the first to fifth thresholds is defined based on codec information including at least one of a block size, a picture type, and a temporal layer index.

3.46. An apparatus in a video system, comprising a processor and a non-volatile memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement the method described in any one of Examples 3.1 to 3.45.

3.47. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for executing the method described in any one of Examples 3.1 to 3.45.

4.1. A video processing method, comprising: determining a motion vector difference (MVD) precision for a current block processed in an affine mode from a motion vector difference (MVD) precision set; performing a conversion between the current block and a bitstream representation of a video including the current block based on the determined MVD precision.

4.2. A method as described in Example 4.1, wherein the MVD represents the difference between the predicted motion vector and the actual motion vector used during the motion compensation process.

4.3. The method as described in Example 4.2, wherein the MVD accuracy set includes multiple different MVD accuracy constituting a geometric sequence.

4.4. The method of Example 4.3, wherein the MVD precision set includes 1/4, 1, and 4 pixel MVD precisions.

4.5. The method as described in Example 4.3, wherein the MVD precision set includes 1/4, 1/2, 1, 2, and 4 pixel MVD precisions.

4.6. The method as described in Example 4.3, wherein the MVD precision set includes 1/16, 1/8 and 1/4 pixel MVD precisions.

4.7. The method of Example 4.1, wherein the current block is a coding unit or a prediction unit.

4.8. The method as described in any one of Examples 4.1-4.7, wherein determining the MVD precision further comprises: determining the MVD precision of the current block based on a syntax element indicating the MVD precision.

4.9. The method of Example 4.8, wherein the syntax element is present when there is a non-zero MVD component of the current block.

4.10. The method of Example 4.8, wherein the syntax element is absent when there is no non-zero MVD component of the current block.

4.11. A method as described in Example 4.8, wherein the syntax element is present regardless of whether there are any non-zero MVD components of the current block.

4.12. The method of Example 4.1, wherein the current block is processed using an affine inter-frame mode or an affine advanced motion vector prediction (AMVP) mode.

4.13. A method as described in Example 4.12, wherein different MVDs of the current block are associated with different MVD precisions.

4.14. The method of Example 4.13, wherein the affine inter-frame pattern is a 4-parameter affine inter-frame pattern with 2 control points, and one MVD is used for each control point in each prediction direction.

4.15. The method of Example 4.14, wherein the two control points are associated with different MVD accuracies.

4.16. The method of Example 4.13, wherein the affine inter-frame pattern is a 6-parameter affine inter-frame pattern with 3 control points, and one MVD is used for each control point in each prediction direction.

4.17. The method of Example 4.16, wherein the three control points are associated with different MVD accuracies.

4.18. A method as described in Example 4.15, wherein there are two syntax elements to indicate different MVD precisions of the two control points.

4.19. A method as described in Example 4.17, wherein there are three syntax elements to indicate different MVD precisions of the three control points.

4.20. The method of Example 4.1, wherein the MVD precision set is determined based on the encoding and decoding information of the current block.

4.21. The method of Example 4.20, wherein the encoding and decoding information includes the quantization level of the current block.

4.22. The method of Example 4.21, wherein a coarser set of MVD precisions is selected for larger quantization levels.

4.23. The method of Example 4.21, wherein a finer set of MVD precisions is selected for smaller quantization levels.

4.24. A method as described in any of Examples 4.1-4.23, wherein the method is not applied if the current block satisfies a specific condition.

4.25. The method of Example 4.24, wherein the specific condition specifies at least one of: the size of the current block, the slice type of the current block, the picture type of the current block, and the slice type of the current block.

4.26. The method as described in Example 4.24, wherein the specific condition specifies that the number of samples contained in the current block is less than a first threshold.

4.27. The method of Example 4.24, wherein the specific condition specifies that the minimum dimension of the width and height of the current block is less than or not greater than a second threshold.

4.28. The method of Example 4.24, wherein the specific condition specifies that the width of the current block is less than or not greater than a third threshold, and/or the height of the current block is less than or not greater than a fourth threshold.

4.29. The method of Example 4.24, wherein the specific condition specifies that the width of the current block is greater than or not less than a third threshold, and/or the height of the current block is greater than or not less than a fourth threshold.

4.30. A method as described in Example 4.24, wherein the method is applied at the sub-block level when the width and/or height of the block to which the sub-block belongs is equal to or greater than the fifth threshold.

4.31. A method as described in Example 4.30, wherein the current block is divided into a plurality of sub-blocks, and each of the plurality of sub-blocks is further subjected to a bidirectional optical flow (BIO) process in the same manner as a normal codec block having a size equal to the size of the sub-block.

4.32. A method as described in any of Examples 4.26-4.30, wherein each of the first to fifth thresholds is predefined or signaled at a sequence parameter set (SPS) level, or a picture parameter set (PPS) level, or a picture level, or a slice level, or a slice level.

4.33. The method of Example 4.32, wherein each of the first to fifth thresholds is defined based on coding information including at least one of a block size, a picture type, and a temporal layer index.

4.34. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement the method described in any one of Examples 4.1 to 4.33.

4.35. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for performing the method described in any one of Examples 4.1 to 4.33.

5.1. A video processing method, comprising: determining unupdated motion information associated with a current block; updating the unupdated motion information based on multiple decoder-side motion vector derivation (DMVD) methods to generate updated motion information for the current block; and performing a conversion between the current block and a bitstream representation of a video including the current block based on the updated motion information.

5.2. The method of Example 5.1, wherein the plurality of DMVD methods include at least two of the following: bidirectional optical flow (BIO) refinement, decoder-side motion vector refinement (DMVR), frame rate up-conversion (FRUC) technology, and template matching technology.

5.3. The method as described in Example 5.2, wherein the multiple DMVD methods are executed on the unupdated motion information of the current block in a simultaneous manner, and the unupdated motion vector of the unupdated motion information is input as a search starting point for each of the multiple DMVD methods.

5.4. The method as described in Example 5.2, wherein the multiple DMVD methods are executed in a cascade manner on the unupdated motion information of the current block, and the updated motion vector of the updated motion information generated by one DMVD method is input as the search starting point of the next DMVD method.

5.5. The method as described in Example 5.4, wherein the one DMVD method is DMVR, and the next DMVD method is BIO, wherein DMVR is performed on the unupdated motion information of the current block to generate the updated motion information, and the updated motion vector of the updated motion information is input as the search starting point of BIO.

5.6. The method of any one of Examples 5.1 to 5.5, wherein the updating of the non-updated motion information based on multiple decoder-side motion vector derivation (DMVD) methods to generate updated motion information of the current block further comprises: deriving multiple sets of updated motion information by the multiple DMVD methods, generating a final set of updated motion information from the multiple sets of updated motion information.

5.7. The method as described in Example 5.6, wherein generating the final set of updated motion information from the multiple sets of updated motion information further comprises: generating the final set of updated motion information based on the average or weighted average of the multiple sets of updated motion information.

5.8. The method of Example 5.6, wherein generating the final set of updated motion information from the multiple sets of updated motion information further comprises: generating the final set of updated motion information by filtering the multiple sets of updated motion information using a median filter.

5.9. The method as described in Example 5.6, wherein generating the final set of updated motion information from the multiple sets of updated motion information further comprises: assigning different priorities to the multiple DMVD methods, and selecting a set of updated motion information derived by the DMVD method with the highest priority as the final set of updated motion information.

5.10. The method of Example 5.9, wherein the highest priority is assigned to the decoder-side motion vector refinement (DMVR).

5.11. The method as described in any one of Examples 5.1 to 5.5, wherein the conversion between the current block and the bitstream representation of the video including the current block based on the updated motion information further includes: performing motion compensation respectively using multiple sets of updated motion information derived by the multiple DMVD methods to obtain multiple sets of motion compensation results, and generating the current block based on the average or weighted average of the multiple sets of motion compensation results.

5.12. The method as described in any one of Examples 5.1 to 5.5, wherein the conversion between the current block and the bitstream representation of the video including the current block based on the updated motion information further includes: performing motion compensation respectively using multiple sets of updated motion information derived by the multiple DMVD methods to obtain multiple sets of motion compensation results, and generating the current block by filtering the multiple sets of motion compensation results using a median filter.

5.13. The method as described in any one of Examples 5.1 to 5.5, wherein the updating of the non-updated motion information based on multiple decoder-side motion vector derivation (DMVD) methods to generate updated motion information of the current block further includes: assigning different priorities to the multiple DMVD methods, selecting a valid DMVD method with the highest N priorities from the multiple DMVD methods, N is an integer and N>=1, and generating updated motion information for the current block based on the N DMVD methods.

5.14. A method as described in any one of Examples 5.1 to 5.13, wherein the current block is a prediction unit.

5.15. A method as described in any one of Examples 5.1 to 5.14, wherein the unupdated motion information includes an unupdated motion vector and a reference picture for each predicted direction.

5.16. A method as described in any one of Examples 5.1-5.15, wherein the method is not applied if the current block satisfies a specific condition.

5.17. The method of Example 5.16, wherein the specific condition specifies at least one of: the size of the current block, the slice type of the current block, the picture type of the current block, and the slice type of the current block.

5.18. The method as described in Example 5.16, wherein the specific condition specifies that the number of samples contained in the current block is less than a first threshold.

5.19. The method of Example 5.16, wherein the specific condition specifies that the minimum dimension of the width and height of the current block is less than or not greater than a second threshold.

5.20. The method of Example 5.16, wherein the specific condition specifies that the width of the current block is less than or not greater than a third threshold, and/or the height of the current block is less than or not greater than a fourth threshold.

5.21. The method as described in Example 5.16, wherein the specific condition specifies that the width of the current block is greater than or not less than a third threshold, and/or the height of the current block is greater than or not less than a fourth threshold.

5.22. A method as described in Example 5.16, wherein the method is applied at the sub-block level when the width and/or height of the block to which the sub-block belongs is equal to or greater than the fifth threshold.

5.23. The method of Example 5.22, wherein the current block is divided into a plurality of sub-blocks, and each of the plurality of sub-blocks is further subjected to a bidirectional optical flow (BIO) process in the same manner as a normal codec block having a size equal to the size of the sub-block.

5.24. A method as described in any of Examples 5.18-5.22, wherein each of the first to fifth thresholds is predefined or signaled at a sequence parameter set (SPS) level, or a picture parameter set (PPS) level, or a picture level, or a slice level, or a slice level.

5.25. The method of Example 5.24, wherein each of the first to fifth thresholds is defined based on codec information including at least one of a block size, a picture type, and a temporal layer index.

5.26. A device in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement the method described in any one of Examples 5.1 to 5.25.

5.27. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for performing the method described in any one of Examples 5.1 to 5.25.

6.1. A video processing method, comprising: generating an intermediate prediction for the current block based on first motion information associated with the current block; updating the first motion information to second motion information based on the intermediate prediction; and generating a final prediction for the current block based on the second motion information.

6.2. A method as in Example 6.1, wherein the first motion information is obtained using information decoded from a bitstream representation including the current block.

6.3. A method as in Example 6.1 or 6.2, wherein updating the first motion information includes using optical flow-based refinement.

6.4. The method of Example 6.3, wherein the use of optical flow-based refinement includes using at least one of the first motion information, spatial gradient, temporal gradient, and optical flow of each sub-block/pixel within the prediction block generated by using the first motion information to update the first motion information to the second motion information.

6.5. A method as in Example 6.4, wherein the second motion information is used at least to perform motion compensation and generate a final prediction for each sub-block/pixel within the current block.

6.6. The method of Example 6.1 or 6.2, wherein generating the intermediate prediction includes a first interpolation filtering process, and generating the final prediction includes a second interpolation filtering process, and wherein the first interpolation filtering process uses a second filter different from the second interpolation filtering process.

6.7. The method of Example 6.6, wherein the first or second interpolation filter processing may use a 4-tap filter, or a 6-tap filter or a bilinear filter.

6.8. A method as in any of Examples 6.6 to 6.7, wherein the filter taps or filter coefficients used in the interpolation filtering process are predefined.

6.9. A method as in Example 6.8, wherein the filter taps or filter coefficients used in the interpolation filtering process are based on codec information.

6.10. The method of Example 6.9, wherein the encoding and decoding information includes at least one of the following: the size of the current block, the shape of the current block, the predicted direction, and the slice type.

6.11. A method as in any of Examples 6.6-6.10, wherein different filters are selected for different blocks.

6.12. A method as in any of Examples 6.6-6.11, wherein one or more candidate sets of multiple filters are predefined, and a filter is selected for the current block from one or more candidate sets of the predefined multiple filters.

6.13. A method as in Example 6.11, wherein the filter selected for the current block is indicated by a signaled index or is dynamically derived without signaling.

6.14. A method as in any of Examples 6.1-6.13, wherein the intermediate prediction is generated using integer motion vectors and no interpolation filtering is applied in the generation of the intermediate prediction.

6.15. A method as in Example 6.14, wherein the fractional motion vector is rounded to the nearest integer motion vector.

6.16. The method of Example 6.15, wherein, in response to there being more than one nearest integer motion vector for a fractional motion vector, the fractional motion vector is rounded to the smallest of the nearest integer motion vectors.

6.17. A method as in Example 6.15, wherein, in response to there being more than one nearest integer motion vector for a fractional motion vector, the fractional motion vector is rounded to the largest of the nearest integer motion vectors.

6.18. The method of Example 6.15, wherein, in response to there being more than one nearest integer motion vector for the fractional motion vector, the fractional motion vector is rounded to the nearest integer motion vector that is closest to zero.

6.19. The method of Example 6.15, wherein the fractional motion vector is rounded to the nearest integer motion vector that is not less than the fractional motion vector.

6.20. A method as in Example 6.15, wherein the fractional motion vector is rounded to the nearest integer motion vector that is not greater than the fractional motion vector.

6.21. A method as in any of Examples 6.1-6.20, wherein use of the method is signaled in a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, a codec tree unit (CTU), a codec unit (CU), or a CTU group.

6.22. A method as in any of Examples 6.1-6.21, wherein whether the method is applied depends on the codec information.

6.23. The method of Example 6.22, wherein the encoding and decoding information includes at least one of the following: the size of the current block, the shape of the current block, the predicted direction, and the slice type.

6.24. A method as in Example 6.22, wherein the method is not automatically applied when the first condition is met.

6.25. The method of Example 6.24, wherein the first condition at least specifies encoding and decoding the current block using an affine mode.

6.26. A method as in Example 6.22, wherein the method is automatically applied when the second condition is met.

6.27. A method as in Example 6.26, wherein the second condition at least specifies that the block is encoded and decoded using bi-prediction and the block size is greater than a predetermined threshold.

6.28. A method as in method 6.24, wherein the first condition specifies at least one of: a size of the current block, a slice type of the current block, a picture type of the current block, and a slice type of the current block.

6.29. A method as in method 6.24, wherein the first condition specifies that the number of samples contained in the current block is less than a first threshold.

6.30. A method as in method 6.24, wherein the first condition specifies that the minimum dimension of the width and height of the current block is less than or not greater than the second threshold.

6.31. A method as in method 6.24, wherein the first condition specifies that the width of the current block is less than or not greater than a third threshold, and/or the height of the current block is less than or not greater than a fourth threshold.

6.32. A method as in method 6.24, wherein the first condition specifies that the width of the current block is greater than or not less than a third threshold, and/or the height of the current block is greater than or not less than a fourth threshold.

6.33. A method as in method 6.24, wherein the method is applied at the sub-block level when the width and/or height of the block to which the sub-block belongs is equal to or greater than the fifth threshold.

6.34. A method as in method 6.33, wherein the current block is partitioned into a plurality of sub-blocks, and each of the plurality of sub-blocks is further subjected to bidirectional optical flow (BIO) processing in the same manner as a normal codec block having a size equal to the sub-block size.

6.35. A method as in any of methods 6.24-6.33, wherein each of the first to fifth thresholds is predefined or signaled at a sequence parameter set (SPS) level, or a picture parameter set (PPS) level, or a picture level, or a slice level, or a slice level.

6.36. A method as in method 6.35, wherein each of the first to fifth thresholds is defined based on codec information including at least one of a block size, a picture type, and a temporal layer index.

6.37. A device in a video system, comprising a processor and a non-transitory memory, wherein the non-transitory memory stores instructions, wherein the instructions, when executed by the processor, cause the processor to implement any of the methods of Examples 6.1 to 6.36.

6.38. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for performing a method as in any of Examples 6.1 to 6.36.

6.39. A video decoding device, comprising: a processor configured to implement any method of Examples 6.1 to 6.36.

6.40. A video encoding device, comprising: a processor configured to implement the method of any one of Examples 6.1 to 6.36.

In this document, the term "video processing" may refer to video encoding, video decoding, video compression, or video decompression. For example, a video compression algorithm may be applied during the conversion from a pixel representation of a video to a corresponding bitstream representation, or vice versa.

It can be understood from the foregoing that this article has described specific embodiments of the currently disclosed technology for illustrative purposes, but various modifications can be made without departing from the scope of the present invention. Therefore, the currently disclosed technology is not limited except for the scope of the attached patent application.

The implementation and functional operation of the subject matter described in this patent document can be implemented in various systems, digital electronic circuits, or in computer software, firmware or hardware, including the structures disclosed in this specification and their structural equivalents, or in one or more combinations thereof. The subject matter described in this specification can be implemented as one or more computer program products, that is, one or more temporary and non-temporary computer program instruction modules encoded and decoded on a computer-readable medium, for being run by a data processing device or controlling the operation of a data processing device. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a storage device, a composition of matter that affects a machine-readable propagated signal, or a combination of one or more thereof. The terms "data processing unit" and "data processing apparatus" include all devices, equipment, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the apparatus may include code that creates an operating environment for the computer program in question, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, and a combination of one or more thereof.

A computer program (also called a program, software, software application, script, or code) may be written in any form of programming language (including compiled or interpreted languages) and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language file), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, subroutines, or portions of code). Computer programs can be deployed and executed on one or more computers, which are located at one site or distributed across multiple sites and interconnected by a communications network.

The processing and logic flows described in this specification may be performed by one or more programmable processors running one or more computer programs to perform functions by operating on input data and generating output. The processing and logic flows may also be performed by, and the apparatus may also be implemented as, dedicated logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

For example, processors suitable for running computer programs include general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor that executes instructions and one or more storage devices that store instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or be operatively coupled to one or more mass storage devices to receive data from them or transfer data to them, or both. However, a computer need not necessarily have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and storage devices, including, for example, semiconductor storage devices such as EPROM, EEPROM, and flash memory. The processor and memory may be supplemented by, or incorporated into, dedicated logic circuitry.

It is intended that the description together with the drawings be regarded as exemplary only, wherein exemplary means example. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. In addition, the use of "or" is intended to include "and/or" unless the context clearly indicates otherwise.

Although this patent document contains many details, they should not be construed as limitations on the scope of any invention or claim, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features described in this patent document in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments, or in any suitable subcombination. Furthermore, although features may be described as functioning in certain combinations, or even initially claimed as such, in some cases one or more features in a combination may be deleted from the claimed combination, and the claimed combination may be directed to subcombinations or variations of subcombinations.

Similarly, although operations are described in a particular order in the accompanying drawings, this should not be understood as requiring that the operations be performed in the particular order or sequential order shown, or that all of the operations shown be performed, in order to achieve the desired results. In addition, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

Only some embodiments and examples are described, and other embodiments, enhancements and variations can be made based on what is described and illustrated in this patent document.

4300:Methods

4310~4330: Steps

Claims (30)

A video processing method, comprising: generating an intermediate prediction for the current block using a bidirectional optical flow BIO based on first motion information associated with the current block; scaling the intermediate prediction and updating the first motion information to second motion information based on the scaling of the intermediate prediction; and performing motion compensation instead of performing the BIO based on the second motion information to generate a final prediction for each sub-block/pixel in the current block. As described in claim 1, wherein the first motion information is obtained using information decoded from a bit stream representation including the current block. As described in claim 1 or 2, wherein updating the first motion information includes using optical flow-based refinement. As described in item 3 of the patent application, the use of optical flow-based refinement includes using the first motion information to update the first motion information to the second motion information by using at least one of the spatial gradient, temporal gradient, and optical flow of each sub-block/pixel in the prediction block generated using the first motion information. As described in claim 1, wherein generating the intermediate prediction includes a first interpolation filtering process, and generating the final prediction includes a second interpolation filtering process, and wherein the first filter used in the first interpolation filtering process is different from the second filter used in the second interpolation filtering process. As described in claim 5, the first or second interpolation filter processing may use a 4-tap filter, or a 6-tap filter or a bilinear filter. A method as described in any one of items 5 to 6 of the patent application scope, wherein the filter taps or filter coefficients used in the interpolation filtering process are predefined. As described in claim 7, the filter taps or filter coefficients used in the interpolation filtering process are based on codec information. As described in patent application scope 8, the encoding and decoding information includes at least one of the following: the size of the current block, the shape of the current block, the predicted direction and the slice type. As described in claim 5, different filters are selected for different blocks. As described in claim 5, one or more candidate sets of multiple filters are predefined, and a filter is selected for the current block from one or more candidate sets of the predefined multiple filters. As described in claim 10, the filter selected for the current block is indicated by an index signaled by signaling or dynamically derived without signaling. As described in claim 1, wherein the intermediate prediction is generated using an integer motion vector and no interpolation filtering is applied in the generation of the intermediate prediction. A method as described in claim 13, wherein the fractional motion vector is rounded to the nearest integer motion vector. As described in claim 14, wherein, in response to the existence of more than one closest integer motion vector for the fractional motion vector, the fractional motion vector is rounded to the smallest of the closest integer motion vectors. As described in claim 14, wherein, in response to the existence of more than one nearest integer motion vector for the fractional motion vector, the fractional motion vector is rounded to the largest of the nearest integer motion vectors. As described in claim 14, wherein, in response to there being more than one nearest integer motion vector for the fractional motion vector, the fractional motion vector is rounded to the nearest integer motion vector that is closest to zero. As described in claim 14, the fractional motion vector is rounded to one of the nearest integer motion vectors that is not less than the fractional motion vector. As described in claim 14, wherein the fractional motion vector is rounded to one of the nearest integer motion vectors that is not greater than the fractional motion vector. As described in item 1 of the patent application scope, the use of the method is notified by signaling in a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, a codec tree unit (CTU), a codec unit (CU) or a CTU group. A method as described in item 1 of the patent application scope, wherein whether to apply the method depends on encoding and decoding information. As described in item 21 of the patent application scope, the encoding and decoding information includes at least one of the following: the size of the current block, the shape of the current block, the predicted direction and the slice type. A method as claimed in claim 21, wherein the method is not automatically applied when the first condition is met. As described in item 23 of the patent application, the first condition at least specifies encoding and decoding the current block using an affine mode. A method as described in claim 21, wherein the method is automatically applied when the second condition is met. As described in claim 25, the second condition at least specifies that the block is encoded and decoded using dual prediction, and the block size is greater than a predetermined threshold. A device in a video system, comprising a processor and a non-temporary memory, wherein the non-temporary memory stores instructions, wherein the instructions, when executed by the processor, enable the processor to implement the method described in any one of items 1 to 26 of the patent application scope. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for executing a method as described in any one of items 1 to 26 of the patent application scope. A video decoding device, comprising: a processor, which is configured to implement the method described in any one of items 1 to 26 of the patent application scope. A video encoding device includes: a processor configured to implement the method described in any one of items 1 to 26 of the patent application scope.