JP2023134521A - Decoder side motion vector derivation - Google Patents

Abstract

To provide a video and image encoding, decoding technique.SOLUTION: A visual media processing method includes performing a conversion between a current block of visual media data and a corresponding coded representation of the visual media data, the conversion of the current block including determining whether the use of one or both of a bi-directional optical flow (BIO) technique and a decoder-side motion vector refinement (DMVR) technique to the current block is enabled or disabled, and the determining the use of the BIO technique or the DMVR technique being based on a cost criterion associated with the current block.SELECTED DRAWING: Figure 10

Description

Cross-Reference to Related Applications Under the applicable patent laws and/or regulations under the Paris Convention, this application is filed under International Patent Application No. PCT/CN2019/081155, filed on April 2, 2019, filed on May 7, 2019 The purpose is to timely claim the priority rights and benefits of International Patent Application No. PCT/CN2019/085796. For all purposes under United States law, the entire disclosure of the above application is incorporated by reference as part of the disclosure herein.

The present specification relates to video and image encoding and decoding techniques.

Digital video accounts for the largest amount of bandwidth usage on the Internet and other digital communication networks. It is expected that the bandwidth demands for digital video usage will continue to increase as the number of connected user equipment capable of receiving and displaying video increases.

In one example aspect, a method of processing video is disclosed. The method includes converting between a current block of visual media data and a corresponding coding representation of visual media data, wherein converting the current block employs bidirectional optical flow (BIO) techniques or determining whether use of one or both of decoder-side motion vector refinement (DMVR) techniques is enabled or disabled for the current block; determining the use of BIO techniques or DMVR techniques; is based on the cost criteria associated with the current block.

In another exemplary aspect, a method of processing video is disclosed. The method includes converting between a current block of visual media data and a corresponding coding representation of the visual media data, wherein the converting the current block includes decoder-side motion vector refinement (DMVR). ) technique is enabled or disabled for use on the current block, and the DMVR technique includes a mean sum removed of absolute differences. (MRSAD) comprising refining the motion information of the current block based on a cost criterion other than a cost criterion.

In another exemplary aspect, a method of processing video is disclosed. The method includes converting between a current block of visual media data and a corresponding coding representation of the visual media data, wherein the converting the current block includes bidirectional optical flow (BIO). determining whether use of one or both of a BIO technique or a decoder-side motion vector refinement (DMVR) technique is enabled or disabled for the current block; Determining the usage is based on calculating that a difference in mean values of a pair of reference blocks associated with the current block exceeds a threshold.

In another exemplary aspect, a method of processing video is disclosed. The method comprises: modifying a first reference block to generate a first modified reference block; modifying a second reference block to generate a second modified reference block; said first reference block and said second reference block are both associated with a current block of visual media data; and said first modified reference block and said second modified reference block. determining a difference between a sum of absolute transformed differences (SATD) and a mean removed sum of absolute transformed differences (SATD); of absolute transformed differences) (MRSATD),
Contains one or more of the following: sum of squares error (SSE), mean removed sum of squares error (MRSSE), mean difference, or slope value , and performing a transformation between the current block of visual media data and a corresponding coding representation of the visual media data, the transformation comprising determining the first reference block and the second reference block. and the use of said difference between said second modified reference block generated from respectively modifying the reference block.

In another exemplary aspect, a method of processing video is disclosed. The method includes determining a temporal gradient or a modified temporal gradient using a reference picture associated with a current block of visual media data, the method comprising: determining a temporal gradient or a modified temporal gradient; The gradient includes determining a difference between the reference pictures and performing a transformation between the current block of visual media data and a corresponding coding representation of the visual media data, the transformation comprising: using bidirectional optical flow (BIO) techniques based in part on the temporal gradient or the modified temporal gradient.

In another exemplary aspect, a method of processing video is disclosed. The method includes determining a first temporal gradient using a reference picture associated with a first video block or a sub-block thereof; and a reference picture associated with a second video block or sub-block thereof. and modifying the first temporal gradient to generate a modified first temporal gradient and a modified second temporal gradient. and performing a modification of the second temporal gradient, the modification of the first temporal gradient associated with the first video block comprising: modifying the second temporal gradient associated with the second video block. and converting the first video block and the second video block into the corresponding coding representations.

In another exemplary aspect, a method of processing video is disclosed. The method includes modifying one or both of a first cross-reference block and a second cross-reference block associated with the current block; determining a spatial gradient associated with the current block according to applying a bidirectional optical flow (BIO) technique based on using the one or both of the second cross-reference blocks; , performing a transformation between the current block and a corresponding coding representation, the transformation comprising using the spatial gradient associated with the current block; .

In another exemplary aspect, a method of processing video is disclosed. The method is characterized in that a flag signaled at the block level causes the processing unit to perform one or both of a decoder-side motion vector refinement (DMVR) technique or a bidirectional optical flow (BIO) technique for the current block. making a determination indicating, at least in part, that the current block should be enabled; and converting between the current block and a corresponding coding representation, wherein the coding representation is based on the DMVR technique and/or the coding representation. or performing a conversion that includes the flag indicating whether to enable the one or both of the BIO techniques.

In another exemplary aspect, a method of processing video is disclosed. The method includes determining, by a processing unit, that a decoder-side motion vector refinement (DMVR) technique should be enabled for a current block, the determining determining only on the height of the current block. and converting between the current block and a corresponding coding representation.

In another exemplary aspect, a method of processing video is disclosed. The method includes performing a transformation between a current block of visual media data and a corresponding coding representation of the visual media data, the transformation comprising decoder side motion vector refinement (DMVR) for the current block. or Bidirectional Optical Flow (BIO) technique, the rules associated with the DMVR technique are compliant with application to the BIO technique, and the rules associated with the DMVR technique are compliant with application to the BIO technique; Determining whether the use of the technique or the one or both of the DMVR techniques is enabled or disabled is based on applying the rules.

In another example aspect, the method described above may be implemented by a video decoder that includes a processing device.

In another example aspect, the method described above may be implemented by a video encoder that includes a processing device.

In yet another exemplary aspect, these methods may be implemented in the form of processor-executable instructions and stored on a computer-readable program medium.

These and other aspects are further described herein.

An example of bilateral matching is shown. An example of template matching is shown. Figure 3 shows an example of unidirectional motion estimation (ME) in frame rate up conversion (FRUC). An example of an optical flow trajectory is shown. An example of bidirectional optical flow (BIO) without block expansion is shown. An example of bidirectional optical flow (BIO) without block expansion is shown. An example of bilateral matching using 6-point search is shown. Examples of adaptive integer search patterns and half-sample search patterns are shown. FIG. 1 is a block diagram showing an example of a video processing device. FIG. 2 is a block diagram illustrating an example implementation of a video encoder. 3 is a flowchart illustrating an example of a video processing method. 3 is a flowchart illustrating an example of a video processing method. 3 is a flowchart illustrating an example of a video processing method. 3 is a flowchart illustrating an example of a video processing method. 3 is a flowchart illustrating an example of a video processing method. 3 is a flowchart illustrating an example of a video processing method. 3 is a flowchart illustrating an example of a video processing method. FIG. 1 is a block diagram illustrating an example video processing system in which the disclosed techniques may be implemented. 3 is a flowchart illustrating an example of a video processing method. 3 is a flowchart illustrating an example of a video processing method. 3 is a flowchart illustrating an example of a video processing method. 3 is a flowchart illustrating an example of a video processing method. 3 is a flowchart illustrating an example of a video processing method. 3 is a flowchart illustrating an example of a video processing method. 3 is a flowchart illustrating an example of a video processing method. 3 is a flowchart illustrating an example of a video processing method. 3 is a flowchart illustrating an example of a video processing method. 3 is a flowchart illustrating an example of a video processing method.

To improve video compression rates, researchers are constantly seeking new techniques for encoding video. This specification provides various techniques that can be used by a video bitstream decoder to improve the quality of decompressed or decoded digital video. Additionally, a video encoder may implement these techniques during the encoding process to reconstruct decoded frames used for further encoding.

Section headings are used herein to improve readability and are not intended to limit the scope of the techniques and embodiments described in each chapter to that section only. Additionally, although we have used specific terminology from various existing video codec standards, the disclosed technology is not limited solely to these video standards or their successors, but may also apply to other video codec standards. It is possible. Furthermore, it is understood that in some cases techniques are disclosed that use corresponding coding steps, and corresponding decoding steps in reverse order are performed at the decoder. Coding may also be used to perform transcoding, where the video is represented from one coding representation (eg, one bit rate) to another coding representation (eg, a different bit rate).

1. overview

TECHNICAL FIELD This specification relates to video coding technology. Specifically, it relates to motion compensation in video coding. It may be applied to an existing video coding standard such as HEVC, or it may be applied to finalize a standard (Versatile Video Coding). The invention is also applicable to future video coding standards or video codecs.

2. background

Video coding standards have evolved primarily through the development of the well-known ITU-T and ISO/IEC standards. ITU-T is H. 261 and H. ISO/IEC created MPEG-1 and MPEG-4 Visual, and both organizations created H.263. 262/MPEG-2 Video and H.262/MPEG-2 Video. 264/MPEG-4 AVC (Advanced Video Coding) and H.264/MPEG-4 AVC (Advanced Video Coding). They jointly created the H.265/HEVC standard. H. Since H.262, video coding standards are based on a hybrid video coding structure where temporal prediction and transform coding are utilized. In 2015, VCEG and MPEG jointly established the Joint Video Exploration Team (JVET) to explore future video coding technologies beyond HEVC. Since then, many new methods have been adopted by JVET and incorporated into a reference software called JEM (Joint Exploration Mode). In April 2018, the Joint Video Expert Team (JVET) was established between VCEG (Q6/16) and ISO/IEC JTC1 SC29/WG11 (MPEG), and has achieved a 50% bit rate reduction compared to HEVC. The goal is to formulate a VVC standard.

The latest version of the VVC draft, namely Universal Video Coding (Draft 2), can be found below.
http://phenix. it-sudparis. eu/jvet/doc_end_user/documents/11_Ljubljana/wg11/JVET-K1001-v7. zip. The latest reference software for VVC, called VTM, can be found below.
https://vcgit. hhi. fraunhofer. de/jvet/VVCSoftware_VTM/tags/VTM-2.1.

FIG. 9 is a block diagram of an example implementation of a video encoder. Figure 9 shows that the encoder implementation incorporates a feedback path in which the video encoder also performs video decoding functions (reconstructing the compressed representation of the video data for use in encoding the next video data). show.

2.1 Pattern matching motion vector derivation

PMMVD (Pattern Matched Motion Vector Derivation) mode is a special merge mode based on FRUC (Frame-Rate Up Conversion) technology. In this mode, the motion information of the block is not signaled and is derived at the decoder side.

If that merge flag is true, the FRUC flag is signaled to the CU. If the FRUC flag is false, a merge index can be signaled and normal merge mode is used. If the FRUC flag is true, an additional FRUC mode flag is signaled to indicate which method (bilateral matching or template matching) is used to derive motion information for the block.

On the encoder side, the decision whether to use FRUC merge mode for a CU is based on the RD cost selection, just as it is done for regular merge candidates. That is, RD cost selection is used to check both two matching modes (bilateral matching and template matching) for one CU. The one leading to the minimum cost is further compared to other CU modes. If the FRUC matching mode is the most efficient one, set the FRUC flag to true for the CU and use the associated matching mode.

The motion derivation process in FRUC merge mode has two steps. First, CU-level motion search is performed, and then sub-CU-level motion refinement is performed. At the CU level, we derive the initial motion vector for the entire CU based on bilateral matching or template matching. First, we generate a list of MV candidates and select the candidate that leads to the minimum matching cost as the starting point for further CU level improvement. Then, a local search is performed based on bilateral matching or template matching near the starting point, and the MV result with the minimum matching cost is set as the MV for the entire CU. Next, using the derived CU motion vector as a starting point, the motion information at the sub-CU level is further refined.

For example, to derive W×H CU motion information, the following derivation process is performed. In the first stage, the MV for the entire W×H CU is derived. In the second stage, the CU is further divided into M×M sub-CUs. The value of M is calculated as in (16), where D is a predefined division depth and is set to 3 by default in JEM. Then, the MV of each sub-CU is derived as follows.

As shown in Fig. 1, this bilateral matching derives the motion information of the current CU by finding the closest matching between two blocks along the motion trajectory of the current CU in two different reference pictures. used for Assuming continuous motion trajectories, motion vectors MV0 and MV1 pointing to the two reference blocks are proportional to the temporal distance between the current picture and the two reference pictures, e.g. TD0 and TD1. As a special case, if the current picture is temporally between two reference pictures, and the temporal distances from the current picture to the two reference pictures are the same, then bilateral matching is mirror-based. This will be the MV for the future.

As shown in Figure 2, by finding the closest match between the template in the current picture (neighboring block above and/or to the left of the current CU) and the block in the reference picture (same size as the template). , template matching is used to derive the motion information of the current CU. Besides the aforementioned FRUC merge mode, template matching also applies to AMVP mode. In JEM, like HEVC, AMVP has two candidates. New candidates are derived by using a template matching method. If the newly derived candidate by template matching is different from the first existing AMVP candidate, it is inserted at the beginning of the AMVP candidate list, and then the list size is set to 2 (removing the second existing AMVP candidate). meaning). When applied to AMVP mode, only CU level search is applied.
CU level MV candidate set

The CU level MV candidate set may include:
・Original AMVP candidate when the current CU is in AMVP mode,
・All merge candidates,
- multiple MVs within an interpolated MV field, as introduced in Section 2.1.1.3;
・Motion vectors in the upper and left neighborhoods

When using bilateral matching, each valid MV of the merge candidate is used as input to generate MV pairs assuming bilateral matching. For example, one valid MV of a merge candidate is (MVa, refa) in the reference list A, and the reference picture refb of its paired bilateral MV is found in the other reference list B, refa and refb is on a different side of the current picture in time. If no such refb is available in reference list B, determine refb as a different reference from refa, and the temporal distance to the current picture is the minimum value in list B. After determining refb, MVb is derived by scaling MVa based on the temporal distance between the current picture and refa, refb.

The four MVs from the interpolated MV field are also added to the CU level candidate list. Specifically, the interpolated MVs of the positions (0, 0), (W/2, 0), (0, H/2), (W/2, H/2) of the current CU are added. .

When applying FRUC in AMVP mode, the original AMVP candidates are also added to the CU level MV candidate set.

At the CU level, add up to 15 MVs for AMVP CUs and up to 13 MVs for merge CUs to the candidate list.

Sub-CU level MV candidate set

The sub-CU level MV candidate set may include the following:
・MV determined from CU level search,
・MV near the top, left, top left, and top right,
a scaled version of the juxtaposed MV from the reference picture;
・Up to 4 ATMVP candidates,
・Up to 4 STMVP candidates

The scaled MV from the reference picture is derived as follows. Traverse all reference pictures in both lists. The MV at the array position of the sub-CU in the reference picture is scaled with respect to the reference of the starting CU level MV.

ATMVP and STMVP candidates are limited to the first four candidates

At the sub-CU level, up to 17 MVs are added to the candidate list.

Generation of interpolated MV field

Before coding a frame, generate an interpolated motion field for the entire picture based on one ME. Then, this motion field may be used later as an MV candidate at the CU level or sub-CU level.

First, the motion field of each reference picture in both reference lists is traversed at the 4x4 block level. For each 4x4 block, if the motion associated with the block passes through the 4x4 block of the current picture (shown in Figure 3) and has not yet been assigned an interpolated motion, then based on the temporal distances TD0 and TD1 (Similar to MV scaling of TMVP in HEVC), scale the motion of the reference block to the current picture and assign the scaled motion to the block of the current frame. If a 4x4 block is not assigned a scaled MV, the block's motion is marked as unavailable in the interpolated motion field.

Interpolation and matching costs

Motion compensated interpolation can be performed if one motion vector points to one fractional sample position. To reduce complexity, we use bilinear interpolation for both bilateral and template matching instead of the usual 8-tap HEVC interpolation.

The matching cost calculation is slightly different for different steps. When selecting a candidate from a CU-level candidate set, the matching cost is the sum of absolute differences (SAD) of bilateral matching or template matching. After determining the starting MV, the matching cost C of bilateral matching in sub-CU level search is calculated as follows.

Here, w is a weighting factor empirically set to 4, and MV and MV ^S denote the current MV and starting MV, respectively. SAD is still used as matching cost for template matching in sub-CU level search.

In FRUC mode, MV is derived by using only luminance samples. The derived motion is used for both luminance and saturation for MC inter prediction. After determining the MV, final MC is performed using an 8-tap interpolation filter for luminance and a 4-tap interpolation filter for chroma.

Improvement of MV

MV refinement is a pattern-based MV search with criteria of bilateral matching cost or template matching cost. JEM supports two search patterns: unrestricted center-biased diamond search (UCBDS) and adaptive traversal search for MV refinement at the CU and sub-CU levels, respectively. For both CU and sub-CU level MV refinement, the MV is directly searched with an accuracy of 1/4 luminance sample MV, followed by 1/8 luminance sample MV refinement. The search range of MV refinement for CU and sub-CU steps is set equal to 8 luminance samples.

Selection of prediction direction in template matching FRUC merge mode

In bilateral matching merge mode, bidirectional prediction is always applied to derive the motion information of a CU based on the closest matching between two blocks along the current CU's motion trajectory in two different reference pictures. be done. There is no such limitation for template matching merge mode. In template matching merge mode, the encoder can choose between single prediction from list0, single prediction from list1, or bidirectional prediction for the CU. The selection is performed as follows based on the template matching cost.

When costBi≦factor*min (cost0, cost1), bidirectional prediction is used.
In other cases, if cost0≦cost1, use single prediction from list0.
If not,
Use a single prediction from list1.

Here, cost0 is the SAD of list0 template matching, cost1 is the SAD of list1 template matching, and costBi is the SAD of bidirectional predictive template matching. When the value of factor is 1.25, it means that the selection process is biased toward bidirectional prediction.

This inter-prediction direction selection is only applied to the CU-level template matching process.

Hybrid intra and inter prediction

JVET-L0100 proposes multiple hypothesis prediction using hybrid intra and inter prediction as one method for generating multiple hypotheses.

When applying multi-hypothesis prediction to intra mode refinement, multi-hypothesis prediction combines one intra prediction and one merge index prediction. In the merge CU, if the flag is true, signal one flag for merge mode and select one intra mode from the intra candidate list. For the luminance component, the intra candidate list is derived from four intra prediction modes including DC, planar, horizontal, and vertical modes, and the size of the intra candidate list is 3 or 4 depending on the shape of the block. I can do it. Horizontal mode eliminates the intra mode list if the CU width is greater than twice the CU height, and vertical mode is removed from the intra mode list if the CU height is greater than twice the CU width. be done. A weighted average is used to combine one intra prediction mode selected by the intra mode index and one merge indexed prediction selected by the merge index. For chroma components, DM is always applied without extra signaling. The weights for combining predictions are written as: If DC or planar mode is selected, or if the width or height of the CB is less than 4, equal weights are applied. When the horizontal/vertical mode is selected for a CB whose width and height are four or more, first, one CB is divided into four equal area areas in the vertical/horizontal direction. Each weight set is expressed as (w_intra _i , w_inter _i ), where i is 1 to 4, (w_intra ₁ , w_inter ₁ )=(6,2), (w_intra ₂ ,w_inter ₂ )=(5,3 ), (w_intra ₃ , w_inter ₃ ) = (3, 5), (w_intra ₄ , w_inter ₄ ) = (2, 6), then it is applied to the corresponding region. (w_intra ₁ , w_inter ₁ ) is for the region closest to the reference sample, and (w_intra ₄ , w_inter ₄ ) is for the region furthest from the reference sample. A composite prediction can then be calculated by summing the two weighted predictions and the 3-bit right shift. Also, the intra-prediction mode for the intra-hypothesis of the predictor can be omitted to refer to the following nearby CUs.

Bidirectional optical flow (BIO)

In BIO, motion compensation is first performed to generate the first prediction (in each prediction direction) of the current block. The first prediction is used to derive the spatial gradient, temporal gradient, and optical flow for each subblock/pixel within the block, which are used to derive the second prediction, i.e., for each subblock/pixel. Generate the final prediction. The details will be explained below.

Bidirectional optical flow (BIO) is an improvement on sample-wise motion performed on top of block-wise motion compensation for bidirectional prediction. Sample-level motion refinement does not use signaling.

Here, the luminance value from the reference k (k=0, 1) after block motion compensation is I ^(k) , and ∂I ^(k) /∂x and ∂I ^(k) /∂y are respectively I ⁽ Let be the horizontal and vertical components of the gradient of ^k) . Assuming optical flow is enabled, the motion vector field (v _x , v _y ) is given by the following equation.

By combining this optical flow equation with Hermitian interpolation for the motion trajectory of each sample, both functional values I ^(k) at both ends and derivatives ∂I ^(k) /∂x, ∂I ^(k) /∂ A unique cubic polynomial matching y is obtained. The value of this polynomial at t=0 is the BIO prediction, such as:

Here, τ ₀ and τ ₁ indicate the distance to the reference frame, as shown in FIG. 4. The distances τ ₀ and τ ₁ are calculated as follows based on the POC of Ref0 and Ref1. τ ₀ = POC (current) - POC (Ref0), τ ₁ = POC (Ref1) - POC (current). If both predictions come from the same time direction (both from the past or both from the future), they have different signs (i.e., τ ₀ and τ ₁ <0). Here, BI0 is only applied if the predictions are not from the same time (i.e. τ ₀ ≠ τ ₁ ), and both reference regions have non-zero motion (MVx ₀ , MVy ₀ , MVx ₁ , MVy ₁ ≠0), and the block motion vector is proportional to the temporal distance (MVx ₀ /MVx ₁ =MVy ₀ /MVy ₁ =-τ ₀ /τ ₁ ).

The motion vector field (v _x , v _y ) is determined by minimizing the difference Δ between the values at points A and B (the intersection of the motion trajectory and the reference frame plane on FIG. 9). The model uses only the first linear term of the local Taylor expansion for Δ as follows.

All values in Equation 5 depend on the sample position (i', j') and have been omitted from the notation so far. Assuming that the motion is consistent in the local surrounding area, Δ can be minimized inside a (2M+1)×(2M+1) square window Ω centered at the current prediction point (i,j). can. where M is equal to 2.

For this optimization problem, JEM uses a simple approach that first minimizes vertically and then horizontally. The result is as follows.

In the formula:

In order to avoid division by zero or very small numbers, we introduce regularization parameters r and m in Equation 7 and Equation 8.
r=500・4 ^d-8 (10)
m=700・4 ^d-8 (11)
Here, d is the bit depth of the video sample.

To make the memory access for BIO the same as normal bidirectional predictive motion compensation, all predicted and gradient values I(k), ∂I ^(k) /∂x only for the position within the current block. , ∂I ^(k) /∂y. In Equation 9, a (2M+1)×(2M+1) square window Ω centered at the current prediction point on the boundary of the prediction block accesses positions outside the block (as shown in Figure 5(a)). can. In JEM, the values of I ^(k) , ∂I ^(k) /∂x, ∂I ^(k) /∂y outside the block are set equal to the nearest available number inside the block. . For example, this may be implemented as padding, as shown in Figure 5(b).

Using BIO, the motion field can be refined sample by sample. To reduce computational complexity, JEM uses a block-based BIO design. Calculate motion refinement based on 4x4 blocks. In block-based BIO, we integrate the values of s _n in Equation 9 for all samples in the 4x4 block, and then use this integrated value of s _n to move the BIO motion for the 4x4 block. Derive the vector offset. Specifically, the following formula is used for BIO derivation based on blocks.

where b _k represents the set of samples belonging to the kth 4×4 block of the prediction block, replacing s _n in equations 7 and 8 with ((s _{n, bk} ) >> 4), and Derive the motion vector offset.

In some cases, BIO's MV management may be unreliable due to noise or irregular motion. Therefore, in BIO, the magnitude of the MV regimen is clipped to the threshold thBIO. The threshold is determined based on whether the current picture's reference pictures are all from one direction. If all reference pictures of the current picture are from one direction, set the threshold to 12×2 ^14-d , otherwise set the threshold to 12×2 ^13-d .

The gradient of BIO is calculated simultaneously with motion compensation interpolation using calculations based on HEVC motion compensation processing (2D separable FIR). The input for this 2D separable FIR is the same reference frame sample as for motion compensation processing and fractional position (fracX, fracY) according to the fractional part of the block motion vector. For horizontal gradient ∂I/∂x, the signal is first interpolated vertically using BIOfilterS corresponding to fractional position fracY with descaling shift d-8, then gradient filter BIOfilterG to fractional position fracX. Apply a corresponding horizontal descaling shift by 18-d. For vertical gradient ∂I/∂y, first apply the first gradient filter vertically using BIOfilterS corresponding to fractional position fracY with descaling shift d-8, then apply BIOfilterS horizontally direction, the signal displacement corresponding to the fractional position fracX is descaled by 18-d. To maintain reasonable complexity, the interpolation filter lengths for gradient computation BIOfilterG and signal displacement BIOfilterF are shorter (6 taps). Table 1 shows the filters used for gradient calculation of different fractional positions of block motion vectors in BIO. Table 2 shows interpolation filters that can be used to generate prediction signals in BIO.

In the present JEM, BIO is applied to all bi-directionally predicted blocks if the two predictions are from different reference pictures. If LIC is enabled for the CU, BIO is disabled.

In this JEM, OBMC is applied to one block after normal MC processing. To reduce computational complexity, no BIO is applied during OBMC processing. That is, BIO is applied only in MC processing of one block when using its own MV, and is not applied in MC processing when using MV of a neighboring block in OBMC processing.

A two-step early termination method is used to conditionally invalidate BIO operations based on the similarity of the two predicted signals. Early termination is applied first at the CU level and then at the sub-CU level. Specifically, the proposed method first calculates the SAD between the L0 prediction signal and the L1 prediction signal at the CU level. Assuming that BIO is applied only to luminance, only luminance samples can be considered for SAD calculation. If the CU level SAD is below a predefined threshold, BIO processing is completely disabled for the entire CU. The CU level threshold is set to 2 ^(BDepth-9) for each sample. If BIO processing is not disabled at the CU level and the current CU includes multiple sub-CUs, calculate the SAD of each sub-CU in the CU. Then, at the sub-CU level, it is determined whether to enable or disable bio-processing based on a predefined sub-CU level SAD threshold set to 3*2 ^(BDepth-10) for each sample.

2.4 BDOF specifications in VVC

The specifications of BDOF (JVET-N1001-v2) are as follows.

8.5.7.4 Bidirectional optical flow prediction processing

The inputs to this process are as follows.
- two variables nCBW and nCBH, defining the width and height of the current coding block;
- two (nCBW+2)×(nCBH+2) luminance prediction sample arrays predSamplesL0 and predSamplesL1,
- prediction list usage flags predFlagL0 and predFlagL1,
- reference indexes refIdxL0 and refIdxL1,
- Bidirectional optical flow utilization flag bdofUtilizationFlag[xIdx][yIdx], where xIdx=0. ．． (nCbW>>2)-1, yIdx=0. ．． (nCbH>>2)-1

The output of this process is an (nCBW) x (nCBH) array pbSamples of luminance prediction sample values.

The variables bitDepth, shift1, shift2, shift3, shift4, offset4, and mvRefineThres are derived as follows.
- The variable bitDepth is set equal to BitDepth _Y.
- The variable shift1 is set equal to Max(2,14-bitDepth).
- The variable shift2 is set equal to Max(8, bitDepth-4).
- The variable shift3 is set equal to Max(5, bitDepth-7).
- The variables shift4 and Max(3,15-bitDepth) are set equal and the variable offset4 is set equal to 1<<(shift4-1).
- The variable mvRefineThres is set equal to Max(2,1<<(13-bitDepth)).

xIdx=0. ．． (nCbW>>2)-1 and yIdx=0. ．． If (nCbH>>2)-1, the following applies.
- Set variable xSb equal to (xIdx<<2)+1 and set ySb equal to (yIdx<<2)+1.
- If bdofUtilizationFlag[xSbIdx][yIdx] is equal to FALSE, then x=xSb-1. ．． xSb+2, y=ySb-1. ．． For ySb+2, the predicted sample value of the current subblock is derived as follows.
pbSamples[x][y]=Clip3(0, (2 ^bitDepth )-1, (predSamplesL0[x+1][y+1]+offset2+predSamplesL1[x+1][y+1])>>shift2) (8-852)
- Otherwise (bdofUtilizationFlag[xSbIdx][yIdx] equals TRUE), the predicted sample value of the current sub-block is derived as follows:

- x=xSb-1. ．． xSb+4, y=ySb-1. ．． For ySb+4, the following ordered steps apply.

1. The position (h _x , v _y ) of each corresponding sample position (x, y) in the predicted sample array is derived as follows.
h _x =Clip3(1,nCbW,x) (8-853)
v _y =Clip3(1,nCbH,y) (8-854)

2. The variables gradientHL0[x][y], gradientVL0[x][y], gradientHL1[x][y], and gradientVL1[x][y] are derived as follows.
gradientHL0[x][y] = (predSamplesL0[h _x +1] [v _y ] - predSampleL0[h _x -1] [v _y ]) >> shift1 (8-855)
gradientVL0[x][y]=(predSampleL0[h _x ][v _y +1]-predSampleL0[h _x ][v _y -1])>>shift1 (8-856)
gradientHL1[x][y] = (predSamplesL1[h _x +1] [v _y ] - predSampleL1[h _x -1] [v _y ]) >> shift1 (8-857)
gradientVL1[x][y]=(predSampleL1[h _x ][v _y +1]-predSampleL1[h _x ][v _y -1])>>shift1 (8-858)

3. The variables temp[x][y], tempH[x][y] and tempV[x][y] are derived as follows.
diff [x] [y] = (predSamplesL0[h _x ] [v _y ] >> shift2) - (predSamplesL1 [h _x ] [v _y ] >> shift2) (8-859)
tempH[x][y]=(gradientHL0[x][y]+gradientHL1[x][y])>>shift3 (8-860)
tempV[x][y]=(gradientVL0[x][y]+gradientVL1[x][y])>>shift3 (8-861)

- The variables sGx2, sGy2, sGxGy, sGxdI, sGydI are derived as follows.
sGx2=Σ _i Σ _j (tempH[xSb+i][ySb+j]*tempH[xSb+i][ySb+j])with i,j=-1. ．． 4 (8-862)
sGy2=Σ _i Σ _j (tempV[xSb+i][ySb+j]*tempV[xSb+i][ySb+j])with i,j=-1. ．． 4 (8-863)
sGxGy=Σ _i Σ _j (tempH[xSb+i][ySb+j]*tempV[xSb+i][ySb+j])with i,j-1. ．． 4 (8-864)
sGxdI=Σ _i Σ _j (-tempH[xSb+i][ySb+j]*diff[xSb+i][ySb+j])with i,j=-1. ．． 4 (8-865)
sGydI=Σ _i Σ _j (-tempV[xSb+i][ySb+j]*diff[xSb+i][ySb+j])with i,j=-1. ．． 4 (8-866)

- The horizontal and vertical motion offsets of the current subblock are derived as follows:
v _x =sGx2>0? Clip3(-mvRefineThres, mvRefineThres, -(sGxdI<<3)>>Floor(Log2(sGx2))): 0 (8-867)
v _y =sGy2>0? Clip3 (-mvRefineThres, mvRefineThres, ((sGydI<<3)-((v _x *sGxGy _m )<<12+v _x *sGxGy _s )>>1)>>Floor(Log2(sGx2))): 0 (8- 868)

- x=xSb-1. ．． xSb+2, y=ySb-1. ．． For ySb+2, the predicted sample value of the current subblock is derived as follows.
bdofOffset=Round((v _x *(gradientHL1[x+1][y+1]-gradientHL0[x+1][y+1]))>>1)+Round((v _y *(gradientVL1[x+1][y+1]-gradientVL0[ x+1][ y+1]))>>1) (8-869)
[Ed. (JC): The Round() operation is defined for float inputs. The Round( ) operation seems redundant here because the input is an integer value. Recommender should confirm] pbSamples[x][y]=Clip3(0, (2 ^bitDepth )-1, (predSamplesL0[x+1][y+1]+offset4+predSamplesL1[x+1][y+1]+bdofOffset) >> shif t4) (8- 870)

The spatial gradient is calculated as follows.
gradientHL0[x][y] = (predSamplesL0[h _x +1] [v _y ] - predSampleL0[h _x -1] [v _y ]) >> shift1 (8-855)

On the other hand, the temporal gradient is calculated as follows.
diff [x] [y] = (predSamplesL0[h _x ] [v _y ] >> shift2) - (predSamplesL1 [h _x ] [v _y ] >> shift2) (8-859)

Thus, spatial and temporal gradient calculations are not aligned.

2.5 Improvement of motion vector on decoder side

In the bidirectional prediction calculation, in order to predict one block area, bidirectional prediction blocks configured using the motion vectors (MV) of list0 and the MV of list1 are combined to form one prediction signal. In JVET-K0217, a decoder-side motion vector refinement (DMVR) method, two motion vectors for bidirectional prediction are further refined by a bilateral matching process.

In the proposed method, DMVR is applied only in merge mode and skip mode if the following conditions hold:
(POC-POC0)*(POC-POC1)<0,
where POC is the picture order count of the current picture to be encoded, and POC0 and POC1 are the picture order counts of the references to the current picture.

The signaled merge candidate pair is used as input to the DMVR process and is called the initial motion vector (MV0, MV1). The search points searched by DMVR comply with the motion vector difference mirroring condition. That is, the DMVR check point represented by the pair of candidate motion vectors (MV0', MV1') follows the following two equations.
MV0'=MV0+MV _diff
MV1'=MV1-MV _diff
Here, MV _diff represents a point in the search space in one reference picture.

After constructing the search space, a regular 8-tap DCTIF interpolation filter is used to construct a one-sided prediction. By using the MRSAD (sum of mean separation absolute differences) between the two predictions (Figure 6) and the search point, we compute the bilateral matching cost function and find the search point that results in the minimum cost. , as the improved MV pair. For MRSAD calculations, we use 16-bit precision samples (which are the output of interpolation filtering) and do not apply clipping and rounding operations before MRSAD calculations. The reason for not applying rounding and clipping is to reduce internal buffer requirements.

FIG. 6 shows an example of bilateral matching using six-point search.

In the proposed method, integer precision search points are selected using an adaptive pattern method. First, the cost corresponding to the center point (pointed to by the initial motion vector) is calculated. The other four costs (code shapes) are calculated by two predictions and are located on opposite sides of the center point. Select the last 6th point in this corner by the gradient of the cost (FIG. 7) calculated earlier.

FIG. 7 shows examples of adaptive integer and half-sample search patterns.

The output of the DMVR process is a pair of refined motion vectors corresponding to the minimum cost.

After one iteration, the refinement process ends if the minimum cost is reached at the center point of the search space, ie, if the motion vector does not change. If not, consider the best cost as further center and continue this process, while the lowest cost does not correspond to the center point and does not exceed the search range.

A half-sample precision search is applied only if the application of a half-pixel search does not exceed the search range. In this case, only four MRSAD calculations are performed corresponding to the positive shape points around the center point, which are selected as the best during the integer precision search. Finally, output the refined motion vector pair corresponding to the minimum cost point.

Some simplifications and improvements are further proposed in JVET-L0163.
Reference sampling padding

Reference sample padding is applied to enlarge the reference sample block pointed to by the first motion vector. Let the sizes of the coding blocks be "w" and "h", and assume that blocks of size w+7 and h+7 are retrieved from the reference picture buffer. The searched buffer is then expanded by two samples in each direction by repeatedly padding the samples using the closest sample. This expanded reference sample block is then used to obtain a refined motion vector (which may deviate from the initial motion vector 2 samples in each direction) before generating the final prediction.

Note that according to this variant, external memory access in DMVR is completely eliminated without any coding loss.

Bilinear interpolation instead of 8-tap DCTIF

According to the present proposal, bilinear interpolation is applied during the DMVR search process, which means that the predictions used for MRSAD computation are generated using bilinear interpolation. Once the final refined motion vector is obtained, a regular 8-tap DCTIF interpolation filter is applied to generate the final prediction.

Disabling DMVR for small blocks

For blocks 4x4, 4x8, 8x4, DMVR is disabled.

Early termination based on MV difference between merge candidates

We impose additional conditions on the DMVR to limit the MV refinement process. Thereby, DMVR is conditionally disabled if the following conditions are met:

The MV difference between the selected merge candidate and any of the previous merge candidates in the same merge list is less than a predefined threshold (i.e. less than 64 pixels, less than 256 pixels, and a CU of at least 256 pixels, respectively). , the spacing is 1/4, 1/2, and 1 pixel width).

Early termination based on SAD cost at center search coordinates

The initial motion vector of the current CU is used to calculate the sum of absolute differences (SAD) of the two predicted signals (L0, L1 prediction). If the SAD is not larger than the predefined threshold, i.e. 2 per sample ^(BDepth-9) , then DMVR is skipped, otherwise DMVR is used to refine the two motion vectors of the current block. Still applicable.

Conditions for applying DMVR

When implemented in BMS2.1, the application condition of DMVR is (POC-POC1)×(POC-POC2)<0, which is replaced by the new condition (POC-POC1)==(POC2-POC). This means that DMVR is only applied if the reference picture is in the opposite temporal direction and equidistant to the current picture.

MRSAD calculation using every other row

The MRSAD cost is calculated only for odd rows of one block and does not take into account even sample rows. This reduces the number of operations for MRSAD calculation by half.

2.6 Related methods

Patent Application No. PCT/CN2018/098691 entitled "Motion Refinement for Visual Media Coding" filed on August 4, 2018 (herein incorporated by reference) ) are proposed herein, an MV update method and a two-step inter prediction method. The derived MV between reference block 0 and reference block 1 in BIO is scaled and added to the original motion vectors of list 0 and list 1. On the other hand, motion compensation is performed using the updated MV, and the second inter prediction is generated as the final prediction. The temporal gradient is modified by removing the average difference between reference block 0 and reference block 1.

2.7 DMVR in VVC Draft 4

The usage of DMVR in JVET-M1001_v7 (VVC working draft 4, 7th edition) is defined as follows.
- dmvrFlag is set equal to 1 if all of the following conditions are true:
- sps_dmvr_enabled_flag is equal to 1 - The current block is not coded in triangle prediction mode, AMVR affine mode, subblock mode (including merge affine mode, ATMVP mode).
- merge_flag[xCb][yCb] is equal to 1 - both predFlagL0[0][0] and predFlagL1[0][0] are equal to 1 - mmvd_flag[xCb][yCb] is equal to 0 - DiffPicOrderCnt(currPic , RefPicList[0][refIdxL0]) is equal to DiffPicOrderCnt(RefPicList[1][refIdxL1], currPic).
- cbHeight is 8 or more - cbHeight*cbWidth is 64 or more

3. Examples of issues that the embodiment attempts to solve

In BIO, during the early termination stage, the difference between two reference blocks or subblocks is calculated, while also calculating the temporal gradient. Since the temporal gradient is actually the difference (or right-shifted difference) between the two reference pixels, there is no point in calculating both this difference and the temporal gradient.

In DMVR, MRSAD calculations are used to determine one block refinement motion vector.

In BIO, use SAD calculation to determine whether BIO should be enabled/disabled for one block or one sub-block using all samples of one block/one sub-block. , but this increases computational complexity.

This calculation method differs depending on the spatial gradient and the temporal gradient.

4. Examples of embodiments

Express SATD as the sum of absolute transformation differences, MRSATD as the sum of mean-separated absolute transformation differences, SSE as the sum of squared errors, and MRSSE as the sum of mean-separated squared errors.

The detailed techniques below should be considered as examples to illustrate general concepts. These techniques should not be interpreted in a narrow sense. Furthermore, these inventions can be combined in any way.

In the following description, SatShift (x, n) is defined as follows.

Shift(x,n) is defined as Shift(x,n)=(x+offset0)>>n.

In one example, offset0 and/or offset1 are set to (1<<n)>>1 or (1<<(n-1)). In another embodiment, offset0 and/or offset1 are set to 0.

In another example, offset0=offset1=((1<<n)>>1)-1 or ((1<<(n-1)))-1.

In the gradient calculation of a BDOF, the difference between two neighboring (spatially nearby or temporally neighboring) samples and/or the difference between non-adjacent samples can be calculated, and the right shift is used during the gradient calculation. may be carried out. Assume that two neighboring samples are neig0 and neig1, the right shift value is 1, and the calculated slope is grad. Note that shift1 may be different for the spatial gradient and the temporal gradient.

1. It is proposed to adjust the methods used to calculate spatial and temporal gradients.
a. In one example, the slope is calculated according to the shifted sample difference.
i. Alternatively, the slope is calculated according to the difference in the modified samples (eg, by shifting).
b. In one example, a subtraction may be performed before the right shift in the gradient calculation. For example, grad=(neig0-neig1) >> shift1.
c. In one example, in the gradient computation, the subtraction may be performed after the right shift. For example, grad=(neig0>>shift1)-(neig1>>shift1).
d. In one example, the gradient calculation may include a subtraction before the right shift and an offset before the right shift. For example, grad=(neig0-neig1+offset)>>shift1. This offset may be equal to 1<<(shift1-1) or 1<<shift1>>1.
e. In one example, a right shift may be followed by a subtraction, and an offset may be added before a right shift in the gradient calculation. For example, grad=((neig0+offset)>>shift1)-((neig1+offset)>>shift1). This offset may be equal to 1<<(shift1-1) or 1<<shift1>>1.
f. In one example, this slope may be calculated as SatShift(neig0-neig1, shift1).
i. Alternatively, this slope may be calculated as SatShift(neig0, shift1)-SatShift(neig1, shift1).

2. It is proposed to use other criteria, for example SATD or MRSATD or SSE or MRSSE or mean difference or slope values, to determine the activation/deactivation of BIO and/or DMVR in the early termination phase.
a. In one example, block-level and sub-block-level enabling/disabling decisions may select different rules, eg, a rule with SAD and a rule with SATD.
b. In one example, in a block/subblock, if the slope value (horizontal and/or vertical) or the average slope value or range of slope values satisfies a condition (e.g., is greater than a threshold or outside a predetermined range) , BIO and/or DMVR may be disabled.
c. It is proposed that the criteria used to determine enabled/disabled BIO/DMVR may be signaled from the encoder to the decoder in the VPS/SPS/PPS/slice header/tile group header.

3. To determine the improved motion vector of one block in DMVR processing, it is proposed to use other criteria, such as SATD, MRSATD, or SSE, or MRSSE to replace MRSAD.
a. In one example, an improved motion vector of one sub-block in DMVR processing, eg, SATD, MRSATD, or SSE, or MRSSE, replaces MRSAD.
b. In one example, when SATD (or MRSATD) is applied, the entire block is divided into M×N subblocks, and SATD (or MRSATD) is calculated for each subblock. The SATD (or MRSATD) of all or some of the sub-blocks are summed to obtain the SATD (or MRSATD) value of the entire block.

4. If the difference between the average values of two reference blocks of one block is greater than a threshold (T1), BIO and/or DMVR may be disabled.
a. If the average value difference between two reference subblocks of one subblock is greater than a threshold (T2), BIO may be disabled.
b. Threshold value T1 and/or T2 may be defined in advance.
c. The threshold T1 and/or T2 may depend on the block size.

5. It is noted that during the early termination stage of BIO, the reference block or/and sub-block may be modified first before calculating the difference between two reference blocks/sub-blocks (e.g. SAD/SATD/SSE etc.). Suggested.
a. In one example, an average of a reference block or/and subblock may be calculated and then subtracted by the reference block or/and subblock.
b. In one example, the method is described in application no. PCT/CN2018/096384 (with reference to (incorporated herein) and may be used to calculate the average value of the reference block and/or sub-blocks, ie the average value is used to calculate several representative positions.

6. During the early termination stage of BIO and/or DMVR, the difference between two reference blocks and/or subblocks (e.g., SAD/SATD/SSE/MRSAD/MRSATD/MRSSE, etc.) is calculated for some representative positions. It is proposed that only the following may be calculated.
a. In one example, only even row differences are calculated for a block or/and subblock.
b. In one example, only the four corner sample differences of one block/subblock are calculated for this block or/and subblock.
c. In one example, the method is described in U.S. Provisional Application No. 62/693,412 entitled "Decoder Side Motion Vector Derivation in Video Coding," filed on July 2, 2018. (herein incorporated by reference) and may be used to select representative locations.
d. In one example, the difference between two reference blocks (eg, SAD/SATD/SSE/MRSAD/MRSATD/MRSSE, etc.) may be calculated only for some representative subblocks.
e. In one example, the differences across blocks/subblocks are obtained by summing the differences (eg, SAD/SATD/SSE/MRSAD/MRSATD/MRSSE, etc.) calculated for representative locations or subblocks.

7. Temporal gradient (The temporal gradient at position (x, y) is defined as G(x, y) = P0(x, y) - P1(x, y), where P0(x, y) and P1(x , y) represents the predicted value at (x, y) from two different reference pictures) or the modified temporal gradient is used as the difference (instead of SAD) in the early termination stage of BIO. , Consequently, it is proposed that the threshold used for early termination may be adjusted.
a. In one example, the sum of the absolute values of the temporal gradients is calculated and used as the difference between two reference blocks or/and subblocks.
b. In one example, the sum of the absolute values of the temporal gradients is calculated only at some representative positions of the block and/or subblock.
c. In one example, the method is disclosed in a US patent. In order to select representative positions, Provisional Application No. 62/693 entitled “Decoder Side Motion Vector Derivation in Video Coding” filed on July 2, 2018 , 412 (herein incorporated by reference) may be used.

8. It is proposed that the temporal gradient modification process may be performed adaptively on different blocks/subblocks.
a. In one example, the temporal gradient is modified such that only if the absolute mean difference (or SAD/SATD/SSE, etc.) between the two reference blocks is greater than a threshold T, e.g. T=4. be done.
b. In one example, the temporal gradient is modified only if the average absolute difference (or SAD/SATD/SSE, etc.) between two reference blocks is less than a threshold T, eg, T=20.
c. In one example, the temporal gradient is determined only if the average absolute difference (or SAD/SATD/SSE, etc.) between two reference blocks falls within the range [T1, T2], e.g. T1=4, T2=20 It is corrected so that
d. In one example, if the average absolute difference (or SAD/SATD/SSE, etc.) between two reference blocks is greater than a threshold T, for example T (T=40), the BIO is disabled. .
e. In one example, these thresholds may be implicitly predefined.
f. In one example, these thresholds may be signaled at the SPS/PPS/picture/slice/tile level.
g. In one example, these thresholds may be different for different CUs, LCUs, slices, tiles, or pictures.
i. In one example, these thresholds may be designed based on decoded/encoded pixel values.
ii. In one example, these thresholds may be designed differently for different reference pictures.
h. In one example, the temporal gradient is modified only if the (absolute) average of the two (or any one of the two) reference blocks is greater than a threshold T, for example T=40.
i. In one example, the temporal gradient is modified such that only if the (absolute) average of the two (or any one of the two) reference blocks is less than a threshold T, e.g. T=100.
j. In one example, the temporal gradient is determined only if the (absolute) average of the two (or any one of the two) reference blocks falls within the range [T1, T2], e.g. T1=40, T2 =100.
k. In one example, the temporal gradient is such that the (absolute) average of the two (or any one of the two) reference blocks is the average absolute difference (or SAD/SATD, etc.) of T (in one example, T=4. 5) is corrected only if it is greater/less than the multiplied value.
l. In one example, the temporal gradient is such that the (absolute) average of two (or any one of the two) reference blocks is equal to the average absolute difference (or SAD/SATD, etc.) [T1, T2] (in one example, T1=4.5, T2=7).

9. Note that it is proposed that in the hybrid intra and inter prediction mode, the two inter reference blocks may be modified when calculating the spatial gradient in BIO or may be modified before performing the entire BIO procedure. be done.
a. In one example, the intra- and inter-predicted blocks in each prediction direction are weighted averaged (using the same weighting method as the hybrid inter- and inter-prediction) to derive spatial gradients in the BIO, denoted wAvgBlkL0 and wAvgBlkL1. Generate two new prediction blocks that will be used to
b. In one example, wAvgBlkL0 and wAvgBlkL1 are used to generate a predictive block for the current block, denoted predBlk. Then, wAvgBlkL0, wAvgBlkL1, and predBlk are further used as the BIO procedure, and the improved prediction block generated by BIO is used as the final prediction block.

10. It is proposed that a DMVR or/and BIO flag may be signaled at the block level to indicate whether DMVR and/or BIO is enabled for the block.
a. In one example, such flags may be signaled only in AMVP mode, and in merge mode, such flags may be inherited from spatially and/or temporally neighboring blocks.
b. In one example, whether BIO or/and DMVR is enabled may be determined jointly by a signaled flag and an on-the-fly decision (eg, a SAD-based decision during an early termination phase). A signaled flag can indicate whether the on-the-fly decision is correct.
c. Such flags are not reported for single prediction blocks.
d. Such a flag may not be signaled for bidirectional predictive blocks where the two reference pictures are both previous or subsequent pictures in display order.
e. If POC_diff(curPic, ref0) is not equal to POC_diff(ref1, curPic), such a flag may not be signaled to the bidirectional prediction block, and POC_diff() ref0 and ref1 are the reference pictures of the current picture.
f. Such flags are not signaled for intra-coding blocks. Alternatively, further such flags are not signaled for blocks coded in hybrid intra and inter prediction mode. Alternatively, such a flag is not signaled for the current picture reference block, ie the reference picture is the current picture.
g. Whether a flag is signaled may depend on the block size. For example, if the block size is less than a threshold, no such flag will be signaled. Alternatively, such a flag is not signaled if the width and/or height of the block is greater than or equal to the threshold.
h. Whether a flag is signaled may depend on the accuracy of the motion vector. For example, if the motion vector is of integer precision, no such flag will be signaled.
i. If such a flag is not signaled, it may be implicitly derived to be true or false.
j. A flag may be signaled in the slice header/tile header/PPS/SPS/VPS to indicate whether the method is enabled.
k. Such signaling methods depend on the temporal layer of the picture and may, for example, be disabled for pictures in higher temporal layers.
l. Such signaling methods depend on the QP of the picture and may be disabled, for example, for pictures with high QP.

11. Instead of checking both block height and block size, it is proposed to decide whether to enable or disable DMVR according to block height only.
a. In one example, DMVR may be enabled if the block height is greater than T1 (eg, T1=4).
b. In one example, DMVR may be enabled if the block height is greater than or equal to T1 (eg, T1=8).

12. The above method applied to DMVR/BIO is only applicable to other decoder-side motion vector derivation (DMVD) methods, such as optical flow-based prediction improvement for affine modes.
a. In one example, condition checks for determining DMVR and BIO usage may be adjusted, such as whether block heights meet the same threshold.
i. In one example, DMVR and BIO may be enabled if the block height is greater than or equal to T1 (eg, T1=8).
ii. In one example, DMVR and BIO may be enabled if the block height is greater than T1 (eg, T1=4).

5. Embodiment

5.1 Embodiment #1

In other words, "cbHeight*cbWidth is 64 or more" is deleted.

5.2 Embodiment #2

i. One case

8.5.7.4 Bidirectional optical flow prediction processing

The inputs to this process are as follows.
- two variables nCBW and nCBH, defining the width and height of the current coding block;
- two (nCBW+2)×(nCBH+2) luminance prediction sample arrays predSamplesL0 and predSamplesL1,
- prediction list usage flags predFlagL0 and predFlagL1,
- reference indexes refIdxL0 and refIdxL1,
- Bidirectional optical flow utilization flag bdofUtilizationFlag[xIdx][yIdx], where xIdx=0. ．． (nCbW>>2)-1, yIdx=0. ．． (nCbH>>2)-1

The output of this process is an (nCBW) x (nCBH) array pbSamples of luminance prediction sample values.

The variables bitDepth, shift1, shift2, shift3, shift4, offset4, and mvRefineThres are derived as follows.
- The variable bitDepth is set equal to BitDepth _Y.
- The variable shift1 is set equal to Max(2,14-bitDepth).
- The variable shift2 is set equal to Max(8, bitDepth-4).
- The variable shift3 is set equal to Max(5, bitDepth-7).
- The variables shift4 and Max(3,15-bitDepth) are set equal and the variable offset4 is set equal to 1<<(shift4-1).
- The variable mvRefineThres is set equal to Max(2,1<<(13-bitDepth)).

xIdx=0. ．． (nCbW>>2)-1 and yIdx=0. ．． If (nCbH>>2)-1, the following applies.
- Set variable xSb equal to (xIdx<<2)+1 and set ySb equal to (yIdx<<2)+1.
- If bdofUtilizationFlag[xSbIdx][yIdx] is equal to FALSE, then x=xSb-1. ．． xSb+2, y=ySb-1. ．． For ySb+2, the predicted sample values of the current subblock are:
pbSamples[x][y]=Clip3(0, (2 ^bitDepth )-1, (predSamplesL0[x+1][y+1]+offset2+predSamplesL1[x+1][y+1])>>shift2) (8-852)
- Otherwise (bdofUtilizationFlag[xSbIdx][yIdx] equals TRUE), the predicted sample value of the current sub-block is derived as follows:

- x=xSb-1. ．． xSb+4, y=ySb-1. ．． For ySb+4, the following ordered steps apply.

4. The position (h _x , v _y ) of each corresponding sample position (x, y) in the predicted sample array is derived as follows.
h _x =Clip3(1,nCbW,x) (8-853)
v _y =Clip3(1,nCbH,y) (8-854)

5. The variables gradientHL0[x][y], gradientVL0[x][y], gradientHL1[x][y], and gradientVL1[x][y] are derived as follows.
gradientHL0[x][y] = (predSamplesL0[h _x +1] [v _y ] - predSampleL0[h _x -1] [v _y ]) >> shift1 (8-855)
gradientVL0[x][y]=(predSampleL0[h _x ][v _y +1]-predSampleL0[h _x ][v _y -1])>>shift1 (8-856)
gradientHL1[x][y] = (predSamplesL1[h _x +1] [v _y ] - predSampleL1[h _x -1] [v _y ]) >> shift1 (8-857)
gradientVL1[x][y]=(predSampleL1[h _x ][v _y +1]-predSampleL1[h _x ][v _y -1])>>shift1 (8-858)

- The variables sGx2, sGy2, sGxGy, sGxdI, sGydI are derived as follows.
sGx2=Σ _i Σ _j (tempH[xSb+i][ySb+j]*tempH[xSb+i][ySb+j])with i,j=-1. ．． 4 (8-862)
sGy2=Σ _i Σ _j (tempV[xSb+i][ySb+j]*tempV[xSb+i][ySb+j])with i,j=-1. ．． 4 (8-863)
sGxGy=Σ _i Σ _j (tempH[xSb+i][ySb+j]*tempV[xSb+i][ySb+j])with i,j-1. ．． 4 (8-864)
sGxdI=Σ _i Σ _j (-tempH[xSb+i][ySb+j]*diff[xSb+i][ySb+j])with i,j=-1. ．． 4 (8-865)
sGydI=Σ _i Σ _j (-tempV[xSb+i][ySb+j]*diff[xSb+i][ySb+j])with i,j=-1. ．． 4 (8-866)

- The horizontal and vertical motion offsets of the current subblock are derived as follows:
v _x =sGx2>0? Clip3(-mvRefineThres, mvRefineThres, -(sGxdI<<3)>>Floor(Log2(sGx2))): 0 (8-867)
v _y =sGy2>0? Clip3 (-mvRefineThres, mvRefineThres, ((sGydI<<3)-((v _x *sGxGy _m )<<12+v _x *sGxGy _s )>>1)>>Floor(Log2(sGx2))): 0 (8- 868)

- x=xSb-1. ．． xSb+2, y=ySb-1. ．． For ySb+2, the predicted sample value of the current subblock is derived as follows.
bdofOffset=Round((v _x *(gradientHL1[x+1][y+1]-gradientHL0[x+1][y+1]))>>1)+Round((v _y *(gradientVL1[x+1][y+1]-gradientVL0[ x+1][ y+1]))>>1) (8-869)
[Ed. (JC): The Round() operation is defined for float inputs. The Round( ) operation seems redundant here because the input is an integer value. Recommender should confirm] pbSamples[x][y]=Clip3(0, (2 ^bitDepth )-1, (predSamplesL0[x+1][y+1]+offset4+predSamplesL1[x+1][y+1]+bdofOffset) >> shif t4) (8- 870)

ii. One case

8.5.7.4 Bidirectional optical flow prediction processing

The inputs to this process are as follows.
- two variables nCBW and nCBH, defining the width and height of the current coding block;
- two (nCBW+2)×(nCBH+2) luminance prediction sample arrays predSamplesL0 and predSamplesL1,
- prediction list usage flags predFlagL0 and predFlagL1,
- reference indexes refIdxL0 and refIdxL1,
- Bidirectional optical flow utilization flag bdofUtilizationFlag[xIdx][yIdx], where xIdx=0. ．． (nCbW>>2)-1, yIdx=0. ．． (nCbH>>2)-1

The output of this process is an (nCBW) x (nCBH) array pbSamples of luminance prediction sample values.

The variables bitDepth, shift1, shift2, shift3, shift4, offset4, and mvRefineThres are derived as follows.
- The variable bitDepth is set equal to BitDepth _Y.
- The variable shift1 is set equal to Max(2,14-bitDepth).
- The variable shift2 is set equal to Max(8, bitDepth-4).
- The variable shift3 is set equal to Max(5, bitDepth-7).
- The variables shift4 and Max(3,15-bitDepth) are set equal and the variable offset4 is set equal to 1<<(shift4-1).
- The variable mvRefineThres is set equal to Max(2,1<<(13-bitDepth)).

xIdx=0. ．． (nCbW>>2)-1 and yIdx=0. ．． If (nCbH>>2)-1, the following applies.
- Set variable xSb equal to (xIdx<<2)+1 and set ySb equal to (yIdx<<2)+1.
- If bdofUtilizationFlag[xSbIdx][yIdx] is equal to FALSE, then x=xSb-1. ．． xSb+2, y=ySb-1. ．． For ySb+2, the predicted sample value of the current subblock is derived as follows.
pbSamples[x][y]=Clip3(0, (2 ^bitDepth )-1, (predSamplesL0[x+1][y+1]+offset2+predSamplesL1[x+1][y+1])>>shift2) (8-852)
- Otherwise (bdofUtilizationFlag[xSbIdx][yIdx] equals TRUE), the predicted sample value of the current sub-block is derived as follows:

- x=xSb-1. ．． xSb+4, y=ySb-1. ．． For ySb+4, the following ordered steps apply.

7. The position (h _x , v _y ) of each corresponding sample position (x, y) in the predicted sample array is derived as follows.
h _x =Clip3(1,nCbW,x) (8-853)
v _y =Clip3(1,nCbH,y) (8-854)

9. The variables temp[x][y], tempH[x][y] and tempV[x][y] are derived as follows.
diff [x] [y] = (predSamplesL0[h _x ] [v _y ] >> shift2) - (predSamplesL1 [h _x ] [v _y ] >> shift2) (8-859)
tempH[x][y]=(gradientHL0[x][y]+gradientHL1[x][y])>>shift3 (8-860)
tempV[x][y]=(gradientVL0[x][y]+gradientVL1[x][y])>>shift3 (8-861)

- The variables sGx2, sGy2, sGxGy, sGxdI, sGydI are derived as follows.
sGx2=Σ _i Σ _j (tempH[xSb+i][ySb+j]*tempH[xSb+i][ySb+j])with i,j=-1. ．． 4 (8-862)
sGy2=Σ _i Σ _j (tempV[xSb+i][ySb+j]*tempV[xSb+i][ySb+j])with i,j=-1. ．． 4 (8-863)
sGxGy=Σ _i Σ _j (tempH[xSb+i][ySb+j]*tempV[xSb+i][ySb+j])with i,j-1. ．． 4 (8-864)
sGxdI=Σ _i Σ _j (-tempH[xSb+i][ySb+j]*diff[xSb+i][ySb+j])with i,j=-1. ．． 4 (8-865)
sGydI=Σ _i Σ _j (-tempV[xSb+i][ySb+j]*diff[xSb+i][ySb+j])with i,j=-1. ．． 4 (8-866)

- The horizontal and vertical motion offsets of the current subblock are derived as follows:
v _x =sGx2>0? Clip3(-mvRefineThres, mvRefineThres, -(sGxdI<<3)>>Floor(Log2(sGx2))): 0 (8-867)
v _y =sGy2>0? Clip3 (-mvRefineThres, mvRefineThres, ((sGydI<<3)-((v _x *sGxGy _m )<<12+v _x *sGxGy _s )>>1)>>Floor(Log2(sGx2))): 0 (8- 868)

- x=xSb-1. ．． xSb+2, y=ySb-1. ．． For ySb+2, the predicted sample value of the current subblock is derived as follows.
bdofOffset=Round((v _x *(gradientHL1[x+1][y+1]-gradientHL0[x+1][y+1]))>>1)+Round((v _y *(gradientVL1[x+1][y+1]-gradientVL0[ x+1][ y+1]))>>1) (8-869)
[Ed. (JC): The Round() operation is defined for float inputs. The Round( ) operation seems redundant here because the input is an integer value. Recommender should confirm] pbSamples[x][y]=Clip3(0, (2 ^bitDepth )-1, (predSamplesL0[x+1][y+1]+offset4+predSamplesL1[x+1][y+1]+bdofOffset) >> shif t4) (8- 870)

iii. One case

8.5.7.4 Bidirectional optical flow prediction processing

The inputs to this process are as follows.
- two variables nCBW and nCBH, defining the width and height of the current coding block;
- two (nCBW+2)×(nCBH+2) luminance prediction sample arrays predSamplesL0 and predSamplesL1,
- prediction list usage flags predFlagL0 and predFlagL1,
- reference indexes refIdxL0 and refIdxL1,
- Bidirectional optical flow utilization flag bdofUtilizationFlag[xIdx][yIdx], where xIdx=0. ．． (nCbW>>2)-1, yIdx=0. ．． (nCbH>>2)-1

The output of this process is an (nCBW) x (nCBH) array pbSamples of luminance prediction sample values.

xIdx=0. ．． (nCbW>>2)-1 and yIdx=0. ．． If (nCbH>>2)-1, the following applies.
- Set variable xSb equal to (xIdx<<2)+1 and set ySb equal to (yIdx<<2)+1.
- If bdofUtilizationFlag[xSbIdx][yIdx] is equal to FALSE, then x=xSb-1. ．． xSb+2, y=ySb-1. ．． For ySb+2, the predicted sample value of the current subblock is derived as follows.
pbSamples[x][y]=Clip3(0, (2 ^bitDepth )-1, (predSamplesL0[x+1][y+1]+offset2+predSamplesL1[x+1][y+1])>>shift2) (8-852)
- Otherwise (bdofUtilizationFlag[xSbIdx][yIdx] equals TRUE), the predicted sample value of the current sub-block is derived as follows:

- x=xSb-1. ．． xSb+4, y=ySb-1. ．． For ySb+4, the following ordered steps apply.

10. The position (h _x , v _y ) of each corresponding sample position (x, y) in the predicted sample array is derived as follows.
h _x =Clip3(1,nCbW,x) (8-853)
v _y =Clip3(1,nCbH,y) (8-854)

- The variables sGx2, sGy2, sGxGy, sGxdI, sGydI are derived as follows.
sGx2=Σ _i Σ _j (tempH[xSb+i][ySb+j]*tempH[xSb+i][ySb+j])with i,j=-1. ．． 4 (8-862)
sGy2=Σ _i Σ _j (tempV[xSb+i][ySb+j]*tempV[xSb+i][ySb+j])with i,j=-1. ．． 4 (8-863)
sGxGy=Σ _i Σ _j (tempH[xSb+i][ySb+j]*tempV[xSb+i][ySb+j])with i,j-1. ．． 4 (8-864)
sGxdI=Σ _i Σ _j (-tempH[xSb+i][ySb+j]*diff[xSb+i][ySb+j])with i,j=-1. ．． 4 (8-865)
sGydI=Σ _i Σ _j (-tempV[xSb+i][ySb+j]*diff[xSb+i][ySb+j])with i,j=-1. ．． 4 (8-866)

- The horizontal and vertical motion offsets of the current subblock are derived as follows:
v _x =sGx2>0? Clip3(-mvRefineThres, mvRefineThres, -(sGxdI<<3)>>Floor(Log2(sGx2))): 0 (8-867)
v _y =sGy2>0? Clip3 (-mvRefineThres, mvRefineThres, ((sGydI<<3)-((v _x *sGxGy _m )<<12+v _x *sGxGy _s )>>1)>>Floor(Log2(sGx2))): 0 (8- 868)

- x=xSb-1. ．． xSb+2, y=ySb-1. ．． For ySb+2, the predicted sample value of the current subblock is derived as follows.
bdofOffset=Round((v _x *(gradientHL1[x+1][y+1]-gradientHL0[x+1][y+1]))>>1)+Round((v _y *(gradientVL1[x+1][y+1]-gradientVL0[ x+1][ y+1]))>>1) (8-869)
[Ed. (JC): The Round() operation is defined for float inputs. The Round( ) operation seems redundant here because the input is an integer value. Recommender should confirm] pbSamples[x][y]=Clip3(0, (2 ^bitDepth )-1, (predSamplesL0[x+1][y+1]+offset4+predSamplesL1[x+1][y+1]+bdofOffset) >> shif t4) (8- 870)

FIG. 8 is a block diagram of a video processing device 800. Apparatus 800 may be used to implement one or more of the methods described herein. Apparatus 800 may be implemented by a smartphone, tablet, computer, Internet of Things (IoT) receiver, or the like. Apparatus 800 may include one or more processing units 802, one or more memories 804, and video processing hardware 806. One or more processing devices 802 may be configured to implement one or more methods described herein. Memory(s) 804 may be used to store data and code used to implement the methods and techniques described herein. Video processing hardware 806 may be used to implement the techniques described herein in hardware circuitry. Video processing hardware 806 may be partially or fully included within processing device 802 in the form of dedicated hardware or a graphical processing unit (GPU) or dedicated signal processing block.

FIG. 10 is a flowchart of a method 1000 for processing video. The method 1000 includes determining (1005) a feature of a first video block, the feature including a difference between reference blocks associated with the first video block, the difference being an absolute value. One of the following: Sum of Transformed Differences (SATD), Sum of Mean Separated Absolute Transformed Differences (MRSATD), Sum of Squared Errors (SSE), Sum of Mean Separated Squared Errors (MRSSE), Difference in Means, or Slope Values and determining an operating state of one or both of a bidirectional optical flow (BIO) technique or a decoder-side motion vector refinement (DMVR) technique based on the characteristics of the first video block. determining (1010) the operational state is one of enabled or disabled; and one or both of the BIO technique or the DMVR technique. further processing the first video block in accordance with the operating state of (1015).

FIG. 11 is a flowchart of a method 1100 for processing video. The method 1100 includes modifying (1105) a first reference block to generate a first modified reference block and modifying a second reference block to generate a second modified reference block. the first reference block and the second reference block are associated with a first video block; performing a difference (1110) between the reference block and the reference block, the difference being the sum of absolute transform differences (SATD), the sum of mean separated absolute transform differences (MRSATD), and the sum of squared errors (SSE). , the first modified reference block and the second modified further processing the first video block based on a difference between the first video block and a reference block (1115).

FIG. 12 is a flowchart of a method 1200 for processing video. The method 1200 includes determining (1205) a difference between a portion of a first reference block associated with a first video block and a portion of a second reference block, the difference being an absolute value. One of the following: Sum of Transformed Differences (SATD), Sum of Mean Separated Absolute Transformed Differences (MRSATD), Sum of Squared Errors (SSE), Sum of Mean Separated Squared Errors (MRSSE), Difference in Means, or Slope Values and further processing the first video block based on the difference (1210).

FIG. 13 is a flowchart of a method 1300 for processing video. The method 1300 includes determining (1305) a temporal gradient or a modified temporal gradient using a reference picture associated with a first video block, the method comprising: determining a gradient indicative of a difference between the reference pictures; and further processing the first video block using a bidirectional optical flow (BIO) coding tool according to the difference (1310). and, including.

FIG. 14 is a flowchart of a method 1400 for processing video. The method 1400 includes determining 1405 a temporal gradient using a reference picture associated with a first video block and modifying the temporal gradient to generate a modified temporal gradient. further processing the first video block using the modified temporal gradient (1415).

FIG. 15 is a flowchart of a method 1500 of processing video. Method 1500 includes modifying (1505) one or both of a first cross-reference block and a second cross-reference block associated with a first video block; determining (1510) a spatial gradient according to a bidirectional optical flow coding tool (BIO) using one or both of the modified second cross-reference blocks; further processing the first video block (1515).

FIG. 16 is a flowchart of a method 1600 for processing video. Method 1600 determines whether a flag signaled at the block level should enable one or both of decoder-side motion vector refinement (DMVR) or bidirectional optical flow (BIO) for a first video block. (1605) and further processing the first video block (1610), the method comprising applying one or both of DMVR and BIO in accordance with the flag. and performing the processing including.

Referring to methods 1000, 1100, 1200, 1300, 1400, 1500, and 1600, several examples of determining the use of bidirectional optical flow (BIO) or decoder-side motion vector refinement (DMVR) are provided herein. Described in Chapter 4. For example, as described in Section 4, differences between reference blocks may be determined and used to enable or disable BIO or DMVR.

Referring to methods 1000, 1100, 1200, 1300, 1400, 1500 and 1600, video blocks are encoded in a video bitstream that may achieve bit efficiency by using bitstream generation rules for motion information prediction. Good too.

The method may include operating states of the BIO technique or the DMVR technique being different at a block level and a sub-block level.

The method includes determining that one or more of the slope value, the average of the slope values, or the range of slope values is included within a threshold range, and determining the operating condition includes: , based on a determination that the gradient value, the average of the gradient values, or the range of gradient values is included within a threshold range.

The method includes determining an operating state signaled from an encoder to a decoder in a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a tile group header, or a slice header. The information may further include actions taken based on the information.

The method may include determining an improved motion vector of the first video block based on SATD, MRSATD, SSE or MRSSE, and performing further processing based on the improved motion vector.

The method may include determining the refined motion vector based on SATD or MRSATD, the method comprising determining SATD or MRSATD for each sub-block of the first video block. and generating the SATD or MRSATD for the first video block based on the sum of the SATD or MRSATD for each sub-block, the further processing of the first video block comprising: This is done based on SATD or MRSATD.

The method may include determining that a difference between the average values of two reference blocks of the first video block is greater than a threshold, and based on the difference between the average values of the two reference blocks. , one or both of the BIO or DMVR is in a disabled operating state.

The method may include determining that the difference between the average values of two reference sub-blocks among the sub-blocks of the first video block is greater than a threshold, Based on the difference in average values, one or both of the BIO or DMVR is in a disabled operating state.

The method may include predefining the threshold.

The method may include determining a size of the first video block, and the threshold is based on the size of the first video block.

The method may include modifying the first reference block and the second reference block including subtracting an average of the first reference block from the first reference block.

The method may include the portions of the first reference block and the second reference block including an even number of rows.

The method may include the portions of the first reference block and the second reference block including corner samples.

The method may include the portions of the first reference block and the second reference block including representative sub-blocks.

The method may include summing differences between the representative sub-blocks to generate a difference of the first reference block or the second reference block.

The method may include that the difference relates to a sum of absolute values of the temporal gradients.

The method may include modifying the temporal gradient based on an average absolute difference between the reference blocks being greater than a threshold.

The method may include the threshold being four.

The method may include modifying the temporal gradient based on a mean absolute difference between the reference blocks being less than a threshold.

The method may include the threshold being twenty.

The method may include modifying the temporal slope based on an average absolute difference between reference blocks that is within a threshold range.

The method may include BIO being in a disabled operating state based on the average absolute difference being greater than a threshold.

The method may include the threshold value or the threshold range being indicated at a VPS, SPS, PPS, picture, slice, or tile level.

The method may include varying the threshold value or the threshold range for different coding units (CUs), maximum coding units (LCUs), slices, tiles, or pictures.

The method may include the threshold or threshold range being based on decoded or encoded pixel values.

The method may include the threshold or the threshold range being based on a reference picture.

The method may include determining the spatial gradient comprising determining a weighted average of intra-predicted blocks and inter-predicted blocks in each prediction direction.

The method includes: providing the flag in an advanced motion vector prediction (AMVP) mode; and inheriting the flag from one or both of a spatially neighboring block or a temporally neighboring block in a merge mode. and can include.

The method may include the flag not being signaled for a single prediction block.

The method may include not signaling the flag for bidirectional predictive blocks using a reference picture that is a previous picture or a subsequent picture in display order.

The method may include not signaling the flag for bi-directionally predicted blocks.

The method may include a flag not being signaled for intra-coded blocks.

The method may include not signaling the flag for blocks coded in a hybrid intra and inter prediction mode.

The method may include signaling the flag based on a dimension of the first video block.

The method may include flags being signaled at VPS, SPS, or PPS.

The method may include the flag being based on a temporal layer of a picture associated with the first video block.

The method may include the flag being based on a quantization parameter (QP) of a picture associated with the first video block.

FIG. 17 is a block diagram illustrating an example video processing system 1700 in which various techniques disclosed herein may be implemented. Various implementations may include some or all of the modules of system 1700. System 1700 may include an input unit 1702 for receiving video content. The video content may be received in raw or uncompressed format, eg, 8 or 10 bit multi-component pixel values, or in compressed or encoded format. Input unit 1702 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, passive optical networks (PON), and wireless interfaces such as Wi-Fi or cellular interfaces.

System 1700 may include a coding module 1704 that can implement various coding or encoding methods described herein. Coding module 1704 may reduce the average bit rate of the video from input unit 1702 to the output of coding module 1704 to generate a coded representation of the video. Therefore, this coding technique is sometimes referred to as a video compression or video transcoding technique. The output of coding module 1704 may be stored or transmitted via a connected communication, as represented by module 1706. The bitstream (or coding) representation of the stored or communicated video received at input unit 1702 is used by module 1708 to generate pixel values or displayable video that is sent to display interface unit 1710. You may. The process of producing user-viewable video from a bitstream representation is sometimes referred to as video restoration. Furthermore, while we refer to certain video processing operations as "coding" operations or tools, it is understood that the coding tools or operations are used at the encoder and that the corresponding decoding tools or operations that reverse the coding result are performed by the decoder. It will be.

Examples of peripheral bus interface units or display interface units may include a Universal Serial Bus (USB) or a High Definition Multimedia Interface (HDMI) or a Display Port or the like. Examples of storage interfaces include Serial Advanced Technology Attachment (SATA), PCI, IDE interfaces, and the like. The techniques described herein may be implemented in a variety of electronic devices, such as mobile phones, laptops, smartphones, or other devices capable of performing digital data processing and/or video display.

As will be appreciated, the disclosed technique improves compression efficiency when the coding unit being compressed has a shape significantly different from the conventional square block or half-square rectangular block. It may also be implemented in a video encoder or decoder. For example, new coding tools that use long or tall coding units, such as 4x32 or 32x4 size units, can benefit from the disclosed techniques.

In some implementations, the video processing method may be performed as follows.

using a method of filtering to calculate spatial and temporal gradients during the conversion between a video block and a bitstream representation of the video block;
The filtering is used to perform the transformation.
Here, the transformation includes generating a bitstream representation from the pixel values of the video block, or generating pixel values from the bitstream representation.

In some embodiments, the spatial and temporal gradients are calculated using shifted sample differences.

In some embodiments, the spatial and temporal gradients are calculated using modified samples.

Further details of the method are described in section 1 in section 4.

FIG. 18 is a flowchart illustrating an example of a video processing method. The steps of the method are described in Example 2 of Chapter 4 of this application. The method includes converting between a current block of visual media data and a corresponding coding representation of the visual media data (step 1805), wherein converting the current block includes bidirectional optical flow. including determining whether use of one or both of the BIO (BIO) technique or the Decoder Side Motion Vector Refinement (DMVR) technique is enabled or disabled for the current block; Determining usage includes performing a transformation based on cost criteria associated with the current block.

FIG. 19 is a flowchart illustrating an example of a video processing method. The steps of the method are described in Example 3 of Chapter 4 of this application. The method includes converting between a current block of visual media data and a corresponding coding representation of the visual media data (step 1905), wherein the converting the current block is performed at a decoder side. determining whether use of a motion vector refinement (DMVR) technique in the current block is enabled or disabled, the DMVR technique being based on a method other than a mean removed sum of absolute differences (MRSAD) cost criterion; performing a transformation, including refining the motion information of the current block based on a cost criterion.

FIG. 20 is a flowchart illustrating an example of a video processing method. The steps of the method are described in Example 4 of Chapter 4 of this application. The method includes converting between a current block of visual media data and a corresponding coding representation of the visual media data (step 2005), wherein the converting the current block includes bidirectionally determining whether using one or both of optical flow (BIO) techniques or decoder-side motion vector refinement (DMVR) techniques is enabled or disabled for the current block; Alternatively, determining the use of the DMVR technique includes performing a transformation based on calculating that a difference in mean values of a pair of reference blocks associated with the current block exceeds a threshold.

FIG. 21 is a flowchart illustrating an example of a video processing method. The steps of the method are described in Example 6 of Chapter 4 of this application. The method includes modifying a first reference block to generate a first modified reference block and modifying a second reference block to generate a second modified reference block (step 2105). wherein both the first reference block and the second reference block are associated with a current block of visual media data. The method further includes determining a difference between the first modified reference block and the second modified reference block (step 2110), the difference being an absolute transform difference. Contains one or more of the following: sum (SATD), sum of mean separated absolute transformed differences (MRSATD), sum of squared errors (SSE), sum of mean separated squared errors (MRSSE), difference in means, or slope value. , including determining. The method includes performing a transformation between a current block of visual media data and a corresponding coding representation of visual media data (step 2115), the transformation comprising the first reference block and the first reference block. using the difference between the first modified reference block and the second modified reference block generated from respectively modifying a second reference block;
Including performing transformations.

FIG. 22 is a flowchart illustrating an example of a video processing method. The steps of the method are described in Example 7 of Chapter 4 of this application. The method includes using reference pictures associated with a current block of visual media data to determine a temporal gradient or modified temporal gradient indicative of a difference between the reference pictures (step 2205). , the temporal gradient or the modified temporal gradient representing a difference between the reference pictures. The method includes performing a transformation between a current block of visual media data and a corresponding coding representation of the visual media data (step 2210), the transformation comprising a temporal gradient or a modified temporal gradient. The method includes performing a transformation, including using bidirectional optical flow (BIO) techniques based in part on the gradient.

FIG. 23 is a flowchart illustrating an example of a video processing method. The steps of the method are described in Example 8 of Chapter 4 of this application. The method includes determining a first temporal gradient using a reference picture associated with a first video block or a sub-block thereof (step 2305). The method includes determining a second temporal gradient using a reference picture associated with a second video block or a subblock thereof (step 2310). The method includes modifying the first temporal gradient and modifying the second temporal gradient to generate a modified first temporal gradient and a modified second temporal gradient. (step 2315), wherein the modification of the first temporal gradient associated with the first video block includes: Including making corrections, which is different from modification. The method includes converting the first video block and the second video block into their corresponding coding representations (step 2320).

FIG. 24 is a flowchart illustrating an example of a video processing method. The steps of the method are described in Example 9 of Chapter 4 of this application. The method includes modifying one or both of a first cross-reference block and a second cross-reference block associated with the current block (step 2405). The method applies a bidirectional optical (BIO) flow technique based on using said one or both of a modified first cross-reference block and/or said modified second cross-reference block. In particular, it includes determining a spatial gradient associated with said current block (step 2410). The method includes performing a transformation between the current block and a corresponding coding representation (step 2415), the transformation comprising using the spatial gradient associated with the current block. Including performing transformations.

FIG. 25 is a flowchart illustrating an example of a video processing method. The steps of the method are described in Example 10 of Chapter 4 of this application. The method is characterized in that a flag signaled at the block level causes the processing unit to perform one or both of a decoder-side motion vector refinement (DMVR) technique or a bidirectional optical flow (BIO) technique for the current block. and making a determination indicating at least in part that it should be enabled (step 2505). The method comprises (in step 2510) converting between the current block and a corresponding coding representation, wherein the coding representation is based on the one or more of the DMVR technique and/or the BIO technique. and performing a conversion, including said flag indicating whether both are enabled.

FIG. 26 is a flowchart illustrating an example of a video processing method. The steps of the method are described in Example 11 of Chapter 4 of this application. The method includes determining (step 2605) that a decoder-side motion vector refinement (DMVR) technique should be enabled for a current block by a processing unit; It involves making a determination based exclusively on the height of the block. The method includes converting between the current block and a corresponding coding representation (step 2610).

FIG. 27 is a flowchart illustrating an example of a video processing method. The steps of the method are described in Example 12 of Chapter 4 of this application. The method includes (at step 2705) converting between a current block of visual media data and a corresponding coding representation of the visual media data, wherein the conversion is performed at a decoder side for the current block. using rules associated with one or both of a motion vector refinement (DMVR) technique or a bidirectional optical flow (BIO) technique, wherein the rules associated with the DMVR technique are compliant with application to the BIO technique. and determining whether said use of said one or both of said BIO technique or said DMVR technique is enabled or disabled includes performing a transformation based on applying said rule. include.

Some embodiments of the present technology are described in a section-based format.

1. A visual media processing method, the method comprising:
converting between a current block of visual media data and a corresponding coding representation of visual media data;
Transforming this current block enables or disables the use of one or both of Bidirectional Optical Flow (BIO) techniques or Decoder Side Motion Vector Refinement (DMVR) techniques for the current block. including determining whether
Determining the use of BIO or DMVR techniques is based on cost criteria associated with the current block;
Method.

2. The cost criteria are: the sum of absolute transformation differences (SATD), the sum of mean separation absolute transformation differences (MRSATD), the sum of squared errors (SSE), the sum of mean separation squared errors (MRSSE), the average 2. The method of claim 1, based on one or more of a difference in values or a slope value.

3. 3. The method of any one or more of clauses 1-2, wherein the cost metric is associated with a sub-block of the current block.

4. 4. The method of clause 3, wherein the sub-block level cost metric is different from the block level cost metric.

5. disabling application of the BIO technique and/or DMVR technique when one or more of the slope values, average slope values, or range of slope values is determined to be outside a threshold range;
The method according to any one or more of paragraphs 1 to 4, further comprising:

6. 2. The method of claim 1, wherein the cost metric associated with the current block is signaled in the coding representation.

7. 7. The method of claim 6, wherein the cost criteria is signaled in a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a tile group header, or a slice header.

8. A visual media processing method, the method comprising:
converting between a current block of visual media data and a corresponding coding representation of visual media data;
Transforming this current block to use one or both of Bidirectional Optical Flow (BIO) techniques or Decoder Side Motion Vector Refinement (DMVR) techniques is enabled or disabled for the current block. including determining whether
This DMVR technique includes refining the motion information of the current block based on a cost criterion other than the mean removed sum of absolute differences (MRSAD) cost criterion.
Method.

9. The cost criteria associated with the current block can be Sum of Absolute Transformed Differences (SATD), Sum of Mean Separated Absolute Transformed Differences (MRSATD), Sum of Squared Errors (SSE), or Sum of Mean Separated Squared Errors (MRSSE). ).) The method according to clause 8.

10. 10. A method according to any one or more of clauses 8-9, wherein the cost metric is associated with a sub-block of the current block.

11. dividing the current block into a plurality of sub-blocks of size M×N, the cost criterion being based on motion information associated with each of the plurality of sub-blocks;
generating a cost corresponding to each of the plurality of sub-blocks;
11. The method of clause 10, further comprising:

12. 12. The method of clause 11, further comprising summing at least one subset of costs corresponding to each of the plurality of sub-blocks to generate a resulting cost associated with the current block.

13. A visual media processing method, the method comprising:
converting between a current block of visual media data and a corresponding coding representation of visual media data;
Transforming this current block enables or disables the use of one or both of Bidirectional Optical Flow (BIO) techniques or Decoder Side Motion Vector Refinement (DMVR) techniques for the current block. including determining whether
Determining the use of the BIO technique or the DMVR technique is based on calculating that the difference in mean values of a pair of reference blocks associated with the current block exceeds a threshold;
Method.

14. The threshold value is a first threshold value, and further,
disabling application of the BIO technique and/or the DMVR technique if a difference in mean values of a pair of reference subblocks associated with one subblock of the current block exceeds a second threshold; 14. The method of claim 13, comprising:

15. 15. The method according to clause 14, wherein the first threshold and/or the second threshold are predefined numbers.

16. 15. The method of item 14, wherein the first threshold and/or the second threshold are based on dimensions of the current block.

17. A visual media processing method, the method comprising:
modifying a first reference block to generate a first modified reference block; modifying a second reference block to generate a second modified reference block; modifying, wherein both the reference block and the second reference block are associated with a current block of visual media data;
determining a difference between the first modified reference block and the second modified reference block, the difference being a sum of absolute transform differences (SATD), a mean separated absolute transform; determining a difference, including one or more of a sum of differences (MRSATD), a sum of squared errors (SSE), a sum of mean separated squared errors (MRSSE), a difference in means, or a slope value;
performing a transformation between a current block of visual media data and a corresponding coding representation of visual media data, the transformation modifying the first reference block and the second reference block, respectively; performing a transformation including using the difference between the first modified reference block and the second modified reference block generated from
method including.

18. The modifying the first reference block and the base second reference block comprises:
Calculating a first arithmetic mean based on sample values included in the first reference block and a second arithmetic mean based on sample values included in the second reference block;
Clause 17, comprising subtracting the first arithmetic mean from samples included in the first reference block and subtracting the second arithmetic mean from samples included in the second reference block. The method described in.

19. 19. The method of clause 18, wherein the first arithmetic mean and the second arithmetic mean are based on a subset of samples included in the first reference block and the second reference block, respectively.

20. 20. The method according to any one or more of clauses 17-19, wherein the first reference block and/or the second reference block are sub-blocks associated with the current block.

21. A visual media processing method, the method comprising:
using a reference picture associated with a current block of visual media data to determine a temporal gradient or a modified temporal gradient indicative of a difference between the reference pictures, the temporal gradient or the modification determining that the determined temporal gradient represents a difference between the reference pictures;
performing a transformation between a current block of visual media data and a corresponding coded representation of said visual media data, said transformation based in part on a temporal gradient or a modified temporal gradient; performing a transformation, including using a directional optical flow (BIO) technique;
Method.

22. 22. The method of clause 21, further comprising terminating the BIO technique early in response to determining that the temporal slope or modified temporal slope is less than or equal to a threshold.

23. 23. The method of clause 22, further comprising adjusting the threshold based on the number of samples used to calculate the sum of absolute values of the temporal slope or modified slope.

24. 24. The method according to any one or more of clauses 21 to 23, wherein the difference relates to a sum of absolute values of the temporal gradients.

25. any one or more of clauses 21 to 24, wherein the difference between the reference pictures corresponds to a difference between a first part of a first reference picture and a second part of a second reference picture; The method described in.

26. 26. A method according to any one or more of clauses 21-25, wherein the reference picture is associated with a sub-block of the current block.

27. A visual media processing method, the method comprising:
determining a first temporal gradient using a reference picture associated with the first video block or a sub-block thereof;
determining a second temporal gradient using a reference picture associated with the second video block or a sub-block thereof;
modifying the first temporal gradient and modifying the second temporal gradient to generate a modified first temporal gradient and a modified second temporal gradient; , the modification of the first temporal gradient associated with the first video block performs a modification that is different from the modification of the second temporal gradient associated with the second video block. And,
converting the first video block and the second video block into corresponding coding representations.

28. The modification of the first temporal gradient and/or the modification of the second temporal gradient is an average between the reference pictures associated with the first video block and/or the second video block. 28. The method of clause 27, conditionally based on the absolute difference being greater than a threshold.

29. 29. The method of clause 28, wherein the threshold is four.

30. The modification of the first temporal gradient and/or the modification of the second temporal gradient is an average between the reference pictures associated with the first video block and/or the second video block. 28. The method of clause 27, conditionally based on the absolute difference being less than a threshold.

31. 31. The method of clause 30, wherein the threshold is 20.

32. The modification of the first temporal gradient and/or the modification of the second temporal gradient is an average between the reference pictures associated with the first video block and/or the second video block. 28. The method of clause 27, conditionally based on the absolute difference being within a threshold range.

33. the first video block and/or the second video block based on the average absolute difference between the reference pictures associated with the first video block and/or the second video block being greater than a threshold; 33. The method of any of clauses 27-32, further comprising disabling the use of bidirectional optical flow (BIO) techniques for video blocks of.

34. 34. The threshold value or the threshold range is indicated at the VPS, SPS, PPS, picture, slice, or tile level associated with the first video block and/or the second video block. The method described in any one or more of the items.

35. 34. The method according to any one or more of clauses 27 to 33, wherein the threshold value or the threshold value range is an implicitly predefined parameter.

36. The threshold value or the threshold range may be different for different coding units (CUs), maximum coding units (LCUs), slices, tiles, or pictures associated with the first video block and/or the second video block. The method according to any one or more of items 27 to 33.

37. The threshold value or the threshold range is based on decoded or encoded pixel values associated with the first video block and/or the second video block, and is based on any one or more of items 27 to 33. The method described in.

38. 34. A method according to any one or more of clauses 27 to 33, wherein the threshold or threshold range of a first set of reference pictures is different from the threshold or threshold range of a second set of reference pictures.

39. The modification of the first temporal gradient and/or the modification of the second temporal gradient may be based on the average absolute value of the reference pictures associated with the first video block and/or the second video block. 28. The method of clause 27, conditionally based on the difference being greater than a threshold.

40. 40. The method of item 39, wherein the threshold is 40.

41. The modification of the first temporal gradient and/or the modification of the second temporal gradient may be based on the average absolute value of the reference pictures associated with the first video block and/or the second video block. 28. The method of clause 27, conditionally based on the difference being less than a threshold.

42. 42. The method of clause 41, wherein the threshold is 100.

43. The modification of the first temporal gradient and/or the modification of the second temporal gradient may be based on the average absolute value of the reference pictures associated with the first video block and/or the second video block. 28. The method of clause 27, conditionally based on the difference being within a threshold range.

44. The modification of the first temporal gradient and/or the modification of the second temporal gradient may be based on the average absolute value of the reference pictures associated with the first video block and/or the second video block. 28. Conditionally based on the difference being greater than the average absolute difference of reference pictures associated with the first video block and/or the second video block multiplied by a multiplication factor. Method.

45. The modification of the first temporal gradient and/or the modification of the second temporal gradient may be based on the average absolute value of the reference pictures associated with the first video block and/or the second video block. The method of clause 27, conditionally based on the difference being less than the average absolute difference of reference pictures associated with the first video block and/or the second video block multiplied by a multiplication factor. .

46. The method according to any one or more of items 44 to 45, wherein the multiplication factor is 4.5.

47. A visual media processing method, the method comprising:
modifying one or both of a first cross-reference block and a second cross-reference block associated with the current block;
in accordance with applying a Bidirectional Optical (BIO) flow technique based on using said one or both of said modified first cross-reference block and/or said modified second cross-reference block; determining a spatial gradient associated with the current block;
performing a transformation between the current block and a corresponding coding representation, the transformation comprising using the spatial gradient associated with the current block; ,Method.

48. Determining the spatial gradient comprises:
generating two predictive blocks based on a weighted average of an intra-predicted block and an inter-predicted block associated with the current block;
48. The method of clause 47, comprising: using two predictive blocks to determine the spatial gradient associated with a current block.

49. generating an improved prediction block from the two prediction blocks using the BIO technique;
49. The method of clause 48, further comprising: using an improved prediction block to predict subblocks and/or samples of the current block.

50. A visual media processing method, the method comprising:
A flag signaled at the block level by the processing unit to enable one or both of decoder-side motion vector refinement (DMVR) techniques or bidirectional optical flow (BIO) techniques for the current block. making a determination that at least partially indicates that
performing a transformation between said current block and a corresponding coding representation, said coding representation indicating whether said one or both of said DMVR technique and/or said BIO technique is enabled; and performing a transformation that includes the flag.
Method.

51. 51. The method of clause 50, wherein the flag is signaled in a coding expression in response to detecting that advanced motion vector prediction (AMVP) techniques are enabled for the current block.

52. The flag determines whether one of the spatially neighboring blocks or temporally neighboring blocks associated with the current block or 51. The method of clause 50, which is derived from both.

53. 53. The method of clause 52, wherein the flag is inherited from a selected merge candidate if the selected merge candidate is a spatial merge candidate.

54. 53. The method of clause 52, wherein the flag is inherited from the selected fusion candidate if the selected fusion candidate is a temporal fusion candidate.

55. determining whether the one or both of DMVR techniques and/or BIO techniques are enabled using a cost metric associated with the current block and using the flag signaled in the coding expression; 51. The method of clause 50, wherein the determination is accurate.

56. The cost metric associated with the current block is the sum of absolute differences (SAD) between the two reference blocks of the current block, and if the cost metric is greater than a threshold, the DMVR technique and/or the BIO technique 56. The method of clause 55, wherein a determination that one or both are enabled is applied.

57. 51. The method of clause 50, further comprising skipping signaling a flag in a coding representation if the current block is determined to be a single prediction block.

58. skipping signaling a flag in the coding representation when the current block is determined to be a bidirectional predictive block associated with a pair of reference pictures both preceding or following in display order; 51. The method of paragraph 50, comprising:

59. When it is determined that the current block is a bi-directional predictive block associated with a pair of reference pictures having different picture order count (POC) distances from the current picture associated with the current block; 51. The method of clause 50, further comprising skipping signaling of the flag in the coding representation.

60. 51. The method of clause 50, further comprising skipping signaling a flag in the coding representation if the current block is determined to be an intra-coded block.

61. 51. The method of clause 50, further comprising skipping signaling the flag in the coding representation if the current block is determined to be a hybrid intra- and inter-predicted block.

62. 51. The method of claim 50, further comprising skipping signaling of the flag in the coding representation when the current block is determined to be associated with at least one block of the same picture as a reference block. Method.

63. 51. The method of clause 50, further comprising skipping signaling the flag in the coding expression if the current block size is determined to be less than a threshold.

64. 51. The method of clause 50, further comprising skipping signaling the flag in the coding representation if the current block size is determined to be greater than or equal to a threshold.

65. 51. The method of clause 50, further comprising skipping signaling the flag in the coding representation if the precision of motion information associated with the current block is determined to be integer precision.

66. 51. The method of clause 50, further comprising skipping signaling of the flag in the coding representation when a temporal layer associated with a picture containing the current block is determined to exceed a threshold. .

67. 51. The method of clause 50, further comprising skipping signaling the flag in the coding representation when a quantization parameter associated with the current block is determined to exceed a threshold.

68. Any one of paragraphs 50-67, further comprising deriving a value of the flag as true or false in response to determining that signaling of the flag in the coding representation is skipped. The method described above.

69. 68. The method of any one or more of paragraphs 50-67, further comprising enabling one or both of the DMVR technique or the BIO technique if a flag is determined to be true.

70. 68. The method of any one or more of paragraphs 50-67, further comprising disabling one or both of the DMVR technique or the BIO technique if a flag is determined to be false.

71. If the flag is determined to be true, determining that it is correct to enable or disable one or both of the DMVR technique or the BIO technique based on at least one cost criterion; The method according to any one or more of paragraphs 50 to 67, comprising:

72. determining that it is incorrect to enable or disable one or both of the DMVR technique or the BIO technique based on at least one cost criterion if the flag is determined to be false; The method according to any one or more of paragraphs 50 to 67, further comprising.

73. Any one or more of paragraphs 50 to 67, wherein the flag is signaled in a slice header, tile header, video parameter set (VPS), sequence parameter set (SPS), or picture parameter set (PPS). The method described in.

74. The first flag is signaled to indicate whether the DMVR technique is disabled, and the second flag is signaled to indicate whether the BIO technique is disabled. The method according to paragraph 50.

75. of paragraphs 64-74, further comprising disabling the DMVR technique for a slice, tile, video, sequence or picture if the flag for the DMVR technique is determined to be true. The method described in any one or more of the items.

76. Clauses 64-74 further comprising: if the flag for the DMVR technique is determined to be false, the DMVR technique is enabled for a slice, tile, video, sequence, or picture. The method described in any one or more of the following paragraphs.

77. any one of paragraphs 64-74, further comprising disabling the BIO technique for a slice, tile, video, sequence, or picture if the BIO technique flag is determined to be true. The method described above.

78. Clauses 64-74, further comprising: enabling the BIO technique for a slice, tile, video, sequence, or picture if a flag for the BIO technique is determined to be false. The method described in any one or more of the following.

79. A visual media processing method, the method comprising:
determining, by a processing unit, that a decoder-side motion vector refinement (DMVR) technique should be enabled for a current block, said determination being based exclusively on a height of said current block; , making a determination;
converting between the current block and a corresponding coding representation.

80. 80. The method of clause 79, further comprising, in response to determining that a DMVR technique is enabled, verifying that the current block height is greater than or exceeds a threshold parameter. .

81. 81. The method of clause 80, wherein the threshold parameter is equal to four.

82. 81. The method of clause 80, wherein the threshold parameter is equal to eight.

83. A visual media processing method, the method comprising:
performing a transformation between a current block of visual media data and a corresponding coded representation of visual media data, the transformation comprising performing a decoder-side motion vector refinement (DMVR) technique or a bidirectional using rules associated with one or both of optical flow (BIO) techniques, wherein the rules associated with the DMVR technique are compliant with application to the BIO technique;
The method, wherein determining whether the use of the one or both of the BIO technique or the DMVR technique in a current block is enabled or disabled is based on applying the rule.

84. 84. The method of claim 83, wherein the rules for determining whether the DMVR technique is enabled are the same as the rules for determining whether the BIO technique is enabled.

85. 85. The method of clause 84, wherein the rule for determining whether the BIO technique and/or DMVR technique is enabled provides for verifying that the current block height is greater than or equal to a threshold.

86. 84. The rule for determining whether the BIO technique and/or DMVR technique is enabled provides for verifying that both the width and height of the current block are greater than or equal to a threshold. the method of.

87. 87. The method of any one or more of clauses 85 or 86, wherein the threshold is 4 or 8.

88. 85. The method of clause 84, wherein the rule for determining whether the BIO technique and/or DMVR technique is enabled provides for verifying that the current block size is greater than or equal to a threshold.

89. 87. The method of clause 86, wherein the threshold is 64 or 128.

90. The rules for determining whether the BIO technique and/or the DMVR technique are valid specify verifying that the current block is not coded with bidirectional prediction in CU level weight (BCW) mode. , wherein unequal weights are used for the two reference blocks of the two reference lists.

91. A rule for determining whether the BIO technique and/or the DMVR technique is enabled is that the current block has the same picture order count (POC) from the current picture associated with the current block. 85. The method of clause 84, providing for verifying that the bi-directionally predicted block is associated with a pair of reference pictures having a distance.

92. 92. The method of clause 91, wherein the pair of reference pictures includes, in display order, a picture before and a picture after a current picture associated with a current block.

93. A video decoding device comprising a processing device configured to implement the method according to one or more of paragraphs 1 to 92.

94. A video encoding device comprising a processing device configured to implement the method according to one or more of paragraphs 1 to 92.

95. 93. A computer program product having computer code stored thereon, wherein when said code is executed by a processing device, said processing device implements a method according to any of paragraphs 1 to 92. product.

96. A method, apparatus or system as described herein.

Implementations of the disclosed and other solutions, examples, embodiments, modules, and functional operations described herein, including the structures disclosed herein and structural equivalents thereof, include: It may be implemented in digital electronic circuitry, or computer software, firmware, or hardware, or a combination of one or more thereof. The disclosed and other embodiments may be implemented as one or more computer program products, i.e., encoded on a computer-readable medium for implementation by or for controlling the operation of a data processing apparatus. may be implemented as one or more modules of computer program instructions. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter that provides a machine-readable propagated signal, or a combination of one or more of these. The term "data processing apparatus" includes all apparatus, devices, and machines for processing data, including, for example, a programmable processing unit, a computer, or multiple processing units or computers. In addition to hardware, this device includes code that creates an execution environment for the computer program, such as processing device firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these. be able to. A propagated signal is an artificially generated signal, such as a mechanically generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a suitable receiving device.

A computer program (also called a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can also be written as a standalone program. , or as a module, component, subroutine, or other unit suitable for use in a computing environment. Computer programs do not necessarily correspond to files in a file system. A program may be recorded in part of a file that holds other programs or data (for example, one or more scripts stored in a markup language document), or it may be in a single file dedicated to that program. or may be stored in multiple adjustment files (eg, files that store one or more modules, subprograms, or portions of code). It is also possible to deploy a computer program to run on one computer located at one site or on multiple computers distributed at multiple sites and interconnected by a communications network.

The processing and logic flows described herein are performed by one or more programmable processing devices that execute one or more computer programs to perform functions by operating on input data and producing output. be able to. The processing and logic flow can also be performed by special purpose logic circuits, for example FPGAs (Field Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits), and the device can also be implemented as special purpose logic circuits. I can do it.

Processing devices suitable for the execution of a computer program include, for example, both general and special purpose microprocessing devices, as well as any one or more processing devices of any type of digital computer. Generally, a processing device receives instructions and data from read-only memory and/or random access memory. The essential elements of a computer are a processing unit for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer may include one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical disks, or may receive data from these mass storage devices. , or may be operably coupled to transfer data thereto. However, a computer does not need to have such a device. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, such as EPROM, EEPROM, flash storage, magnetic disks such as internal hard disks or removable It includes semiconductor storage devices such as disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processing unit and memory may be supplemented by or incorporated into special purpose logic circuitry.

Although this patent specification contains many details, these should not be construed as limitations on the scope of any subject matter or claims, but rather as specific to particular embodiments of particular technology. It should be interpreted as a description of possible characteristics. Certain features that are described in this patent document in the context of separate embodiments may also be implemented in combination in a single example. Conversely, various features that are described in the context of a single example may be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above and initially claimed as operating in a particular combination, one or more features from the claimed combination may, in some cases, be different from the combination. A combination that can be abstracted and claimed may be directed to subcombinations or variations of subcombinations.

Similarly, although acts are shown in a particular order in the drawings, this does not mean that such acts may be performed in the particular order shown or in sequential order to achieve a desired result; or should not be understood as requiring that all illustrated operations be performed. Also, the separation of various system components in the examples described in this patent specification is not to be understood as requiring such separation in all embodiments.

Only some implementations and examples have been described; other embodiments, extensions and variations are possible based on the content described and illustrated in this patent document.

Cross-reference of related applications
This application is based on Japanese patent application No. 2021-557132 filed on September 24, 2021, and this Japanese patent application is based on international patent application No. PCT/CN2020/082937 filed on April 2, 2020. This international patent application has the priority rights and benefits of international patent application No. PCT/CN2019/081155 filed on April 2, 2019 and international patent application No. PCT/CN2019/085796 filed on May 7, 2019. claim . The entire disclosure of the above application is incorporated by reference as part of the disclosure herein.

Claims (14)

A visual media processing method, the method comprising:
converting between a current block of visual media data and a corresponding coding representation of the visual media data;
The transformation includes the use of rules associated with one or both of a decoder-side motion vector refinement (DMVR) technique or a bidirectional optical flow (BIO) technique for the current block;
the rules associated with the DMVR technique are compliant with application to the BIO technique;
Determining whether the use of the one or both of the BIO technique or the DMVR technique is enabled or disabled for the current block is based on applying the rule. ,
Method. The rules for determining whether the DMVR technique is enabled are the same as the rules for determining whether the BIO technique is enabled;
The method according to claim 1. The rule for determining whether the BIO technique and/or the DMVR technique is enabled provides for verifying that the current block height is greater than or equal to a threshold;
The method according to claim 2. The rule for determining whether the BIO technique and/or the DMVR technique is enabled provides for verifying that both the width and height of the current block are greater than or equal to a threshold;
The method according to claim 2. the threshold value is 4 or 8;
A method according to any one or more of claims 3 or 4. The rule for determining whether the BIO technique and/or the DMVR technique is enabled provides for verifying that the current block size is greater than or equal to a threshold;
The method according to claim 2. the threshold value is 64 or 128;
The method according to claim 4. The rule for determining whether the BIO technique and/or the DMVR technique is enabled verifies that the current block is not coded with bidirectional prediction with CU level weight (BCW) mode. It specifies that unequal weights are used for the two reference blocks from the two reference lists,
The method according to claim 2. The rule for determining whether the BIO technique and/or the DMVR technique is enabled is that the current block has the same picture order count (POC) from the current picture associated with the current block. providing for verifying that the block is a bi-directionally predicted block associated with a pair of reference pictures having a distance;
The method according to claim 2. The pair of reference pictures includes, in display order, a picture before and a picture after the current picture associated with the current block.
The method according to claim 9. comprising a processing device configured to implement the method according to any one of claims 1 to 10;
Video decoding device. comprising a processing device configured to implement the method according to any one of claims 1 to 10;
Video encoding device. A computer program product having computer code stored therein;
The code, when executed by a processing device, causes the processing device to implement the method according to any one of claims 1 to 10.
computer program product. A method, apparatus or system as described herein.

Images