CN110719465B - Extending look-up table based motion vector prediction with temporal information - Google Patents

Abstract

A video processing method, comprising: determining a new motion candidate for video processing by using one or more motion candidates from one or more tables, wherein a table comprises one or more motion candidates and each motion candidate is associated motion information; and performing a conversion between the video block and the encoded representation of the video block based on the new motion candidate.

Description

Extending look-up table based motion vector prediction with temporal information

Cross Reference to Related Applications

The present application claims in time the priority and benefit of international patent application No. PCT/CN2018/095716, filed 7, 14/2018, in accordance with applicable patent laws and/or the provisions of the paris convention. The entire disclosure of International patent application No. PCT/CN2018/095716 is incorporated by reference herein as part of the present disclosure.

Technical Field

This document relates to video coding techniques, devices and systems.

Background

Despite advances in video compression, digital video still uses the greatest bandwidth on the internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth requirements to pre-count digital video usage will continue to grow.

Disclosure of Invention

This document discloses methods, systems, and devices for encoding and decoding digital video using Merge lists of motion vectors.

In one exemplary aspect, a video processing method is disclosed. The video processing method comprises the following steps: determining a new candidate for video processing by averaging two or more selected motion candidates; adding the new candidate to a candidate list; performing a transition between a first video block of the video and a bitstream representation of the video by using the determined new candidate in the candidate list.

In one exemplary aspect, a video processing method is disclosed. The video processing method comprises the following steps: determining a new motion candidate for video processing by using one or more motion candidates from one or more tables, wherein a table comprises one or more motion candidates and each motion candidate is associated motion information; transitions are performed between video blocks and encoded representations of the video blocks based on the new candidates.

In one exemplary aspect, a video processing method is disclosed. The video processing method comprises the following steps: determining new candidates for video processing by always using motion information from more than one spatially neighboring block of a first video block in a current picture and not using motion information from temporal blocks in a different picture than the current picture; a conversion between a first video block in a current picture of the video and a bitstream representation of the video is performed by using the determined new candidate.

In one example aspect, a video processing method is disclosed. The video processing method comprises the following steps: determining new candidates for video processing by using motion information from at least one spatially non-adjacent block of a first video block in a current picture and other candidates derived from or not from the spatially non-adjacent block of the first video block; using the determined new candidate, a conversion between a first video block of the video and a bitstream representation of the video is performed.

In one example aspect, a video processing method is disclosed. The video processing method comprises the following steps: determining new candidates for video processing by using motion information from one or more tables of a first video block in a current picture and motion information from a temporal block in a picture different from the current picture; using the determined new candidate, a conversion between a first video block in a current picture of the video and a bitstream representation of the video is performed.

In one exemplary aspect, a video processing method is disclosed. The video processing method comprises the following steps: determining new candidates for video processing by using motion information from one or more tables of the first video block and motion information from one or more spatially neighboring blocks of the first video block; using the determined new candidate, a conversion between a first video block in a current picture of the video and a bitstream representation of the video is performed.

In one example aspect, a video processing method is disclosed. The video processing method comprises the following steps: maintaining a set of tables, wherein each table includes motion candidates, and each motion candidate is associated with corresponding motion information; performing a conversion between a first video block and a bitstream representation of a video that includes the first video block; and updating the one or more tables by selectively pruning existing motion candidates in the one or more tables based on the encoding/decoding mode of the first video block.

In one exemplary aspect, a video processing method is disclosed. The video processing method comprises the following steps: maintaining a set of tables, wherein each table includes motion candidates, and each motion candidate is associated with corresponding motion information; performing a conversion between a first video block and a bitstream representation of video that includes the first video block; and updating one or more tables to include motion information from one or more temporally neighboring blocks of the first video block as new motion candidates.

In one exemplary aspect, a method of updating a motion candidate table is disclosed. The method comprises the following steps: selectively pruning existing motion candidates in the table based on an encoding/decoding mode of the video block being processed, each motion candidate being associated with corresponding motion information; and updating the table to include motion information of the video block as a new motion candidate.

In one exemplary aspect, a method of updating a motion candidate table is disclosed. The method comprises the following steps: maintaining a motion candidate table, each motion candidate associated with corresponding motion information; and updating the table to include motion information from one or more temporally neighboring blocks of the video block being processed as new motion candidates.

In one exemplary aspect, a video processing method is disclosed. The video processing method comprises the following steps: determining a new motion candidate for video processing by using one or more motion candidates from one or more tables, wherein the tables include the one or more motion candidates and each motion candidate is associated with motion information; and performing a transition between the video block and the encoded representation of the video block based on the new candidate.

In one example aspect, an apparatus in a video system is disclosed. The apparatus includes a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement the various methods described herein.

The various techniques described herein may be implemented as a computer program product stored on a non-transitory computer-readable medium. The computer program product comprises program code for performing the method described herein.

The details of one or more implementations are set forth in the accompanying drawings, and the description below. Other features will be apparent from the description and drawings, and from the claims.

Drawings

Fig. 1 is a block diagram illustrating an example of a video encoder implementation.

Fig. 2 illustrates macroblock partitioning in the h.264 video coding standard.

Fig. 3 illustrates an example of dividing a Coding Block (CB) into Prediction Blocks (PB).

Fig. 4 illustrates an example implementation of the subdivision of a Coding Tree Block (CTB) into CBs and Transform Blocks (TBs). The solid lines represent CB boundaries and the dashed lines represent TB boundaries, including example CTBs with segmentation and corresponding quadtrees.

Fig. 5 shows an example of a binary Quadtree (QTBT) structure for partitioning video data.

Fig. 6 shows an example of video block segmentation.

FIG. 7 illustrates an example of quadtree partitioning.

Fig. 8 shows an example of tree signaling.

Figure 9 shows an example of the derivation process for the Merge candidate list construction.

Fig. 10 shows example positions of the spatial Merge candidates.

Fig. 11 shows an example of a candidate pair in consideration of redundancy check of the spatial Merge candidate.

Fig. 12 shows an example of the location of a second PU partitioned by Nx2N and 2 NxN.

Fig. 13 illustrates an example motion vector scaling of the temporal region Merge candidate.

Fig. 14 shows candidate positions of the time domain Merge candidates and their collocated pictures.

Fig. 15 shows an example of combining bidirectional prediction Merge candidates.

Fig. 16 shows an example of a derivation process of a motion vector prediction candidate.

Fig. 17 shows an example of motion vector scaling of spatial motion vector candidates.

Fig. 18 illustrates an example optional temporal motion vector prediction (ATMVP) for motion prediction of a Coding Unit (CU).

Fig. 19 graphically depicts an example of identification of source blocks and source pictures.

Fig. 20 shows an example of one CU having four sub-blocks and adjacent blocks.

FIG. 21 illustrates an example of two-sided matching.

Fig. 22 illustrates an example of template matching.

Fig. 23 depicts an example of single sided Motion Estimation (ME) in Frame Rate Up Conversion (FRUC).

Fig. 24 shows an example of decoder-side motion vector refinement (DMVR) based on two-sided template matching.

Fig. 25 shows an example of spatial neighboring blocks used to derive illumination compensation IC parameters.

Fig. 26 shows an example of spatial neighboring blocks used to derive spatial Merge candidates.

Fig. 27 illustrates an example of using an adjacent inter-prediction block.

Fig. 28 shows an example of a planar motion vector prediction process.

Fig. 29 shows an example of a position next to the current Coding Unit (CU) line.

FIG. 30 is a block diagram illustrating an example of the structure of a computer system or other control device that may be used to implement portions of the disclosed technology.

FIG. 31 illustrates a block diagram of an example embodiment of a mobile device that may be used to implement portions of the disclosed technology.

Fig. 32 illustrates a flow diagram of an example method for video processing in accordance with the presently disclosed technology.

Fig. 33 shows a flow diagram of an example method for video processing in accordance with the presently disclosed technology.

FIG. 34 illustrates a flow diagram of an example method for video processing in accordance with the presently disclosed technology.

Fig. 35 illustrates a flow diagram of an example method for video processing in accordance with the presently disclosed technology.

FIG. 36 illustrates a flow diagram of an example method for video processing in accordance with the presently disclosed technology.

Fig. 37 shows a flow diagram of an example method for video processing in accordance with the presently disclosed technology.

Fig. 38 shows a flow diagram of an example method for video processing in accordance with the presently disclosed technology.

Fig. 39 illustrates a flow diagram of an example method for video processing in accordance with the presently disclosed technology.

FIG. 40 illustrates a flow diagram of an example method for updating a motion candidate table in accordance with the presently disclosed technology.

FIG. 41 illustrates a flow diagram of an example method for updating a motion candidate table in accordance with the presently disclosed technology.

Fig. 42 illustrates a flow diagram of an example method for video processing in accordance with the presently disclosed technology.

Detailed Description

In order to increase the compression ratio of video, researchers are constantly looking for new techniques to encode video.

1. Introduction to

This document relates to video coding techniques. In particular, it relates to motion information coding (e.g. Merge mode, AMVP mode) in video coding. It can be applied to the existing video coding standard HEVC, or to the standard multifunctional video coding (VVC) to be finalized. But may also be applicable to future video coding standards or video codecs.

Brief discussion of the drawings

Video coding standards have been developed primarily through the development of the well-known ITU-T and ISO/IEC standards. ITU-T developed H.261 and H.263, ISO/IEC developed MPEG-1 and MPEG-4 visuals, and both organizations jointly developed the H.262/MPEG-2 video, H.264/MPEG-4 Advanced Video Coding (AVC), and H.265/HEVC standards. Since h.262, the video coding standard was based on a hybrid video coding structure, in which temporal prediction plus transform coding was employed. An example of a typical HEVC encoder framework is shown in fig. 1.

2.1 segmentation Structure

2.1.1 Partitioning tree structure in H.264/AVC

The core of the coding layer in the previous standard is a macroblock, containing a 16 × 16 block of luma samples and, in the case of the conventional 4.

Intra-coded blocks use spatial prediction to explore spatial correlation between pixels. Two segmentations are defined: 16x16 and 4x4.

Inter-coded blocks use temporal prediction, rather than spatial prediction, by estimating motion between pictures. The motion of a 16x16 macroblock or any sub-macroblock partition thereof can be estimated separately: 16x8, 8x16, 8x8, 8x4, 4x8, 4x4 (see fig. 2). Only one Motion Vector (MV) per sub-macroblock partition is allowed.

2.1.2 Partition tree structure in HEVC

In HEVC, various local characteristics are accommodated by dividing the CTUs into CUs using a quadtree structure (denoted as coding tree). It is decided at the CU level whether to encode a picture region using inter (temporal) prediction or intra (spatial) prediction. Each CU may be further divided into one, two, or four PUs depending on the partition type of the PU. In one PU, the same prediction process is applied and the relevant information is transmitted to the decoder on a PU basis. After obtaining the residual block by applying the prediction process based on the PU partition type, the CU may be partitioned into Transform Units (TUs) according to another quadtree structure similar to a coding tree of the CU. An important feature of the HEVC structure is that it has multiple partitioning concepts, including CU, PU and TU.

In the following, various features involved in hybrid video coding using HEVC are highlighted as follows.

1) Code tree unit and Code Tree Block (CTB) structure: a similar structure in HEVC is a Coding Tree Unit (CTU) that has a size selected by the encoder and may be larger than a conventional macroblock. A CTU consists of a luma CTB and corresponding chroma CTB, as well as syntax elements. The size L × L of the luminance CTB may be chosen to be L =16, 32 or 64 samples, with larger sizes generally enabling better compression. HEVC then supports the partitioning of CTBs into smaller blocks using tree structures and quadtree signaling.

2) Coding Unit (CU) and Coding Block (CB): the quad-tree syntax of a CTU specifies the size and location of its luma and chroma CBs. The root of the quadtree is associated with the CTU. Therefore, the size of the luminance CTB is the maximum size supported by the luminance CB. The partitioning of the luma and chroma CBs of the CTU is jointly signaled. One luma CB and usually two chroma CBs together with the associated syntax form a Coding Unit (CU). A CTB may contain only one CU or may be partitioned into multiple CUs, with each CU having an associated partition, partitioned into Prediction Units (PUs) and transform unit Trees (TUs).

3) Prediction Unit (PU) and Prediction Block (PB): it is decided at the CU level whether to encode a picture region using inter prediction or intra prediction. The root of the PU partition structure is at the CU level. The luma and chroma CBs may be further divided in size and predicted from the luma and chroma Prediction Blocks (PB) depending on basic prediction type decisions. HEVC supports variable PB sizes from 64x64 to 4x4 samples. Fig. 3 shows an example of allowed PBs for an MxM CU.

4) Transform Unit (TU) and Transform Block (TB): the prediction residual is coded using a block transform. The root of the TU tree structure is at the CU level. The luma CB residual may be the same as luma TB or may be further divided into smaller luma transform blocks TB. The same applies to chroma TB. Integer basis functions similar to the Discrete Cosine Transform (DCT) are defined for 4x4, 8x8, 16x16 and 32 x 32 square TBs. For a 4x4 transform of the luma intra prediction residual, an integer transform derived from the Discrete Sine Transform (DST) form may also be specified.

Fig. 4 shows an example of subdividing a CTB into CBs (and conversion blocks (TBs)). The solid line indicates a CB boundary and the dotted line indicates a TB boundary. (a) band-split CTBs. (b) corresponding quadtrees.

2.1.2.1 Tree structured partitioning into transform blocks and units

For residual coding, the CB may be recursively partitioned into Transform Blocks (TBs). The partition is signaled by a residual quadtree. Only square CB and TB partitions are specified, where the block can be recursively divided into four quadrants as shown in fig. 4. For a given luminance CB of size M, the flag indicates whether it is divided into four blocks of size M/2. If further partitioning is possible, each quadrant is assigned a flag indicating whether it is partitioned into four quadrants, as indicated by the maximum depth of the residual quadtree as indicated in the Sequence Parameter Set (SPS). The leaf node blocks generated by the residual quadtree are transform blocks that are further processed by transform coding. The encoder indicates the maximum and minimum luminance TB sizes it will use. When the CB size is larger than the maximum TB size, partitioning is implied. When the division would result in a smaller luminance TB size than the indicated minimum, no division is implied. The chroma TB size is half the luma TB size in each dimension except when the luma TB size is 4 × 4, in which case the area covered by four 4 × 4 luma TBs uses a single 4 × 4 chroma TB. In the case of an intra-predicted CU, the decoded samples of the nearest neighbor TB (either intra CB or out CB) are used as reference data for intra prediction.

Unlike previous standards, HEVC design allows TBs to span multiple PBs for inter-predicted CUs to maximize the potential coding efficiency of TB partitioning that benefits from a quadtree structure.

2.1.2.2 parent node and child node

The CTBs are divided according to a quadtree structure, and their nodes are coding units. The plurality of nodes in the quadtree structure includes leaf nodes and non-leaf nodes. Leaf nodes have no children in the tree structure (i.e., the leaf nodes are not further partitioned). The non-leaf nodes include the root nodes of the tree structure. The root node corresponds to an initial video block (e.g., CTB) of the video data. For each respective non-root node of the plurality of nodes, the respective non-root node corresponds to a video block that is a sub-block of a video block corresponding to a parent node in a tree structure of the respective non-root node. Each respective non-leaf node of the plurality of non-leaf nodes has one or more child nodes in the tree structure.

2.1.3 quad Tree plus binary Tree Block Structure with larger CTU in Joint Exploration Model (JEM)

To explore future video coding technologies beyond HEVC, VCEG and MPEG have together established the joint video exploration team (jfet) in 2015. Since then, JFET has adopted many new approaches and applied them to a reference software named Joint Exploration Model (JEM).

2.1.3.1 QTBT block segmentation structure

Unlike HEVC, the QTBT structure eliminates the concept of multiple partition types, i.e., it eliminates the separation of CU, PU, and TU concepts and supports more flexibility of CU partition shapes. In the QTBT block structure, a CU may be square or rectangular. As shown in fig. 5, a Coding Tree Unit (CTU) is first partitioned by a quadtree structure. The leaf nodes of the quadtree are further partitioned by a binary tree structure. There are two types of partitioning in binary tree partitioning: a symmetrical horizontal division and a symmetrical vertical division. The binary tree leaf nodes are called Coding Units (CUs), and the partitioning is used for prediction and conversion processes without further partitioning. This means that CU, PU and TU have the same block size in the QTBT coding block structure. In JEM, a CU sometimes consists of Coding Blocks (CBs) of different color components, for example, in a P-slice and a B-slice of a 4.

The following parameters are defined for the QTBT segmentation scheme.

-CTU size: the root node size of the quadtree is the same as the concept in HEVC.

-miniqtsize: minimum allowed quadtree leaf node size

-MaxBTSize: maximum allowed binary tree root node size

-MaxBTDePTh: maximum allowed binary tree depth

-MiNBTSize: minimum allowed binary tree leaf node size

In one example of the QTBT segmentation structure, the CTU size is set to 128 × 128 luma samples with two corresponding 64 × 64 chroma sample blocks, the miniqtsize is set to 16 × 16, the maxbtsize is set to 64 × 64, the minibtsize (width and height) is set to 4 × 4, and the maxbtsize is set to 4. Quadtree partitioning is first applied to CTUs to generate quadtree leaf nodes. The size of the quad tree leaf nodes may have a size from 16 × 16 (i.e., miNQTSize) to 128 × 128 (i.e., CTU size). If the leaf quadtree node is 128 x 128, it is not further partitioned by the binary tree because its size exceeds the MaxBTSize (i.e., 64x 64). Otherwise, the leaf quadtree nodes may be further partitioned by the binary tree. Therefore, the leaf nodes of the quadtree are also root nodes of the binary tree, and the binary tree depth thereof is 0. When the binary tree depth reaches MaxBTDePTH (i.e., 4), no further partitioning is considered. When the width of the binary tree node is equal to MiNBTSize (i.e., 4), no further horizontal partitioning is considered. Likewise, when the height of the binary tree nodes is equal to MiNBTSize, no further vertical partitioning is considered. The leaf nodes of the binary tree are further processed by prediction and transformation processes without further partitioning. In JEM, the maximum CTU size is 256 × 256 luminance samples.

Fig. 5 (left) illustrates an example of block partitioning by using QTBT, and fig. 5 (right) illustrates the corresponding tree representation. The solid lines represent quadtree partitions, and the dashed lines represent binary tree partitions. In each partition (i.e., non-leaf) node of the binary tree, a flag is signaled to indicate which partition type (i.e., horizontal or vertical) to use, where 0 represents horizontal partition and 1 represents vertical partition. For quadtree partitioning, there is no need to specify the partition type, because quadtree partitioning always divides one block horizontally and vertically to generate 4 sub-blocks of the same size.

Furthermore, the QTBT scheme supports the ability for luminance and chrominance to have separate QTBT structures. Currently, luminance and chrominance CTBs in one CTU share the same QTBT structure for P-and B-stripes. However, for I-slices, the luminance CTB is partitioned into CUs with a QTBT structure and the chrominance CTB is partitioned into chrominance CUs with another QTBT structure. This means that a CU in an I-slice consists of either a coded block for the luma component or a coded block for the two chroma components, and a CU in a P-slice or a B-slice consists of coded blocks for all three color components.

In HEVC, to reduce memory access for motion compensation, inter prediction of small blocks is restricted so that 4 × 8 and 8 × 4 blocks do not support bi-prediction and 4 × 4 blocks do not support inter prediction. In the QTBT of JEM, these restrictions are removed.

2.1.4 Triplex Tree for Multi-function video coding (VVC)

Also, quad-trees and tree types other than binary trees are supported, as proposed in JFET-D0117. In implementation, two additional Trifurcate Tree (TT) divisions are introduced, namely horizontal and vertical center side trifurcate trees, as shown in fig. 6 (d) and (e).

Fig. 6 shows: a (a) quadtree division, (b) vertical binary tree division, (c) horizontal binary tree division, (d) vertical center side ternary tree division, and (e) horizontal center side ternary tree division.

In some implementations, there are two levels of trees: region trees (quadtrees) and prediction trees (binary or ternary). The CTUs are first partitioned with a Region Tree (RT). The RT leaves may be further partitioned with a Prediction Tree (PT). PT leaves may also be further partitioned with PT until a maximum PT depth is reached. The PT leaf is the basic coding unit. For convenience, it is still referred to as CU. The CU cannot be further divided. Both prediction and transformation are applied to the CU in the same way as JEM. The entire partition structure is called a "multi-type tree".

2.1.5 Split architecture in JFET-J0021

The tree structure called multi-tree (MTT) is a generalization of QTBT. In QTBT, as shown in fig. 5, a Coding Tree Unit (CTU) is first divided in a quadtree structure. The leaf nodes of the quadtree are then further partitioned using a binary tree structure.

The basic structure of MTT consists of two types of tree nodes: region Tree (RT) and Prediction Tree (PT), supporting nine types of partitions, as shown in fig. 7.

Fig. 7 illustrates: the tree partition includes (a) a quadtree partition, (b) a vertical binary tree partition, (c) a horizontal binary tree partition, (d) a vertical ternary tree partition, (e) a horizontal ternary tree partition, (f) a horizontal upward asymmetric binary tree partition, (g) a horizontal downward asymmetric binary tree partition, (h) a vertical left asymmetric binary tree partition, and (i) a vertical right asymmetric binary tree partition.

The zone tree may recursively divide the CTUs into square blocks up to 4x4 sized zone leaf nodes. At each node of the region tree, a prediction tree may be formed from one of three tree types: binary Tree (BT), ternary Tree (TT) and Asymmetric Binary Tree (ABT). In PT division, quadtree division is prohibited in branches of a prediction tree. As with JEM, the luma tree and chroma trees are separated in the I stripe. The signaling method for RT and PT is shown in fig. 8.

2.2 Inter prediction in HEVC/H.265

Each inter-predicted PU has motion parameters of one or two reference picture lists. The motion parameters include a motion vector and a reference picture index. The use of one of the two reference picture lists can also be signaled using inter _ pred _ idc. Motion vectors can be explicitly coded in increments with respect to the predictor, this coding mode being referred to as the Advanced Motion Vector Prediction (AMVP) mode.

When a CU is coded in skip mode, one PU is associated with the CU and there are no significant residual coefficients, no motion vector delta coded or reference picture indices. A Merge mode is specified by which the motion parameters of a current PU may be obtained from neighboring PUs (including spatial and temporal candidates). The Merge mode may be applied to any inter-predicted PU, not just the skip mode. Another option for the Merge mode is the explicit transmission of motion parameters, where the motion vectors, the reference picture indices corresponding to each reference picture list, and the use of the reference picture lists are all explicitly signaled in each PU.

When the signaling indicates that one of the two reference picture lists is to be used, the PU is generated from one sample block. This is called "one-way prediction". Unidirectional prediction is available for both P-slices and B-slices.

When the signaling indicates that two reference picture lists are to be used, a PU is generated from two blocks of samples. This is called "bi-prediction". Bi-directional prediction is available only for B slices.

Details of inter prediction modes specified in HEVC are provided below. The description will start with the Merge mode.

2.2.1 Merge mode

2.2.1.1 Derivation of candidates for Merge mode

When predicting a PU using Merge mode, the index pointing to an entry in the Merge candidate list is analyzed from the bitstream and used to retrieve motion information. The structure of this list is specified in the HEVC standard and can be summarized in the following order of steps:

step 1: initial candidate derivation

Step 1.1: spatial domain candidate derivation

Step 1.2: redundancy check of spatial domain candidates

Step 1.3: time domain candidate derivation

Step 2: additional candidate insertions

Step 2.1: creation of bi-directional prediction candidates

Step 2.2: insertion of zero motion candidates

These steps are also schematically depicted in fig. 9. For spatial Merge candidate derivation, a maximum of four Merge candidates are selected among the candidates located at five different positions. For time domain Merge candidate derivation, at most one Merge candidate is selected among the two candidates. Since the number of candidates per PU is assumed to be constant at the decoder, additional candidates are generated when the number of candidates does not reach the maximum Merge candidate (MaxNumMergeCand) signaled in the slice header. Since the number of candidates is constant, the index of the best Merge candidate is encoded using truncated unary binarization (TU). If the size of the CU is equal to 8, all PUs of the current CU share one Merge candidate list, which is the same as the Merge candidate list of the 2N × 2N prediction unit.

The operations associated with the above steps are described in detail below.

2.2.1.2 spatial domain candidate derivation

In the derivation of the spatial Merge candidates, a maximum of four Merge candidates are selected among the candidates located at the positions shown in fig. 10. The derivation order is A1, B0, A0, and B2. Position B2 is considered only if any PU of position A1, B0, A0 is unavailable (e.g., because it belongs to another slice or slice) or intra-coded. After adding the candidates for the A1 position, redundancy checks are performed on the additions of the remaining candidates, which ensures that candidates with the same motion information are excluded from the list, thereby improving coding efficiency. In order to reduce the computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. In contrast, only the pairs linked with the arrows in fig. 11 are considered, and only when the corresponding candidates for redundancy check do not have the same motion information, the candidates are added to the list. Another source of duplicate motion information is the "second PU" associated with a 2nx 2N different partition. For example, fig. 12 depicts the second PU in the N × 2N and 2N × N cases, respectively. When the current PU is divided into Nx2N, candidates for the A1 position are not considered for list construction. In some embodiments, adding this candidate may result in two prediction units with the same motion information, which is redundant for having only one PU in the coding unit. Likewise, when the current PU is divided into 2N × N, the position B1 is not considered.

2.2.1.3 time-domain candidate derivation

In this step, only one candidate is added to the list. In particular, in the derivation of this temporal target candidate, a scaled motion vector is derived based on the collocated PU that has the smallest picture order count POC difference from the current picture in a given reference picture list. The reference picture lists used to derive the collocated PUs are explicitly signaled in the slice header. The dashed line in fig. 13 shows the derivation of a scaled motion vector for the temporal domain Merge candidate, which is scaled from the motion vector of the collocated PU using POC distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td is defined as the POC difference between the reference picture of the collocated picture and the collocated picture. The reference picture index of the temporal region Merge candidate is set to zero. The actual processing of the scaling process is described in the HEVC specification. For B slices, two motion vectors (one for reference picture list 0 and the other for reference picture list 1) are obtained and combined to be a bi-predictive Merge candidate. The motion vector scaling for the temporal Merge candidate is illustrated.

In collocated PU (Y) belonging to a reference frame, in candidate C ₀ And C ₁ The location of the time domain candidate is selected as shown in fig. 14. If position C ₀ Where PU is unavailable, intra-coded or outside the current CTU, thenPosition of use C ₁ . Otherwise, position C ₀ Is used for the derivation of the time domain Merge candidates.

2.2.1.4 additional candidate insertions

In addition to space-time Merge candidates, there are two additional types of Merge candidates: the bidirectional prediction Merge candidate and the zero Merge candidate are combined. The combined bidirectional predictive Merge candidate is generated using the space-time Merge candidate. The combined bi-directional predicted Merge candidates are only for B slices. A combined bi-prediction candidate is generated by combining the first reference picture list motion parameters of the initial candidate with the second reference picture list motion parameters of the other candidate. If these two tuples provide different motion hypotheses they will form new bi-directional prediction candidates. Fig. 15 shows the case where two candidates in the original list (on the left) are used to create a combined bi-directional prediction Merge candidate added to the final list (on the right), with two candidates of MvL0 and refIdxL0 or MvL1 and refIdxL 1. There are many rules for combining that need to be considered to generate these additional Merge candidates.

Zero motion candidates are inserted to fill the remaining entries in the Merge candidate list to reach the capacity of MaxumMergeCand. These candidates have zero spatial displacement and reference picture indices that start from zero and increase each time a new zero motion candidate is added to the list. The number of reference frames that these candidates use is 1 frame and 2 frames for unidirectional prediction and bidirectional prediction, respectively. Finally, no redundancy check is performed on these candidates.

2.2.1.5 parallel-processed motion estimation regions

To speed up the encoding process, motion estimation may be performed in parallel, thereby deriving motion vectors for all prediction units within a given region simultaneously. Deriving the Merge candidate from the spatial neighborhood may interfere with parallel processing because one prediction unit cannot derive motion parameters from neighboring PUs before completing the associated motion estimation. To mitigate the trade-off between coding efficiency and processing delay, HEVC defines a Motion Estimation Region (MER). The size of the MER may be signaled in the picture parameter set using the syntax element "log2_ parallel _ merge _ level _ minus2" as described below. When defining MER, the Merge candidates falling into the same region are marked as unavailable and are therefore not considered in the list construction.

Picture parameter set original byte sequence payload (RBSP) syntax

General picture parameter set RBSP syntax

Log2_ parallel _ Merge _ level _ minis 2 plus 2 specifies the value of the variable Log2 parmrglvel, which is used in the derivation process of the Merge mode luminance motion vector specified in item 8.5.3.2.2, and the derivation process of the spatial Merge candidate specified in item 8.5.3.2.3. The value of log2_ parallel _ Merge _ level _ MiNus2 should be in the range of 0 to CtbLog2SizeY-2, including 0 and CtbLog2SizeY-2.

The variable Log2parmrgl is derived as follows:

Log2ParMrgLevel＝log2_parallel_Merge_level_MiNus2+2 (7-37)

the values annotated 3-Log2ParMrgLevel represent the built-in functionality of the parallel derivation of the Merge candidate list. For example, when Log2 parmrglvel is equal to 6, a Merge candidate list of all Prediction Units (PUs) and Coding Units (CUs) included in a 64 × 64 block may be derived in parallel.

2.2.2 Motion vector prediction in AMVP mode

Motion vector prediction exploits the spatial-temporal correlation of motion vectors with neighboring PUs, which is used for explicit transmission of motion parameters. The motion vector candidate list is first constructed by checking the availability of temporally adjacent PU locations at the top left, removing redundant candidate locations, and adding a zero vector to make the candidate list length constant. The encoder may then select the best predictor from the candidate list and send a corresponding index indicating the selected candidate. Similar to the Merge index signaling, the index of the best motion vector candidate is encoded using a truncated unary. The maximum value to be encoded in this case is 2 (e.g., fig. 2 to 8). In the following sections, the derivation process of the motion vector prediction candidates will be described in detail.

2.2.2.1 derivation of motion vector prediction candidates

Fig. 16 summarizes the derivation of motion vector prediction candidates.

In motion vector prediction, two types of motion vector candidates are considered: spatial motion vector candidates and temporal motion vector candidates. For the derivation of spatial motion vector candidates, two motion vector candidates are finally derived based on the motion vectors of each PU located at five different positions as shown in fig. 11.

For the derivation of temporal motion vector candidates, one motion vector candidate is selected from two candidates, which are derived based on two different collocated positions. After the first list of spatio-temporal candidates is made, the duplicate motion vector candidates in the list are removed. If the number of potential candidates is greater than two, the motion vector candidate with a reference picture index greater than 1 in the associated reference picture list is removed from the list. If the number of spatial-temporal motion vector candidates is less than two, additional zero motion vector candidates are added to the list.

2.2.2.2 spatial motion vector candidates

In deriving the spatial motion vector candidate, a maximum of two candidates are considered among the five potential candidates, which are from PUs at the positions depicted in fig. 11, which are the same as the position of the motion Merge. The derivation order on the left side of the current PU is defined as A ₀ 、A ₁ And scaled A ₀ Zoom, A ₁ . The derivation order above the current PU is defined as B ₀ 、B ₁ ,B ₂ Zoomed B ₀ Zoomed B ₁ Zoomed B ₂ . Thus, four cases per side can be used as motion vector candidates, two cases not requiring the use of spatial scaling and two cases using spatial scaling. Four different cases are summarized as follows:

-no spatial scaling

(1) Same reference picture list, and same reference picture index (same POC)

(2) Different reference picture lists, but the same reference picture (same POC)

-spatial scaling

(3) Same reference picture list, but different reference pictures (different POCs)

(4) Different reference picture lists, and different reference pictures (different POCs)

First the case of no spatial scaling is checked and then the spatial scaling is checked. Spatial scaling is considered when POC differs between the reference picture of the neighboring PU and the reference picture of the current PU, regardless of the reference picture list. If all PUs of the left candidates are not available or intra coded, scaling of the motion vectors is allowed to aid in parallel derivation of the left and top MV candidates. Otherwise, spatial scaling of the motion vectors is not allowed.

In the spatial scaling process, the motion vectors of neighboring PUs are scaled in a similar manner to the temporal scaling, as shown in fig. 17. The main difference is that given the reference picture list and index of the current PU as input, the actual scaling process is the same as the temporal scaling process.

2.2.2.3 temporal motion vector candidates

All derivation processes of temporal domain Merge candidates are the same as those of spatial motion vector candidates except for the derivation of reference picture index (see, e.g., fig. 6). Signaling a reference picture index to a decoder.

2.2.2.4 Signaling of AMVP information

For AMVP mode, four parts can be signaled in the bitstream, including prediction direction, reference index, MVD, and MV prediction candidate index.

Grammar table:

motion vector difference syntax

2.3 New interframe prediction method in Joint Exploration Model (JEM)

2.3.1 sub-CU-based motion vector prediction

In JEM with QTBT, each CU can have at most one set of motion parameters for each prediction direction. Two sub-CU level motion vector prediction methods are considered in the encoder by partitioning a large CU into sub-CUs and deriving motion information for all sub-CUs of the large CU. An Alternative Temporal Motion Vector Prediction (ATMVP) method allows each CU to obtain multiple sets of motion information from a number of blocks smaller than the current CU in the collocated reference picture. In the spatial-temporal motion vector prediction (STMVP) method, a motion vector of a sub-CU is recursively derived by using a temporal motion vector predictor and a spatial neighboring motion vector.

In order to maintain a more accurate motion field for sub-CU motion prediction, motion compression of the reference frame is currently disabled.

2.3.1.1 optional temporal motion vector prediction

In an Alternative Temporal Motion Vector Prediction (ATMVP) method, motion vector Temporal Motion Vector Prediction (TMVP) is modified by extracting multiple sets of motion information (including motion vectors and reference indices) from a block smaller than the current CU. As shown in fig. 18, the sub-CU is a square N × N block (default N is set to 4).

ATMVP predicts motion vectors of sub-CUs within a CU in two steps. The first step is to identify the corresponding block in the reference map by a so-called time domain vector. The reference picture is called a motion source picture. The second step is to divide the current CU into sub-CUs and obtain the motion vector and the reference index of each sub-CU from the corresponding block of each sub-CU, as shown in fig. 18.

In a first step, the reference picture and the corresponding block are determined from motion information of spatially neighboring blocks of the current CU. To avoid repeated scanning processes of neighboring blocks, the first Merge candidate in the Merge candidate list of the current CU is used. The first available motion vector and its associated reference index are set as the indices of the temporal vector and the motion source picture. In this way, in ATMVP, the corresponding block can be identified more accurately than in TMVP, where the corresponding block (sometimes referred to as a collocated block) is always located in the lower right corner or center position with respect to the current CU. In one example, if the first MThe erge candidate is from the left neighboring block (i.e., A in FIG. 19) ₁ ) The associated MV and reference picture are used to identify the source block and the source picture.

Fig. 19 shows an example of identification of source blocks and source pictures.

In a second step, the corresponding block of the sub-CU is identified by the temporal vector in the motion source picture by adding the temporal vector to the coordinates of the current CU. For each sub-CU, the motion information of its corresponding block (the smallest motion grid covering the central samples) is used to derive the motion information of the sub-CU. After identifying the motion information for the corresponding nxn block, it is converted into a motion vector and reference index for the current sub-CU, as in the TMVP method of HEVC, where motion scaling and other processing is applied. For example, the decoder checks whether a low delay condition is met (i.e., POC of all reference pictures of the current picture is smaller than POC of the current picture) and predicts a motion vector MVy (X equals 0 or 1 and Y equals 1-X) for each sub-CU, possibly using a motion vector MVx (motion vector corresponding to reference picture list X).

2.3.1.2 spatio-temporal motion vector prediction

In this method, the motion vectors of the sub-CUs are recursively derived in raster scan order. Fig. 20 illustrates this concept. Consider an 8 × 8 CU, which contains four 4 × 4 sub-CUs a, B, C, and D. The neighboring 4x4 blocks in the current frame are labeled a, b, c, and d.

The motion derivation of sub-CU a starts by identifying its two spatial neighbors. The first neighbor is the nxn block (block c) above the sub-CU a. If this block c is not available or intra coded, the other N × N blocks above sub-CU a are examined (from left to right, starting at block c). The second neighbor is a block to the left of sub-CU a (block b). If block b is not available or intra coded, the other blocks to the left of sub-CU a are checked (from top to bottom, starting at block b). The motion information obtained by each list from neighboring blocks is scaled to the first reference frame of the given list. Next, the Temporal Motion Vector Prediction (TMVP) of sub-block a is derived following the same procedure as the TMVP specified in HEVC. The motion information of the collocated block at location D is extracted and scaled accordingly. Finally, after retrieving and scaling the motion information, all available motion vectors (up to 3) are averaged for each reference list, respectively. The average motion vector is specified as the motion vector of the current sub-CU.

Fig. 20 shows an example of one CU with four sub-blocks (a-D) and their neighboring blocks (a-D).

2.3.1.3 sub-CU motion prediction mode signaling

The sub-CU mode is enabled as an additional Merge candidate mode and no additional syntax element is needed to signal this mode. Two more Merge candidates are added to the Merge candidate list of each CU to represent ATMVP mode and STMVP mode. If the sequence parameter set indicates that ATMVP and STMVP are enabled, seven large candidates are used at the maximum. The coding logic of the additional Merge candidates is the same as the one in the HM, which means that two additional RD checks need to be performed on two additional Merge candidates for each CU in a P-slice or a B-slice.

In JEM, all bin files indexed by Merge are context coded by CABAC. However, in HEVC, only the first biN file is context coded and the remaining biN files are context bypass coded.

2.3.2 adaptive motion vector difference resolution

In HEVC, when use _ integer _ mv _ flag is equal to 0 in the slice header, a Motion Vector Difference (MVD) (between a motion vector and a prediction motion vector of a PU) is signaled in units of quarter luma samples. In JEM, a locally adaptive motion vector resolution (lamfr) is introduced. In JEM, the MVD may be encoded in units of quarter luma samples, integer luma samples, or four luma samples. The MVD resolution control is at the Coding Unit (CU) level, and the MVD resolution flag conditionally signals each CU that has at least one non-zero MVD component.

For a CU with at least one non-zero MVD component, the first flag will signal to indicate whether quarter luma sample MV precision is used in the CU. When the first flag (equal to 1) indicates that the quarter luma sample MV precision is not used, the other flag signals whether integer luma sample MV precision or four luma sample MV precision is used.

A CU uses quarter-luma sample MV resolution when the first MVD resolution flag of the CU is zero or not coded for the CU (meaning all MVDs in the CU are zero). When a CU uses integer luma sample MV precision or four luma sample MV precision, the MVP in the AMVP candidate list of the CU will be rounded to the corresponding precision.

In the encoder, RD checking at the CU level is used to determine which MVD resolution is to be used for the CU. That is, RD checking at the CU level is performed three times for each MVD resolution. To speed up the encoder speed, the following encoding scheme is applied in JEM.

During the RD check of a CU with normal quarter luma sample MVD resolution, the motion information of the current CU (integer luma sample precision) is stored. The stored motion information (after rounding) is used as a starting point for further small-range motion vector refinement when performing the RD check on the same CU with integer luma sample and 4 luma sample MVD resolutions, so that the time-consuming motion estimation process is not repeated three times.

The RD check of a CU with a MVD resolution of 4 luma samples is conditionally invoked. For a CU, RD checking for the 4 luma sample MVD resolution of the CU will be skipped when RD checking cost for integer luma sample MVD resolution is much larger than RD checking cost for quarter luma sample MVD resolution.

2.3.3 Pattern matching motion vector derivation

The mode matching motion vector derivation (PMMVD) mode is a special Merge mode based on Frame Rate Up Conversion (FRUC) techniques. In this mode, the motion information of the block is not signaled but is derived at the decoder side.

For a CU, when its Merge flag is true, the FRUC flag is signaled. When the FRUC flag is false, the Merge index is signaled and the normal Merge mode is used. When the FRUC flag is true, another FRUC mode flag is signaled to indicate which mode (bilateral matching or template matching) will be used to derive motion information for the block.

At the encoder side, a decision is made whether to use FRUC-Merge mode for the CU based on the RD cost selection made for the normal-Merge candidate. I.e. by using RD cost selection to check two matching patterns (bilateral matching and template matching) for a CU. The mode that results in the lowest cost is further compared to other CU modes. If the FRUC matching pattern is the most efficient pattern, then the FRUC flag is set to true for the CU and the associated matching pattern is used.

The motion derivation process in FRUC Merge mode has two steps: CU-level motion search is performed first and then sub-CU-level motion optimization is performed. At the CU level, an initial motion vector for the entire CU is derived based on bilateral matching or template matching. First, a list of MV candidates is generated and the candidate that results in the lowest matching cost is selected as the starting point for further optimization of the CU level. Then, local search based on bilateral matching or template matching is performed near the starting point, and the MV result of the minimum matching cost is taken as the MV value of the entire CU. Then, the motion information is further refined at sub-CU level, starting from the derived CU motion vector.

For example, the following derivation process is performed for W × H CU motion information derivation. In the first stage, the MV of the entire W × H CU is derived. In the second stage, the CU is further divided into M × M sub-CUs. The value of M is calculated as (16), D is the predefined division depth, and is set to 3 by default in JEM. The MV value for each sub-CU is then derived.

As shown in fig. 21, motion information of a current CU is derived using bilateral matching by finding the closest match between two blocks in two different reference pictures along the motion trajectory of the current CU. Under the assumption of a continuous motion trajectory, the motion vectors MV0 and MV1 pointing to the two reference blocks are proportional to the temporal distance between the current picture and the two reference pictures (i.e., TD0 and TD 1. As a special case, when the current picture is temporally located between the two reference pictures and the temporal distance of the current picture to the two reference pictures is the same, bilateral matching becomes a mirror-based bidirectional MV.

As shown in fig. 22, the motion information of the current CU is derived using template matching by finding the closest match between the template in the current picture (top and/or left neighboring blocks of the current CU) and the block in the reference picture (same size as the template). In addition to the FRUC Merge mode described above, template matching is also applied to the AMVP mode. In JEM, there are two candidates for AMVP, just as in HEVC. New candidates are derived using a template matching method. If the newly derived candidate from template matching is different from the first existing AMVP candidate, it is inserted at the very beginning of the AMVP candidate list and then the list size is set to 2 (i.e., the second existing AMVP candidate is removed). When applied to AMVP mode, only CU level search is applied.

2.3.3.1 CU-LEVEL MV candidate set

The MV candidate sets at the CU level include:

(i) The original AMVP candidate, if the current CU is in AMVP mode,

(ii) All of the Merge candidates are selected from the group,

(iii) Several MVs in the MV field are interpolated.

(iv) Top and left adjacent motion vectors

When using bilateral matching, each valid MV of the Merge candidate is used as an input to generate a pair of MVs that are assumed to be bilaterally matched. For example, one valid MV of the Merge candidate at the reference list a is (MVa, ref) _a ). Then find its paired reference picture ref of the bilateral MV in another reference list B _b So as to ref _a And ref _b Temporally on different sides of the current picture. If the reference ref in reference List B _b If not, then ref will be referenced _b Determined as being equal to the reference ref _a Different references and its temporal distance to the current picture is the minimum distance in list B. Determining a reference ref _b Then, based on the current picture and the reference ref _a Ref, reference _b The temporal distance between them scales MVa to derive MVb.

Four MVs from the interpolated MV field are also added to the CU-level candidate list. More specifically, the interpolated MVs at positions (0, 0), (W/2, 0), (0, H/2), and (W/2, H/2) of the current CU are added.

When FRUC is applied in AMVP mode, the original AMVP candidates are also added to the CU-level MV candidate set.

At the CU level, a maximum of 15 MVs for AMVP CU and a maximum of 13 MVs for Merge CU may be added to the candidate list.

2.3.3.2 sub-CU level MV candidate set

MV candidates set at the sub-CU level include:

(i) The determined MV is searched from the CU level,

(ii) Top, left, upper left and upper right adjacent MVs,

(iii) A scaled version of the collocated MV from the reference picture,

(iv) A maximum of 4 ATMVP candidates,

(v) A maximum of 4 STMVP candidates.

The scaled MVs from the reference pictures are derived as follows. All reference pictures in both lists are traversed. The MVs at collocated positions of sub-CUs in the reference picture are scaled as references to starting CU-level MVs.

ATMVP and STMVP candidates are limited to the first four. At the sub-CU level, a maximum of 17 MVs are added to the candidate list.

2.3.3.3 Generation of interpolated MV fields

Before encoding a frame, an interpolated motion field for the entire picture is generated based on a one-way ME. This motion field can then be used as MV candidate at CU level or sub-CU level.

First, the motion field of each reference picture in the two reference lists is traversed at the 4 × 4 block level. For each 4x4 block, if the motion associated with the block passes through a 4x4 block in the current picture (as shown in fig. 23) and the block is not assigned any interpolated motion, the motion of the reference block is scaled to the current picture according to temporal distances TD0 and TD1 (same as the MV scaling of TMVP in HEVC) and the scaled motion is assigned to the block in the current frame. If no scaled MV is assigned to a 4x4 block, the motion of the block is marked as unavailable in the interpolated motion field.

2.3.3.4 interpolation matching cost

When the motion vector points to a fractional sample position, motion compensated interpolation is required. To reduce complexity, bilinear interpolation is used for both bilateral matching and template matching instead of the conventional 8-tap HEVC interpolation.

The computation of the matching cost is somewhat different at different steps. When selecting candidates from the CU-level candidate set, the matching cost is the absolute sum-difference (SAD) of the bilateral matching or the template matching. After determining the starting MV, the matching cost C of the bilateral matching search at the sub-CU level is calculated as follows:

here, w is a weight coefficient, and is empirically set to 4.MV and MV ^s Indicating a current MV and a starting MV, respectively. SAD is still used as the matching cost for pattern matching searching at sub-CU level.

In FRUC mode, MVs are derived by using only luma samples. The derived motion will be used for MC inter prediction of luminance and chrominance. After the MV is determined, the final MC is performed using an 8-tap (8-taps) interpolation filter for luminance and a 4-tap (4-taps) interpolation filter for chrominance.

2.3.3.5 MV refinement

MV refinement is a pattern-based MV search, with bilateral cost or template matching cost as criteria. In JEM, two search modes are supported-the Unrestricted Centric Biased Diamond Search (UCBDS) and the adaptive cross search, with MV refinement at the CU level and sub-CU level, respectively. For MV refinement at both CU level and sub-CU level, MV refinement is searched directly at quarter luma sample precision, followed by one-eighth luma sample MV refinement. The MV refined search range for the CU and sub-CU step is set to 8 luma samples.

2.3.3.6 selection of prediction Direction in template matching FRUC Merge mode

In the bilateral Merge mode, bi-prediction is always applied, since the motion information of a CU is derived in two different reference pictures based on the closest match between two blocks on the current CU motion trajectory. Template matching Merge patterns do not have this restriction. In template matching Merge mode, the encoder may select a CU from list 0 uni-directional prediction, list 1 uni-directional prediction, or bi-directional prediction. The selection is based on the template matching cost as follows:

if costBi < = factor x min (cost 0, cost 1)

Then bi-directional prediction is used;

otherwise, if cost0< = cost1

Then the one-way prediction in list 0 is used;

if not, then the mobile terminal can be switched to the normal mode,

using the unidirectional prediction in table 1;

where cost0 is the SAD for the list 0 template match, cost1 is the SAD for the list 2 template match, and costBi is the SAD for the bi-predictive template match. The value of factor equals 1.25, meaning that the selection process is biased towards bi-prediction. The inter prediction direction selection may be applied only to the CU-level template matching process.

2.3.4 decoder-side motion vector refinement

In the bi-directional prediction operation, for prediction of one block region, two prediction blocks respectively formed of Motion Vectors (MVs) of list 0 and MVs of list 1 are combined to form a single prediction signal. In the decoder-side motion vector refinement (DMVR) method, the two motion vectors for bi-prediction are further refined by a two-sided template matching process. The bilateral template matching applied in the decoder is used to perform a distortion-based search between the bilateral template and the reconstructed samples in the reference picture to obtain refined MVs without transmitting additional motion information.

In DMVR, the two-sided template is generated as a weighted combination (i.e., average) of two prediction blocks from the initial MV0 of list 0 and MV1 of list 1, respectively. The template matching operation includes calculating a cost metric between the generated template and a sample region in the reference picture (around the initial prediction block). For each of the two reference pictures, the MV yielding the smallest template cost is considered as the updated MV of the list to replace the original MV. In JEM, nine MV candidates are searched for each list. The nine MV candidates include the original MV and 8 surrounding MVs that have a luminance sample offset from the original MV in either the horizontal or vertical direction, or both. Finally, the two new MVs (i.e., MV0 'and MV 1') shown in fig. 24 are used to generate the final bi-directional prediction result. The Sum of Absolute Difference (SAD) is used as a cost measure.

DMVR is applied to the bidirectionally predicted Merge mode without transmitting additional syntax elements, where one MV is from a past reference picture and another MV is from a future reference picture. In JEM, DMVR is not applied when LIC, affine motion, FRUC, or sub-CU Merge candidates are enabled for a CU.

2.3.5 local illumination Compensation

Local Illumination Compensation (IC) uses a scaling factor a and an offset b based on a linear model for illumination change. And adaptively enable or disable local illumination compensation for each inter-mode encoded Coding Unit (CU).

When IC is applied to a CU, the parameters a and b are derived using the least square error method by using neighboring samples of the current CU and their corresponding reference samples. More specifically, as shown in fig. 25, the sub-sampled (2. IC parameters are derived and applied separately for each prediction direction.

When a CU is encoded in the Merge mode, copying the IC flag from the neighboring block in a similar manner to the motion information copy in the Merge mode; otherwise, the IC flag is signaled to the CU to indicate whether LIC is applicable.

When IC is enabled for a picture, an additional CU level RD check is needed to determine whether LIC is applied to the CU. When IC is enabled for a CU, mean-Removed Absolute Sum differences (MR-SAD) and Mean-Removed Absolute Hadamard-Transformed Sum differences (MR-SATD) are used instead of SAD and SATD, respectively, for integer-pixel motion search and fractional-pixel motion search.

To reduce the coding complexity, the following coding scheme is applied in JEM. When there is no significant illumination change between the current picture and its reference picture, IC is disabled for all pictures. To identify this, a histogram of the current picture and each reference picture of the current picture are computed at the encoder. Disabling the IC for the current picture if the histogram difference between the current picture and each reference picture of the current picture is less than a given threshold; otherwise, IC is enabled for the current picture.

2.3.6 Merge/skip mode with bidirectional match refinement

The Merge candidate list is first constructed by inserting the motion vectors and reference indices of spatially neighboring and temporally neighboring blocks into the candidate list using redundancy checking until the number of available candidates reaches the maximum candidate size 19. The Merge candidate list for Merge/skip mode is constructed by inserting Spatial candidates (fig. 26), temporal candidates, affine candidates, advanced Temporal MVP (ATMVP) candidates, spatiotemporal MVP (STMVP) candidates, and additional candidates used in HEVC (combination candidates and zero candidates) according to a predefined insertion order:

(1) Spatial candidates for blocks 1-4

(2) Extrapolated affine candidates for blocks 1-4

(3)ATMVP

(4)STMVP

(5) Virtual affine candidates

(6) Spatial candidates (Block 5) (used only if the number of available candidates is less than 6)

(7) Extrapolated affine candidates (Block 5)

(8) Temporal candidates (as derived in HEVC)

(9) Non-adjacent spatial candidates followed by extrapolated affine candidates (blocks 6 to 49)

(10) Composition candidates

(11) Zero candidates

Note that in addition to STMVP and affine, the IC flag is inherited from the Merge candidate. Also, for the first four spatial candidates, bi-directional prediction candidates are inserted before candidates with unidirectional prediction.

2.3.7 JVET-K0161

In this proposal, the subblock STMVP is not proposed as a spatio-temporal Merge mode. The proposed method uses co-located blocks, which are identical to HEVC/JEM (only 1 picture, here no temporal vector). The proposed method also checks the upper and left spatial positions, which are adjusted in the proposal. Specifically, in order to check adjacent inter prediction information, a maximum of two positions are checked for each of the upper and left sides. The exact position is shown in fig. 27.

Afar (nPbW 5/2, -1), amid (nPbW/2, -1) (note: offset of the top space block above the current block)

Lfar: (-1, nPbH x 5/2), lmid (-1, nPbH/2) (note: offset of left space block above current block)

The average of the motion vectors of the upper block, the left block and the time block is calculated in the same manner as the BMS software implementation. If 3 reference inter-predicted blocks are available.

mvLX[0]＝((mvLX_A[0]+mvLX_L[0]+mvLX_C[0])*43)/128

mvLX[1]＝((mvLX_A[1]+mvLX_L[1]+mvLX_C[1])*43)/128

If only two or one inter-prediction block are available, the average of two or only one mv is used.

2.3.8 JVT-K0135

To generate a smooth fine-grained motion field, fig. 28 gives a brief description of the planar motion vector prediction process.

Planar motion vector prediction is achieved by averaging the horizontal and vertical linear interpolations on a 4x4 block basis as follows.

P(x,y)＝(H×P _h (x,y)+W×P _v (x,y)+H×W)/(2×H×W)

W and H represent the width and height of the block. (x, y) is the coordinates of the current sub-block relative to the upper left sub-block. All distances are represented by the pixel distance divided by 4. P (x, y) is the motion vector of the current sub-block.

Horizontal prediction P of position (x, y) _h (x, y) and vertical prediction P _h (x, y) is calculated as follows:

P _h (x,y)＝(W-1-x)×L(-1,y)+(x+1)×R(W,y)

P _v (x,y)＝(H-1-y)×A(x,-1)+(y+1)×B(x,H)

where L (-1, y) and R (W, y) are motion vectors of the 4x4 blocks to the left and right of the current block. A (x, -1) and B (x, H) are motion vectors of 4x4 blocks above and at the bottom of the current block.

The reference motion information of the left-side column and upper-row neighboring blocks is derived from the spatial neighboring blocks of the current block.

The reference motion information for the right column and bottom row neighboring blocks is derived as follows.

Deriving motion information for a bottom-right temporally adjacent 4x4 block

The motion vector of the right column-adjacent 4 × 4 block is calculated using the derived motion information of the lower right-adjacent 4 × 4 block and the motion information of the upper right-adjacent 4 × 4 block, as described in equation K1.

The motion vector of the bottom row neighboring 4 × 4 block is calculated using the derived motion information of the bottom right neighboring 4 × 4 block and the motion information of the bottom left neighboring 4 × 4 block, as described in equation K2.

R (W, y) = ((H-y-1) × AR + (y + 1) × BR)/H formula K1

B (x, H) = ((W-x-1) × BL + (x + 1) × BR)/W formula K2

Where AR is the motion vector of the upper right spatially neighboring 4x4 block, BR is the motion vector of the lower right temporally neighboring 4x4 block, and BL is the motion vector of the lower left spatially neighboring 4x4 block.

The motion information obtained from the neighboring blocks for each list is scaled to the first reference picture of the given list.

3. Examples of problems addressed by embodiments disclosed herein

The inventors have previously proposed a look-up table based motion vector prediction technique that uses one or more look-up tables storing at least one motion candidate to predict motion information of a block, which may be implemented in various embodiments to provide video coding with higher coding efficiency. Each LUT may include one or more motion candidates, each associated with corresponding motion information. The motion information of the motion candidate may include a prediction direction, a reference index/picture, a motion vector, a LIC flag, an affine flag, a Motion Vector Difference (MVD) precision, and/or an MVD value. The motion information may also include block location information to indicate where the motion information came from.

LUT-based motion vector prediction based on the disclosed techniques may enhance existing and future video coding standards, which are set forth in the examples described below for various implementations. Because LUTs allow encoding/decoding processes to be performed based on historical data (e.g., blocks that have already been processed), LUT-based motion vector prediction may also be referred to as a history-based motion vector prediction (HMVP) method. In LUT-based motion vector prediction methods, one or more tables with motion information from previously encoded blocks are maintained during the encoding/decoding process. These motion candidates stored in the LUT are named HMVP candidates. During encoding/decoding of one block, associated motion information in the LUT may be added to a motion candidate list (e.g., a Merge/AMVP candidate list), and the LUT may be updated after encoding/decoding of one block. The subsequent block is then encoded using the updated LUT. That is, the updating of motion candidates in the LUT is based on the encoding/decoding order of the block. The following examples should be considered as examples to explain the general concept. These examples should not be construed in a narrow manner. Further, these examples may be combined in any manner.

Some embodiments may use one or more look-up tables storing at least one motion candidate to predict motion information of a block. Embodiments may use motion candidates to indicate a set of motion information stored in a look-up table. For the conventional AMVP or Merge mode, embodiments may use AMVP or Merge candidates to store motion information.

Although current LUT-based motion vector prediction techniques overcome the drawbacks of HEVC by using historical data, only information from spatially neighboring blocks is considered.

When a motion candidate from a LUT is used in the AMVP or Merge list construction process, it is inherited directly without any change.

The design of JFET-K0161 is beneficial to coding performance. However, it requires additional derivation of the TMVP, which increases computational complexity and memory bandwidth.

4. Some examples

The following examples should be considered as examples to explain the general concept. These examples should not be construed in a narrow manner. Further, these examples may be combined in any manner.

Some embodiments using the presently disclosed techniques may jointly use motion candidates from LUTs and motion information from temporally neighboring blocks. Furthermore, the complexity reduction of JFET-K0161 is also proposed.

Using motion candidates from LUTs

1. It is proposed to construct a new AMVP/Merge candidate by using motion candidates from the LUT.

a. In one example, a new candidate may be derived by adding/subtracting an offset (or offsets) to/from the motion vector of the motion candidate from the LUT.

b. In one example, a new candidate may be derived by averaging the motion vectors of the selected motion candidates from the LUT.

i. In one embodiment, the averaging may be achieved approximately without a division operation. For example, MVa, MVb, and MVc may be averaged to

Or

For example, when N =7, the average value is (MVa + MVb + MVc) × 42/128 or (MVa + MVb + MVc) × 43/128. Note that the pre-calculation

Or

And stores it in a look-up table.

in one example, only motion vectors with the same reference picture (in both prediction directions) are selected.

in one example, the reference picture in each prediction direction is predetermined and, if necessary, the motion vectors are scaled to the predetermined reference picture.

1. In one example, the first entry (X =0 or 1) in the reference picture list X is selected as the reference picture.

2. Alternatively, for each prediction direction, the most frequently used reference picture in the LUT is selected as the reference picture.

c. In one example, for each prediction direction, a motion vector having the same reference picture as a predetermined reference picture is first selected, and then other motion vectors are selected

2. It is proposed to construct new AMVP/Merge candidates as a function of one or more motion candidates from the LUT and motion information from temporally neighboring blocks.

a. In one example, similar to STMVP or JVT-K0161, new candidates may be derived by averaging the motion candidates from LUTs and TMVPs.

b. In one example, the above blocks (e.g., amid and Afar in FIG. 27) may be replaced by one or more candidates from the LUT. Alternatively, other processes may remain unchanged, as already implemented in JFET-K0161.

3. It is proposed to construct new AMVP/Merge candidates by a function of one or more motion candidates from the LUT, AMVP/Merge candidates from spatially neighboring blocks and/or spatially non-immediately neighboring blocks, and motion information from temporal blocks.

a. In one example, one or more of the above blocks (e.g., amid and Afar in FIG. 27) may be replaced by candidates from the LUT. Alternatively, other processes may remain unchanged, as already implemented in JFET-K0161.

b. In one example, one or more of the left-side blocks (e.g., amid and Afar in FIG. 27) may be replaced with candidates from the LUT. Alternatively, other processes may remain unchanged, as already implemented in JFET-K0161.

4. It is proposed that when inserting the motion information of a block into the LUT, whether or not to prune an existing entry in the LUT may depend on the coding mode of the block.

a. In one example, if the block is encoded in Merge mode, no pruning is performed.

b. In one example, if the block is encoded in AMVP mode, no pruning is performed.

c. In one example, if a block is encoded in AMVP/Merge mode, only the latest M entries of the LUT are pruned.

d. In one example, pruning is always disabled when a block is encoded in sub-block mode (e.g., affine or ATMVP).

5. It is proposed to add motion information from the temporal block to the LUT.

a. In one example, the motion information may be from co-located blocks.

b. In one example, the motion information may be from one or more blocks from different reference pictures.

Associated with STMVP

1. It is proposed to derive new Merge candidates always using spatial Merge candidates, without considering TMVP candidates.

a. In one example, the average of two moving Merge candidates may be utilized.

b. In one example, the spatial Merge candidate and the motion candidate from the LTU may be jointly used to derive a new candidate.

2. It is proposed that the STMVP candidate may be derived using a non-immediately adjacent block (which is not a right or left adjacent block).

a. In one example, the upper block used for the STMVP candidate derivation remains unchanged, while the left block used is changed from the neighboring block to the non-immediately neighboring block.

b. In one example, the left block used for STMVP candidate derivation remains unchanged, while the used upper block is changed from a neighboring block to a non-immediately neighboring block.

c. In one example, candidates for non-immediately adjacent blocks and motion candidates from the LUT may be jointly used to derive new candidates.

3. It is proposed to derive new Merge candidates always using spatial Merge candidates, without considering TMVP candidates.

a. In one example, the average of two moving Merge candidates may be utilized.

b. Alternatively, the average of two, three or more MVs from different locations adjacent or not to the current block may be utilized.

i. In one embodiment, the MV can only be obtained from a location in the current LCU (also referred to as CTU).

in one embodiment, the MV can only be retrieved from a location in the current LCU row.

in one embodiment, MVs may only be obtained from the current LCU row or a location next to the current LCU row. An example is shown in fig. 29. Blocks A, B, C, E, and F are next to the current LCU row.

in one embodiment, MVs may only be retrieved from positions in or next to the current LCU row but not to the left of the upper left corner neighboring block. An example is shown in fig. 29. Block T is the upper left neighboring block. Blocks B, C, E, and F are next to the current LCU row, but not to the left of the top left neighboring block.

c. In one embodiment, spatial Merge candidates and motion candidates from LTUs may be jointly used to derive new candidates

4. It is proposed that the MV of the BR-block for plane motion prediction in fig. 28 is not obtained from temporal MV prediction, but from one entry of the LUT.

5. Proposing motion candidates from the LUT may be used in conjunction with other types of Merge/AMVP candidates (e.g., spatial Merge/AMVP candidates, temporal Merge/AMVP candidates, default motion candidates) to derive new candidates.

In various embodiments of this example and other examples disclosed in this patent document, pruning may include: a) comparing the motion information with the existing entries for uniqueness, and b) if unique, adding the motion information to the list, or c) if not unique, either c 1) not adding the motion information, or c 2) adding the motion information and deleting the matching existing entry. In some implementations, when a motion candidate is added to the candidate list from the table, no pruning operation is invoked.

Fig. 30 is a schematic diagram illustrating an example of the structure of a computer system or other control device 3000 that can be used to implement various portions of the disclosed technology. In fig. 30, computer system 3000 includes one or more processors 3005 and memory 3010 connected by an interconnect 3025. Interconnect 3025 may represent any one or more separate physical buses, point-to-point connections, or both, connected by appropriate bridges, adapters, or controllers. Thus, interconnect 3025 may include, for example, a system bus, a Peripheral Component Interconnect (PCI) bus, a hyper transport or Industry Standard Architecture (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB), an IIC (I2C) bus, or an Institute of Electrical and Electronics Engineers (IEEE) standard 674 bus (sometimes referred to as a "firewire").

The processor 3005 may include a Central Processing Unit (CPU) to control, for example, the overall operation of the host. In some embodiments, the processor 3005 accomplishes this by executing software or firmware stored in the memory 3010. Processor 3005 may be or include one or more programmable general purpose or special purpose microprocessors, digital Signal Processors (DSPs), programmable controllers, application Specific Integrated Circuits (ASICs), programmable Logic Devices (PLDs), or the like, or a combination of such devices.

The memory 3010 may be or include the main memory of a computer system. Memory 3010 represents any suitable form of Random Access Memory (RAM), read Only Memory (ROM), flash memory, etc., or a combination of these devices. In use, the memory 3010 may contain, among other things, a set of machine instructions that, when executed by the processor 3005, cause the processor 3005 to perform operations to implement embodiments of the disclosed technology.

Also connected to processor 3005 by interconnect 3025 is (optional) network adapter 3015. The network adapter 3015 provides the computer system 3000 with the ability to communicate with remote devices (such as storage clients and/or other storage servers) and may be, for example, an ethernet adapter or a fibre channel adapter.

Fig. 31 illustrates a block diagram of an example embodiment of a mobile device 3100 that can be used to implement portions of the disclosed technology. The mobile device 3100 may be a laptop, smartphone, tablet, camera, or other device capable of processing video. The mobile device 3100 includes a processor or controller 3101 to process data and memory 3102 in communication with the processor 3101 to store and/or buffer data. For example, the processor 3101 may include a Central Processing Unit (CPU) or a microcontroller unit (MCU). In some implementations, the processor 3101 may include a Field Programmable Gate Array (FPGA). In some implementations, the mobile device 3100 includes or communicates with a Graphics Processing Unit (GPU), a Video Processing Unit (VPU), and/or a wireless communication unit to implement various visual and/or communication data processing functions of the smartphone device. For example, the memory 3102 may include and store processor-executable code that, when executed by the processor 3101, configures the mobile device 3100 to perform various operations, such as receiving information, commands, and/or data, processing information and data, and transmitting or providing processed information/data to another data device, such as an actuator or external display. To support the various functions of the mobile device 3100, the memory 3102 may store information and data such as instructions, software, values, images, and other data processed or referenced by the processor 3101. For example, the storage function of memory 3102 may be implemented using various types of Random Access Memory (RAM) devices, read Only Memory (ROM) devices, flash memory devices, and other suitable storage media. In some implementations, mobile device 3100 includes an input/output (I/O) unit 3103 to interface processor 3101 and/or memory 3102 with other modules, units, or devices. For example, I/O unit 3103 may interface with processor 3101 and memory 3102 to utilize various wireless interfaces compatible with typical data communication standards, e.g., between one or more computers in the cloud and a user device. In some implementations, the mobile device 3100 can interface with other devices through the I/O unit 3103 using a wired connection. The mobile device 3100 may also be connected to other external interfaces (e.g., data memory) and/or to a visual or audio display device 3104 to retrieve and transmit data and information, which may be processed by a processor, stored by a memory, or displayed on the display device 3104 or an output unit of an external device. For example, display device 3104 may display a video frame that includes a block (CU, PU, or TU) that applies intra block copying based on whether the block was encoded using a motion compensation algorithm in accordance with the disclosed techniques.

In some embodiments, a video decoder device that may implement the methods of sub-block based prediction as described herein may be used for video decoding.

In some embodiments, the video decoding method may be implemented using a decoding apparatus implemented on the hardware platform as described in fig. 30 and fig. 31.

Various embodiments and techniques disclosed in this document may be described in the following list of examples.

Fig. 32 is a flow diagram of an example method 3200 for video processing in accordance with the presently disclosed technology. The method 3200 includes, at operation 3202, determining a new candidate for video processing by averaging two or more selected motion candidates. The method 3200 includes, at operation 3204, adding the new candidate to a candidate list. The method 3200 includes, at operation 3206, performing a transition between a first video block of video and a bitstream representation of the video by using the determined new candidate in the candidate list.

In some embodiments, the candidate list is a Merge candidate list and the determined new candidate is a Merge candidate.

In some embodiments, the Merge candidate list is an inter prediction Merge candidate list or an intra block copy prediction Merge candidate list.

In some embodiments, the one or more tables include motion candidates derived from previously processed video blocks prior to the first video block in the video data.

In some embodiments, there are no spatial candidates and temporal candidates available in the candidate list.

In some embodiments, the selected motion candidates are from one or more tables.

In some embodiments, the averaging is achieved without a division operation.

In some embodiments, said averaging is achieved by multiplication of a sum of motion vectors of said selected motion candidates with a scaling factor.

In some embodiments, the horizontal components of the motion vectors of the selected motion candidates are averaged to derive the horizontal component of a new candidate.

In some embodiments, the vertical components of the motion vectors of the selected motion candidates are averaged to derive the vertical component of the new candidate.

In some embodiments, the scaling factor is pre-calculated and stored in a look-up table.

In some embodiments, only motion vectors with the same reference picture are selected.

In some embodiments, only motion vectors with the same reference picture in both prediction directions are selected in both prediction directions.

In some embodiments, the target reference picture in each prediction direction is predetermined, and the motion vectors are scaled to the predetermined reference picture.

In some embodiments, the first entry in reference picture list X is selected as the target reference picture for the reference picture list, X being 0 or 1.

In some embodiments, for each prediction direction, the most frequently used reference picture in the table is selected as the target reference picture.

In some embodiments, for each prediction direction, motion vectors having the same reference picture as the predetermined target reference picture are first selected, and then other motion vectors are selected.

In some embodiments, the motion candidates from the table are associated with motion information, the motion information comprising at least one of: prediction direction, reference picture index, motion vector value, intensity compensation flag, affine flag, motion vector difference precision, or motion vector difference value.

In some embodiments, method 3200 further comprises updating one or more tables based on the translation.

In some embodiments, the updating of the one or more tables includes updating the one or more tables based on motion information of the first video block of the video after performing the converting.

In some embodiments, method 3200 further comprises performing a transition between a subsequent video block of the video and a bitstream representation of the video based on the updated table.

In some embodiments, the conversion includes an encoding process and/or a decoding process.

In some embodiments, the video encoding device may perform method 2900 and other methods described herein during reconstruction of video for subsequent video.

In some embodiments, an apparatus in a video system may include a processor configured to perform the methods described herein.

In some embodiments, the described methods may be embodied as computer executable code stored on a computer readable program medium.

Fig. 33 is a flow diagram of an example method 3300 for video processing in accordance with the presently disclosed technology. The method 3300 includes determining a new motion candidate for video processing by using one or more motion candidates from one or more tables in operation 3302, where the tables include the one or more motion candidates and each motion candidate is associated motion information. Method 3300 includes, at operation 3304, performing a conversion between the video block and an encoded representation of the video block based on the new candidate.

In some embodiments, the new motion candidate is derived by adding or subtracting an offset to or from motion vectors associated with motion candidates from the one or more tables.

In some embodiments, determining a new motion candidate comprises: the new motion candidate is determined as a function of one or more motion candidates from the one or more tables and motion information from the temporal neighboring blocks.

In some embodiments, determining a new motion candidate comprises: motion candidates and temporal motion vector predictors from one or more tables are averaged.

In some embodiments, averaging the selected motion candidate comprises a weighted average or mean of the motion vectors associated with the selected motion candidate.

In some embodiments, the averaging is achieved without a division operation.

In some embodiments, the averaging is achieved by multiplication of a sum of motion vectors of motion candidates from the one or more tables by the temporal motion vector predictor and a scaling factor.

In some embodiments, the horizontal components of the motion vectors of the motion candidates from the one or more tables are averaged with a temporal motion vector predictor to derive the horizontal component of the new motion candidate.

In some embodiments, averaging the selected horizontal components comprises a weighted average or average of the horizontal components associated with the selected motion candidates.

In some embodiments, averaging the selected vertical components comprises a weighted average or average of the vertical components associated with the selected motion candidates.

In some embodiments, the new motion candidate is determined as a function of one or more motion candidates from one or more tables, merge candidates from spatially neighboring blocks and/or spatially non-immediately neighboring blocks, and motion information from temporally neighboring blocks.

In some embodiments, determining the new candidate comprises: the new motion candidate is determined as a function of one or more motion candidates from one or more tables, advanced Motion Vector Prediction (AMVP) candidates from spatially neighboring blocks and/or spatially non-immediately neighboring blocks, and motion information from temporally neighboring blocks.

In some embodiments, determining the new candidate comprises: the new motion candidate is determined as a function of the one or more motion candidates from the one or more tables and an Advanced Motion Vector Prediction (AMVP) candidate in an AMVP candidate list or a Merge candidate in a Merge candidate list.

In some embodiments, the new motion candidate is added to the Merge candidate list.

In some embodiments, the new motion candidate is added to the AMVP candidate list.

In some embodiments, each of the one or more tables includes a set of motion candidates, where each motion candidate is associated with corresponding motion information.

In some embodiments, the motion candidates are associated with motion information comprising at least one of: prediction direction, reference picture index, motion vector value, intensity compensation flag, affine flag, motion vector difference precision, or motion vector difference value.

In some embodiments, the method further comprises updating one or more tables based on the translation.

In some embodiments, updating the one or more tables includes updating the one or more tables based on motion information of the first video block after performing the conversion.

In some embodiments, the method further comprises performing a conversion between a subsequent video block of the video and the bitstream representation of the video based on the updated table.

Fig. 34 is a flow diagram of an example method 3400 for video processing in accordance with the presently disclosed technology. The method 3400 includes determining new candidates for video processing by always using motion information from more than one spatially neighboring blocks of a first video block in a current picture, and not using motion information from temporal blocks in a picture different from the current picture, in operation 3402. The method 3400 includes performing a conversion between a first video block in a current picture of a video and a bitstream representation of the video by using the determined new candidate at operation 3404.

In some embodiments, the determined new candidate is added to a candidate list, the candidate list comprising a Merge candidate list or an Advanced Motion Vector Prediction (AMVP) candidate list.

In some embodiments, the motion information from more than one spatial neighboring block is a candidate derived from a predefined spatial neighboring block relative to the first video block in the current picture, or a motion candidate from one or more tables.

In some embodiments, the one or more tables include motion candidates derived from previously processed video blocks that were processed prior to a first video block in the video data.

In some embodiments, the candidate derived from a predefined spatial neighboring block relative to the first video block in the current picture is a spatial Merge candidate.

In some embodiments, the new candidate is derived by averaging at least two spatial Merge candidates.

In some embodiments, the new candidate is derived by jointly using the spatial Merge candidate and the motion candidate from one or more tables.

In some embodiments, the new candidate is derived by averaging at least two motion vectors associated with candidates derived from different positions.

In some embodiments, the different location is adjacent to the first video block.

In some embodiments, the motion vector is obtained only from a position in a current largest coding unit to which the first video block belongs.

In some embodiments, the motion vector is only obtained from a position in the current maximum coding unit row.

In some embodiments, the motion vector is only obtained from a location in or next to the current maximum coding unit row.

In some embodiments, the motion vector is only obtained from a position in or next to the current LCU row but not to the left of the upper-left neighboring block.

In some embodiments, the motion vector of the lower right block for planar motion prediction is not obtained from the temporal motion vector prediction candidates, but from an entry of the table.

In some embodiments, the new candidate is derived by jointly using motion candidates from one or more tables and other kinds of Merge/AMVP candidates.

In some embodiments, the motion candidates in the one or more tables are associated with motion information, the motion information comprising at least one of: prediction direction, reference picture index, motion vector value, intensity compensation flag, affine flag, motion vector difference precision, or motion vector difference value.

In some embodiments, the method further comprises updating one or more tables based on the translation.

In some embodiments, updating one or more tables includes updating one or more tables based on motion information of the first video block after performing the conversion.

In some embodiments, the method further comprises performing a conversion between a subsequent video block of the video and the bitstream representation of the video based on the updated table.

In some embodiments, the conversion comprises an encoding process and/or a decoding process.

Fig. 35 is a flow diagram of an example method 3500 for video processing in accordance with the presently disclosed technology. The method 3500 includes determining new candidates for video processing at operation 3502 by using motion information from at least one spatially non-adjacent block of a first video block in a current picture and other candidates derived from spatially non-adjacent blocks of the first video block or not. The method 3500 includes performing, at operation 3504, a conversion between a first video block of a video and a bitstream representation of the video by using the determined new candidate.

In some embodiments, the determined new candidate is added to a candidate list comprising a Merge or Advanced Motion Vector Prediction (AMVP) candidate list.

In some embodiments, the motion information from more than one spatially non-adjacent block is a candidate derived from a predefined spatially non-adjacent block relative to the first video block in the current picture.

In some embodiments, the candidate derived from a predefined spatially non-immediately adjacent block relative to the first video block in the current picture is a spatio-temporal motion vector prediction (STMVP) candidate.

In some embodiments, non-immediately adjacent blocks of the video block are not right-adjacent blocks or left-adjacent blocks of the first video block.

In some embodiments, the upper block of the first video block used for STMVP candidate derivation remains unchanged, while the left block used is changed from a neighboring block to a non-immediately neighboring block.

In some embodiments, the left block of the first video block used for STMVP candidate derivation remains unchanged, while the upper block used is changed from a neighboring block to a non-immediately neighboring block.

Fig. 36 is a flow diagram of an example method 3600 for video processing in accordance with the presently disclosed technology. The method 3600 includes determining a new candidate for video processing by using motion information from one or more tables of a first video block in a current picture and motion information from a temporal block in a picture different from the current picture, at operation 3602. The method 3600 includes performing a conversion between a first video block in a current picture of a video and a bitstream representation of the video by using the determined new candidate, at operation 3604.

In some embodiments, the determined new candidate is added to a candidate list comprising a Merge or AMVP candidate list.

In some embodiments, motion information from one or more tables in the current picture is associated with one or more Historical Motion Vector Prediction (HMVP) candidates selected from the one or more tables, and motion information from temporal blocks in a picture other than the current picture are temporal motion candidates.

In some embodiments, the new candidate is derived by averaging one or more HMVP candidates with one or more temporal motion candidates.

In some embodiments, the one or more tables include motion candidates derived from previously processed video blocks processed prior to a first video block in the video data.

Fig. 37 is a flow diagram of an example method 3700 for video processing in accordance with the presently disclosed technology. Method 3700 includes determining new candidates for video processing at operation 3702 by using motion information from one or more tables of a first video block and motion information from one or more spatially neighboring blocks of the first video block. The method 3700 includes performing a conversion between a first video block in a current picture of a video and a bitstream representation of the video by using the determined new candidate, at operation 3704.

In some embodiments, the determined new candidate is added to a candidate list comprising a Merge or AMVP candidate list

In some embodiments, motion information from one or more tables of the first video block is associated with one or more Historical Motion Vector Prediction (HMVP) candidates selected from the one or more tables, and motion information from one or more spatial neighboring blocks of the first video block is a candidate derived from a predefined spatial block relative to the first video block.

In some embodiments, the candidate derived from the predefined spatial block relative to the first video block is a spatial Merge candidate.

In some embodiments, the new candidate is derived by averaging one or more HMVP candidates and one or more spatial Merge candidates.

In some embodiments, the one or more tables include motion candidates derived from previously processed video blocks that were processed prior to a first video block in the video data.

In some embodiments, the motion candidates from the table are associated with motion information, the motion information comprising at least one of: prediction direction, reference picture index, motion vector value, intensity compensation flag, affine flag, motion vector difference precision, or motion vector difference value.

In some embodiments, the method further comprises updating one or more tables based on the translation.

In some embodiments, updating the one or more tables includes updating the one or more tables based on motion information of the current video block after performing the conversion.

In some embodiments, the method further comprises performing a conversion between a subsequent video block of the video data and a bitstream representation of the video data based on the updated table.

Fig. 38 is a flow diagram of an example method 3800 for video processing in accordance with the presently disclosed technology. The method 3800 includes, at operation 3802, maintaining a set of tables, wherein each table includes motion candidates, and each motion candidate is associated with corresponding motion information; at operation 3804, performing a conversion between a first video block and a bitstream representation of a video including the first video block; and updating the one or more tables by selectively pruning existing motion candidates in the one or more tables based on the encoding/decoding mode of the first video block, in operation 3806.

In some embodiments, the conversion between the first video block and the bitstream representation of the video comprising the first video block is performed based on one or more tables of the set of tables.

In some embodiments, in the case of encoding/decoding the first video block in Merge mode, pruning is omitted.

In some embodiments, in the case of encoding/decoding the first video block in the advanced motion vector prediction mode, the pruning is omitted.

In some embodiments, the latest M entries of the table are pruned in case the first video block is encoded/decoded in the Merge mode or the advanced motion vector prediction mode, where M is a pre-specified integer.

In some embodiments, where the first video block is encoded/decoded in sub-block mode, pruning is disabled.

In some embodiments, the sub-block modes include an affine mode, an optional temporal motion vector prediction mode.

In some embodiments, the pruning includes checking whether there are redundant existing motion candidates in the table.

In some embodiments, the trimming further comprises: inserting motion information associated with the first video block into the table if there is a redundant existing motion candidate in the table, and deleting the redundant existing motion candidate in the table.

In some embodiments, the table is not updated with motion information associated with the first video block if there are redundant existing motion candidates in the table.

In some embodiments, the method further comprises performing a conversion between a subsequent video block of the video and the bitstream representation of the video based on the updated table.

Fig. 39 is a flow diagram of an example method 3900 for video processing in accordance with the presently disclosed technology. The method 3900 includes, at operation 3902, maintaining a set of tables, wherein each table includes motion candidates, and each motion candidate is associated with corresponding motion information; at operation 3904, performing a conversion between a first video block and a bitstream representation of video that includes the first video block; and at operation 3906, updating one or more tables to include motion information from one or more temporally neighboring blocks of the first video block as new motion candidates.

In some embodiments, the conversion between the first video block and the bitstream representation of the video comprising the first video block is performed based on one or more tables of the set of tables.

In some embodiments, the one or more temporally adjacent blocks are co-located blocks.

In some embodiments, the one or more temporal neighboring blocks comprise one or more blocks from different reference pictures.

In some embodiments, the method further comprises performing a conversion between a subsequent video block of the video and the bitstream representation of the video based on the updated table.

FIG. 40 is a flow diagram of an example method 4000 for updating a motion candidate table in accordance with the presently disclosed technology. The method 4000 includes, at operation 4002, selectively pruning existing motion candidates in a table based on an encoding/decoding mode of a video block being processed, each motion candidate being associated with corresponding motion information; and in operation 4004, updating the table to include motion information of the video block as a new motion candidate.

In some embodiments, the latest M entries of the table are pruned if the video block is encoded/decoded in either Merge mode or advanced motion vector prediction mode, where M is a pre-specified integer.

In some embodiments, pruning is disabled if the video block is encoded/decoded in sub-block mode.

In some embodiments, the sub-block modes include an affine mode, an optional temporal motion vector prediction mode.

In some embodiments, the pruning includes checking whether there are redundant existing motion candidates in the table.

In some embodiments, the trimming further comprises: if there are redundant motion candidates in the table, motion information associated with the video block being processed is inserted into the table and the redundant motion candidates in the table are deleted.

In some embodiments, if there are redundant existing motion candidates in the table, the table is not updated with motion information associated with the video block being processed.

Fig. 41 is a flow diagram of an example method 4100 for updating a motion candidate table in accordance with the presently disclosed techniques. The method 4100 includes, at operation 4102, maintaining a motion candidate table, each motion candidate associated with corresponding motion information; and in operation 4104, the table is updated to include motion information from one or more temporally neighboring blocks of the video block being processed as new motion candidates.

In some embodiments, the one or more temporally adjacent blocks are co-located blocks.

In some embodiments, the one or more temporal neighboring blocks comprise one or more blocks from different reference pictures.

In some embodiments, the motion candidates are associated with motion information, the motion information comprising at least one of: prediction direction, reference picture index, motion vector value, intensity compensation flag, affine flag, motion vector difference precision, or motion vector difference value.

In some embodiments, the motion candidates correspond to motion candidates for intra prediction modes used for intra mode encoding.

In some embodiments, the motion candidates correspond to motion candidates that include illumination compensation parameters for IC parametric coding.

Fig. 42 is a flow diagram of an example method 4200 for video processing according to the presently disclosed technology. Method 4200 includes, at operation 4202, determining a new motion candidate for video processing by using one or more motion candidates from one or more tables, wherein the tables include the one or more motion candidates and each motion candidate is associated with motion information; and performing a transition between the video block and the encoded representation of the video block based on the new candidate in operation 4104.

In some embodiments, the determined new candidate is added to a candidate list, which includes a Merge or Advanced Motion Vector Prediction (AMVP) candidate list.

In some embodiments, determining the new candidate comprises: determining the new motion candidate as a function of one or more motion candidates from one or more tables and an Advanced Motion Vector Prediction (AMVP) candidate in an AMVP candidate list or a Merge candidate in a Merge candidate list.

From the foregoing, it will be appreciated that specific embodiments of the disclosed technology have been described herein for purposes of illustration, but that various modifications may be made without deviating from the scope of the invention. Accordingly, the disclosed technology is not limited except as by the appended claims.

The embodiments, modules, and functional operations disclosed herein and otherwise described may be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed herein and structural equivalents thereof, or in combinations of one or more of them. The disclosed embodiments and other embodiments may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them. The term "data processing apparatus" includes all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or groups of computers. The apparatus can include, in addition to hardware, code that creates an execution environment for the computer program, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. Propagated signals are artificially generated signals, e.g., machine-generated electrical, optical, or electromagnetic signals, that are generated to encode information for transmission to suitable receiver apparatus.

A computer program (also known as a program, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

The processes and logic flows described herein can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer does not necessarily have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; a magneto-optical disk; and CDROM and DVD-ROM discs. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

It is intended that the specification, together with the drawings, be considered exemplary only, with the examples being meant as examples. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. In addition, the use of "or" is intended to include "and/or" unless the context clearly indicates otherwise.

While this patent document contains many specifics, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features that are described in this patent document in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various functions described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claim combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

Likewise, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. Moreover, the separation of various system components in the embodiments described herein should not be understood as requiring such separation in all embodiments.

Only a few implementations and examples have been described and other implementations, enhancements and variations can be made based on what is described and illustrated in this patent document.

Claims (22)

1. A video processing method, comprising:

determining a new motion candidate for video processing by using one or more motion candidates from one or more tables, wherein a table comprises one or more motion candidates and each motion candidate is associated motion information; and

performing a conversion between the video block and an encoded representation of the video block based on the new motion candidate;

wherein determining a new motion candidate comprises:

determining new motion candidates as a function of one or more motion candidates from one or more tables and motion information from a temporal neighboring block; or

Determining new motion candidates as a function of one or more motion candidates from one or more tables, advanced motion vector prediction, AMVP, candidates from spatially neighboring blocks and/or spatially non-immediately neighboring blocks, and motion information from temporally neighboring blocks; or

The new motion candidate is determined as a function of the one or more motion candidates from the one or more tables and the AMVP candidate in the advanced motion vector prediction AMVP candidate list or the Merge candidate in the Merge candidate list.

2. The method of claim 1, wherein the new motion candidate is derived by adding or subtracting an offset to or from a motion vector associated with a motion candidate from the one or more tables.

3. The method of claim 1, wherein determining new motion candidates comprises:

the motion candidates and temporal motion vector predictors from the one or more tables are averaged.

4. The method of claim 3, wherein averaging the selected motion candidates comprises a weighted average or an average of motion vectors associated with the selected motion candidates.

5. The method of claim 3, wherein the averaging is accomplished without a division operation.

6. The method of claim 3, wherein the averaging is achieved by multiplication of the temporal motion vector predictor and a scaling factor by a sum of motion vectors of motion candidates from the one or more tables.

7. The method of claim 1, wherein horizontal components of motion vectors of motion candidates from the one or more tables are averaged with a temporal motion vector predictor to derive horizontal components of new motion candidates.

8. The method of claim 7, wherein averaging the selected horizontal components comprises a weighted average or an average of the horizontal components associated with the selected motion candidates.

9. The method of claim 1, wherein a vertical component of a motion vector of a motion candidate from the one or more tables is averaged with a temporal motion vector predictor to derive a vertical component of a new motion candidate.

10. The method of claim 9, wherein averaging the selected vertical component comprises a weighted average or an average of the vertical components associated with the selected motion candidate.

11. The method of claim 1, wherein the one or more motion candidates from the one or more tables are derived from previously processed video blocks in the video data that were processed prior to the video block.

12. The method of claim 1, wherein determining new candidates comprises:

the new motion candidate is determined as a function of one or more motion candidates from one or more tables, merge candidates from spatially neighboring blocks and/or spatially non-immediately neighboring blocks, and motion information from temporally neighboring blocks.

13. The method of any one of claims 1 to 12, wherein the new motion candidate is added to a Merge candidate list.

14. The method of any one of claims 1-12, wherein the new motion candidate is added to an AMVP candidate list.

15. The method of any of claims 1-12, wherein each of the one or more tables comprises a set of motion candidates, wherein each motion candidate is associated with corresponding motion information.

16. The method of any of claims 1-12, wherein the motion candidate is associated with motion information, the motion information comprising at least one of: prediction direction, reference picture index, motion vector value, intensity compensation flag, affine flag, motion vector difference precision, or motion vector difference value.

17. The method of any of claims 1-12, further comprising updating one or more tables based on the translation.

18. The method of claim 17, wherein updating one or more tables comprises updating one or more tables based on motion information of the first video block after performing the conversion.

19. The method of any of claims 1-12, further comprising:

based on the updated table, a conversion between subsequent video blocks of the video and a bitstream representation of the video is performed.

20. A video decoding apparatus comprising a processor configured to implement the method of one or more of claims 1 to 19.

21. A video encoding apparatus comprising a processor configured to implement the method of one or more of claims 1 to 19.

22. A non-transitory computer-readable program medium having code stored thereon, the code comprising instructions that, when executed by a processor, cause the processor to implement a method as recited in one or more of claims 1-19.

Images