TWI820169B - Extension of look-up table based motion vector prediction with temporal information - Google Patents

Abstract

The application provides a method for video processing. This method includes: determining a new candidate for video processing by averaging motion vectors of two or more selected motion candidates; adding the new candidate to a candidate list; and performing a conversion between a first video block of a video and a bitstream representation of the video by using the determined new candidate in the candidate list.

Description

Extending lookup table-based motion vector prediction with temporal information

The present invention relates to video coding technology, equipment and systems.

Cross-references to related applications This invention timely claims the benefit of International Patent Application No. PCT/CN2018/095716, filed on July 14, 2018, and International Patent Application No. PCT/, filed on July 15, 2018, in accordance with the provisions of applicable patent law and/or the Paris Convention. Priority and interests of CN2018/095719. The entire disclosures of International Patent Application Nos. PCT/CN2018/095716 and PCT/CN2018/095719 are incorporated herein by reference as part of the disclosure of the present invention.

Despite advances in video compression, digital video still uses the largest bandwidth on the Internet and other digital communications networks. Bandwidth demands for digital video usage are expected to continue to grow as the number of connected user devices capable of receiving and displaying video increases.

This document discloses methods, systems, and apparatus for encoding and decoding digital video using Merge lists of motion vectors.

In an example aspect, a video processing method is disclosed. The video processing method includes: determining new candidates for video processing by averaging two or more selected motion candidates; adding the new candidates to a candidate list; and using the determined new candidates in the candidate list, Performs a conversion between a first video block of the video and a bitstream representation of the video.

In an example aspect, a video processing method is disclosed. The video processing method includes determining new motion candidates for video processing by using one or more motion candidates from one or more tables, wherein the table includes the one or more motion candidates and each motion candidate is associated Motion information; performs conversion between video blocks and encoded representations of video blocks based on new candidates.

In an example aspect, a video processing method is disclosed. The video processing method includes determining by always using motion information from more than one spatial neighbor block of a first video block in a current picture, and not using motion information from temporal blocks in a picture different from the current picture. New candidates for video processing; performing a conversion between a first video block in the current picture of the video and a bitstream representation of the video by using the determined new candidates.

In an example aspect, a video processing method is disclosed. The video processing method includes: by using motion information from at least one spatially non-immediately adjacent block of a first video block in the current picture, and from a spatially non-immediately adjacent block of the first video block or from a spatially non-immediately adjacent block of the first video block. Other candidates derived from the block are used to determine new candidates for video processing; by using the determined new candidates, a conversion between the first video block of the video and the bit stream representation of the video is performed.

In an example aspect, a video processing method is disclosed. The video processing method includes determining a new video block for video processing by using motion information from one or more tables of a first video block in a current picture and motion information from a temporal block in a picture different from the current picture. Candidate; performs a conversion between the first video block in the current picture of the video and the bitstream representation of the video by using the determined new candidate.

In an example aspect, a video processing method is disclosed. The video processing method includes determining new candidates for video processing by using motion information from one or more tables of a first video block and motion information from one or more spatially neighboring blocks of the first video block. ; Perform a conversion between the first video block in the current picture of the video and the bitstream representation of the video by using the determined new candidate.

In an example aspect, a video processing method is disclosed. The video processing method includes: maintaining a set of tables, wherein each table includes motion candidates, and each motion candidate is associated with corresponding motion information; and executing a first video block and a bit stream of a video including the first video block. converting between representations; and updating the one or more tables by selectively pruning existing motion candidates in the one or more tables based on the encoding/decoding mode of the first video block.

In an example aspect, a video processing method is disclosed. The video processing method includes: maintaining a set of tables, wherein each table includes motion candidates, and each motion candidate is associated with corresponding motion information; executing a first video block and a bit stream of a video including the first video block converting between representations; and updating one or more tables to include motion information from one or more temporally adjacent blocks of the first video block as new motion candidates.

In one example aspect, a method of updating a motion candidate table is disclosed. The method includes: selectively pruning existing motion candidates in a table, each motion candidate being associated with corresponding motion information, based on the encoding/decoding mode of the video block being processed; and updating the table to include The motion information of video blocks is used as new motion candidates.

In one example aspect, a method of updating a motion candidate table is disclosed. The method includes maintaining a table of motion candidates, each motion candidate being associated with corresponding motion information; and updating the table to include motion information from one or more temporally adjacent blocks of the video block being processed as new campaign candidates.

In an example aspect, a video processing method is disclosed. The video processing method includes determining new motion candidates for video processing by using one or more motion candidates from one or more tables, wherein the table includes the one or more motion candidates and each motion candidate is associated with motion information is associated; and based on the new candidate, a conversion is performed between the video block and the encoded representation of the video block.

In an example aspect, an apparatus in a video system is disclosed. The apparatus includes a processor and non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the various methods described herein.

Various technologies described herein may be implemented as a computer program product stored on non-transitory computer-readable media. A computer program product includes program code for performing the methods described herein.

The details of one or more implementations are set forth in the attachments, figures, and the description below. Other features will be apparent from the description and drawings, and from the patent claims.

In order to improve the compression ratio of videos, researchers are constantly looking for new technologies to encode videos.

1. introduce

This document is related to video encoding technology. Specifically, it is related to motion information coding (such as Merge mode, AMVP mode) in video coding. It can be applied to the existing video coding standard HEVC, or to the yet-to-be-finalized standard Versatile Video Coding (VVC). May also apply to future video encoding standards or video transcoders.

briefly discuss

Video coding standards have been developed primarily through the development of the well-known ITU-T and ISO/IEC standards. ITU-T developed H.261 and H.263, ISO/IEC developed MPEG-1 and MPEG-4 visual, and the two organizations jointly developed H.262/MPEG-2 video, H.264/MPEG-4 Advanced Video Coding (AVC) and H.265/HEVC standards. Since H.262, video coding standards are based on a hybrid video coding structure, in which temporal prediction plus transform coding is used. An example of a typical HEVC encoder framework is shown in Figure 1.

2.1 split structure

2.1.1 H.264/AVC Split tree structure in

The core of the coding layer in the previous standard was the macro block, which contained a 16×16 block of luma samples and, in the case of conventional 4:2:0 color sampling, two corresponding 8×8 blocks of chroma samples.

The intra coding block uses spatial prediction to explore the spatial correlation between pixels. Two splits are defined: 16x16 and 4x4.

Inter-coded blocks use temporal prediction, rather than spatial prediction, by estimating motion between pictures. The motion of a 16x16 macroblock or any of its sub-macroblock partitions can be estimated individually: 16x8, 8x16, 8x8, 8x4, 4x8, 4x4 (see Figure 2). Only one motion vector (MV) is allowed per sub-macroblock partition.

2.1.2 HEVC Split tree structure in

In HEVC, various local characteristics are accommodated by dividing the CTU into CUs using a quadtree structure (represented as a coding tree). It is decided at the CU level whether to encode a picture region using inter (temporal) prediction or intra (spatial) prediction. Depending on the PU's split type, each CU can be further divided into one, two, or four PUs. In a PU, the same prediction process is applied, and the relevant information is transferred to the decoder on a PU basis. After obtaining the residual block by applying prediction processing based on the PU partition type, the CU can be partitioned into transform units (TUs) according to another quadtree structure similar to the coding tree of the CU. An important feature of the HEVC structure is that it has multiple partition concepts, including CU, PU and TU.

In the following, various features involved in hybrid video coding using HEVC are highlighted below.

1) Coding tree unit and coding tree block (CTB) structure: A similar structure in HEVC is the coding tree unit (CTU), which has a size selected by the encoder and can be larger than traditional macroblocks. A CTU consists of a luma CTB and the corresponding chroma CTB as well as syntax elements. Luminance The size L×L of the CTB can be chosen to be L=16, 32 or 64 samples, with larger sizes generally enabling better compression. Then, HEVC supports dividing the CTB into smaller blocks using tree structure and quad-tree signaling.

2) Coding unit (CU) and coding block (CB): The quadtree syntax of CTU specifies the size and position of its brightness and chroma CB. The root of the quadtree is associated with the CTU. Therefore, the size of the luma CTB is the largest size supported by the luma CB. The division of CTU luminance and chroma CB is jointly signaled. A luma CB and usually two chroma CBs and associated syntax form a coding unit (CU). The CTB can contain only one CU, or it can be divided to form multiple CUs, and each CU has an associated partition, divided into prediction units (PU) and transformation unit trees (TU).

3) Prediction unit (PU) and prediction block (PB): It is decided at the CU level whether to use inter prediction or intra prediction to encode the picture area. The root of the PU partition structure is at the CU level. Depending on the underlying prediction type decision, the luma and chroma CBs can be further partitioned in size and predicted from luma and chroma prediction blocks (PBs). HEVC supports variable PB sizes from 64×64 to 4×4 samples. Figure 3 shows an example of allowed PBs for an MxM CU.

4) Transform Unit (TU) and Transform Block (TB): Block transform is used to encode the prediction residual. The root of the TU tree structure is at the CU level. The luma CB residual may be the same as the luma TB, or it may be further divided into smaller luma transform blocks TB. The same applies to Chroma TB. Integer basis functions similar to the discrete cosine transform (DCT) are defined for square TBs of 4×4, 8×8, 16×16 and 32×32. For a 4×4 transform of the luma intra prediction residual, it is also possible to specify an integer transform derived from the discrete sine transform (DST) form.

Figure 4 shows an example of subdividing a CTB into CBs (and Transform Blocks (TBs)). Solid lines indicate CB boundaries, and dashed lines indicate TB boundaries. (a) CTB with segmentation. (b) Corresponding quadtree.

2.1.2.1 Tree structure partitioning into transformation blocks and units

For residual coding, CB can be recursively split into transform blocks (TB). The partitioning is signaled by a residual quadtree. Only square CB and TB partitions are specified, where blocks can be recursively divided into four quadrants, as shown in Figure 4. For a given brightness CB of size M×M, the flag indicates whether it is divided into four blocks of size M/2×M/2. If further partitioning is possible, as indicated by the maximum depth of the residual quadtree indicated in the Sequence Parameter Set (SPS), each quadrant is assigned a flag indicating whether it should be divided into four quadrants. The leaf node blocks generated by the residual quadtree are transform blocks that are further processed by transform coding. The encoder indicates the maximum and minimum brightness TB size it will use. When the CB size is larger than the maximum TB size, partitioning is implied. No partitioning is implied when partitioning would result in a smaller luma TB size than the indicated minimum. The chroma TB size is half the luma TB size in each dimension, except when the luma TB size is 4×4, in which case the area covered by four 4×4 luma TBs uses a single 4×4 Chroma TB. In the case of intra-predicted CUs, the decoded samples of the nearest neighboring TB (inside or outside the CB) are used as reference materials for intra prediction.

Unlike previous standards, for inter-predicted CUs, the HEVC design allows a TB to span multiple PBs to maximize the potential coding efficiency benefiting from the quadtree structure of TB partitioning.

2.1.2.2 Parent node and child node

The CTB is divided according to the quadtree structure, and its nodes are coding units. Multiple nodes in a quadtree structure include leaf nodes and non-leaf nodes. Leaf nodes have no child nodes in the tree structure (i.e., leaf nodes are not further divided). Non-leaf nodes include the root node of the tree structure. The root node corresponds to the initial video block (eg, CTB) of the video material. For each respective non-root node of the plurality of nodes, the respective non-root node corresponds to a video block that is a child block of the video block corresponding to the parent node in the tree structure of the respective non-root node. Each respective non-leaf node of the plurality of non-leaf nodes has one or more child nodes in the tree structure.

2.1.3 joint exploration model ( JEM ) has a larger CTU Quadtree plus binary tree block structure

To explore future video coding technologies beyond HEVC, VCEG and MPEG jointly established the Joint Video Exploration Team (JVET) in 2015. Since then, JVET has adopted many new methods and implemented them into a reference software called the Joint Exploration Model (JEM).

2.1.3.1 QTBT block partition structure

Unlike HEVC, the QTBT structure eliminates the concept of multiple partition types, that is, it eliminates the separation of CU, PU, and TU concepts, and supports more flexibility in CU partition shapes. In QTBT block structure, CU can be square or rectangular. As shown in Figure 5, the coding tree unit (CTU) is first divided using a quad-tree structure. The quad leaf nodes are further divided by the binary tree structure. There are two types of partitioning in binary tree partitioning: symmetric horizontal partitioning and symmetric vertical partitioning. Binary leaf nodes are called coding units (CUs), and this partitioning is used for prediction and transformation processing without further partitioning. This means that CU, PU and TU have the same block size in QTBT coding block structure. In JEM, a CU is sometimes composed of coding blocks (CB) of different color components. For example, in the P strip and B strip of the 4:2:0 chroma format, a CU contains one luma CB and two chroma components. Degree CB, and a CU sometimes consists of a single component CB, for example, in the case of I-strips, a CU contains only one luminance CB or only two chroma CBs.

The following parameters are defined for the QTBT segmentation scheme.

–CTU size: The root node size of the quadtree, the same concept as in HEVC.

– MiNQTSize : Minimum allowed size of four-fork leaf nodes

– MaxBTSize : The maximum allowed binary tree root node size

– MaxBTDePTh : Maximum allowed binary tree depth

– MiNBTSize : Minimum allowed binary leaf node size

In an example of the QTBT partitioning structure, the CTU size is set to 128×128 luma samples with two corresponding blocks of 64×64 chroma samples, MiNQTSize is set to 16×16, MaxBTSize is set to 64×64, MiNBTSize (width and height) is set to 4×4 and MaxBTSize is set to 4. Quadtree partitioning is first applied to the CTU to generate quadtree leaf nodes. The size of a quad leaf node can have dimensions from 16×16 (i.e., MiNQTSize ) to 128×128 (i.e., CTU size). If a leaf quadtree node is 128×128, it will not be further divided by the binary tree because its size exceeds MaxBTSize (i.e., 64×64). Otherwise, leaf quadtree nodes can be further split by binary trees. Therefore, the quad leaf node is also the root node of the binary tree, and its binary tree depth is 0. When the binary tree depth reaches MaxBTDePTh (i.e., 4), no further partitioning is considered. When the width of a binary tree node is equal to MiNBTSize (i.e., 4), further horizontal divisions are not considered. Likewise, when the height of a binary tree node is equal to MiNBTSize , further vertical divisions are not considered. The leaf nodes of the binary tree are further processed through prediction and transformation processing without further segmentation. In JEM, the maximum CTU size is 256×256 luminance samples.

Figure 5 (left) illustrates an example of block segmentation by using QTBT, and Figure 5 (right) illustrates the corresponding tree representation. Solid lines represent quadtree splits, and dashed lines represent binary tree splits. In each partition (i.e., non-leaf) node of the binary tree, a flag is signaled to indicate which type of partition (i.e., horizontal or vertical) is used, where 0 indicates a horizontal partition and 1 indicates a vertical partition. For quadtree partitioning, there is no need to specify the partition type, because quadtree partitioning always divides a block horizontally and vertically to generate 4 sub-blocks of the same size.

In addition, the QTBT solution supports brightness and chroma capabilities with separate QTBT structures. Currently, the luma and chroma CTBs in a CTU share the same QTBT structure for P-strips and B-strips. However, for I-strips, a QTBT structure is used to split the luma CTB into CUs, and another QTBT structure is used to split the chroma CTB into chroma CUs. This means that a CU in an I-slice consists of coding blocks for the luma component or two chroma components, and a CU in a P-slice or B-slice consists of coding blocks for all three color components.

In HEVC, in order to reduce memory access for motion compensation, inter prediction of small blocks is restricted, so that 4×8 and 8×4 blocks do not support bidirectional prediction, and 4×4 blocks do not support inter prediction. In JEM's QTBT, these restrictions are removed.

2.1.4 Versatile video coding ( VVC ) ternary tree

As proposed in JVET-D0117, tree types other than quadtrees and binary trees are also supported. In the implementation, two additional ternary tree (TT) partitions are introduced, namely, horizontal and vertical center-side ternary trees, as shown in Figure 6(d) and (e).

Figure 6 shows: (a) quadtree segmentation, (b) vertical binary tree segmentation (c) horizontal binary tree segmentation (d) vertical center-side ternary tree segmentation, and (e) horizontal center-side ternary tree segmentation.

In some implementations, there are two levels of trees: a region tree (quadtree) and a prediction tree (binary or ternary tree). First, the CTU is divided using a region tree (RT). RT leaves can be further divided using prediction trees (PT). The PT lobe can also be further divided with PT until the maximum PT depth is reached. The PT leaf is the basic coding unit. For convenience, it is still called CU. CU cannot be divided further. Both predictions and transformations are applied to CU in the same way as JEM. The entire partitioning structure is called a "multitype tree".

2.1.5 JVET-J0021 segmentation structure in

The tree structure called multi-tree type (MTT) is a generalization of QTBT. In QTBT, as shown in Figure 5, coding tree units (CTUs) are first divided using a quad-tree structure. Then the four-fork leaf nodes are further divided using a binary tree structure.

The basic structure of MTT consists of two types of tree nodes: regional tree (RT) and prediction tree (PT), supporting nine types of divisions, as shown in Figure 7.

Figure 7 illustrates: (a) quadtree segmentation, (b) vertical binary tree segmentation, (c) horizontal binary tree segmentation, (d) vertical ternary tree segmentation, (e) horizontal ternary tree segmentation, (f) horizontal upward non-tree segmentation Symmetric binary tree segmentation, (g) horizontal downward asymmetric binary tree segmentation, (h) vertical left asymmetric binary tree segmentation, and (i) vertical right asymmetric binary tree segmentation.

The zone tree can recursively divide the CTU into square blocks, up to 4x4 sized zone leaf nodes. At each node of the region tree, a prediction tree can be formed from one of three tree types: binary tree (BT), ternary tree (TT), and asymmetric binary tree (ABT). In PT partitioning, quadtree splitting in branches of the prediction tree is prohibited. Like JEM, the luma and chroma trees are separated in the I strip. The signaling methods of RT and PT are shown in Figure 8.

2.2 Inter prediction in HEVC/H.265

Each inter-predicted PU has motion parameters for one or two reference picture lists. Motion parameters include motion vectors and reference picture indexes. The use of one of the two reference picture lists may also be signaled using inter_pred_idc . Motion vectors can be explicitly coded as deltas relative to the predictor, a coding mode called Advanced Motion Vector Prediction (AMVP) mode.

When a CU is encoded in skip mode, a PU is associated with the CU and has no significant residual coefficients, no encoded motion vector increments, or reference picture indexes. Specifies a Merge mode by which the motion parameters of the current PU are obtained from neighboring PUs (including spatial and temporal candidates). Merge mode can be applied to any inter-predicted PU, not just skip mode. An alternative to Merge mode is the explicit transmission of motion parameters, where the motion vectors, the reference picture index corresponding to each reference picture list, and the use of the reference picture list are explicitly signaled in each PU.

A PU is generated from a block of samples when signaling indicates that one of the two reference picture lists is to be used. This is called "one-way prediction". One-way prediction is available for both P-strips and B-strips.

When signaling indicates that two reference picture lists are to be used, a PU is generated from two blocks of samples. This is called "bidirectional prediction". Bidirectional prediction is only available for B slices.

Details of the inter prediction modes specified in HEVC are provided below. The description will start with Merge mode.

2.2.1 Merge model

2.2.1.1Merge Derivation of Pattern Candidates

When predicting a PU using Merge mode, the index pointing to the entry in the Merge candidate list is analyzed from the bitstream and used to retrieve motion information. The structure of this list is specified in the HEVC standard and can be summarized in the following sequence of steps:

Step 1: Initial candidate derivation

Step 1.1: Airspace candidate derivation

Step 1.2: Redundancy Check of Airspace Candidates

Step 1.3: Time domain candidate derivation

Step 2: Append candidate insertions

Step 2.1: Creation of Bidirectional Prediction Candidates

Step 2.2: Insertion of Zero Motion Candidates

These steps are also schematically depicted in Figure 9. For spatial Merge candidate derivation, up to four Merge candidates are selected among candidates located at five different locations. For time-domain Merge candidate derivation, at most one Merge candidate is selected among the two candidates. Since the number of candidates per PU is assumed to be constant at the decoder, additional candidates are generated when the number of candidates does not reach the maximum number of Merge candidates ( MaxNumMergeCand ) signaled in the slice header. Since the number of candidates is constant, the index of the best Merge candidate is encoded using truncated unary binarization (TU). If the size of the CU is equal to 8, all PUs of the current CU share a Merge candidate list, which is the same as the Merge candidate list of the 2N×2N prediction unit.

The operations related to the above steps are described in detail below.

2.2.1.2 Airspace candidate derivation

In the derivation of spatial Merge candidates, up to four Merge candidates are selected among the candidates located at the positions shown in Figure 10. The derivation sequence is A1, B1, B0, A0 and B2. Location B2 is only considered if any PU at location A1, B1, B0, A0 is unavailable (for example, because it belongs to another stripe or slice) or is internally coded. After adding candidates at the A1 position, a redundancy check is performed on the addition of the remaining candidates, which ensures that candidates with the same motion information are excluded from the list, thereby improving coding efficiency. In order to reduce the computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. Instead, only pairs connected by arrows in Figure 11 are considered, and candidates are added to the list only if the corresponding candidate for redundancy check does not have the same motion information. Another source of replicated motion information is the "second PU" associated with a different 2N×2N partition. For example, Figure 12 depicts the second PU in the N×2N and 2N×N cases respectively. When the current PU is divided into N×2N, candidates for the A1 position are not considered for list construction. In some embodiments, adding this candidate may result in two prediction units with the same motion information, which is redundant with only one PU in the coding unit. Likewise, when the current PU is partitioned into 2N×N, location B1 is not considered.

2.2.1.3 time domain Candidate derivation

In this step, only one candidate is added to the list. In particular, in the derivation of this temporal Merge candidate, scaled motion vectors are derived based on the collocated PU with the smallest picture order count POC difference from the current picture in the given reference picture list. The reference picture list used to derive collocated PUs is explicitly signaled in the slice header. The dashed line in Figure 13 shows the acquisition of the scaled motion vector of the temporal Merge candidate, which is scaled from the motion vector of the collocated PU using the POC distances tb and td, where tb is defined as the distance between the reference picture of the current picture and the current picture. POC difference, and td is defined as the POC difference between the reference picture of the collocated picture and the collocated picture. The reference picture index of the temporal Merge candidate is set to zero. The actual handling of scaling is described in the HEVC specification. For the B slice, two motion vectors (one for reference picture list 0 and the other for reference picture list 1) are obtained and combined to make them bidirectional prediction Merge candidates. Illustration showing motion vector scaling for temporal Merge candidates.

In the collocated PU (Y) belonging to the reference frame, the position of the temporal candidate is selected between candidates C ₀ and C ₁ , as shown in Figure 14. If the PU at position C ₀ is unavailable, internally coded, or outside the current CTU, then position C ₁ is used. Otherwise, position C ₀ is used for the derivation of time-domain Merge candidates.

2.2.1.4 Additional candidate insertion

In addition to space-time Merge candidates, there are two additional types of Merge candidates: combined bidirectional prediction Merge candidates and zero Merge candidates. Combined bidirectional prediction Merge candidates are generated using space-time Merge candidates. Combined bi-prediction Merge candidates are only used for B-slices. A combined bidirectional prediction candidate is generated by combining the first reference picture list motion parameters of the initial candidate with the second reference picture list motion parameters of another candidate. If these two tuples provide different motion hypotheses, they will form new bidirectional prediction candidates. Figure 15 shows a case where two candidates from the original list (on the left) are used to create a combined bidirectional prediction Merge candidate added to the final list (on the right) with two of MvL0 and refIdxL0 or MvL1 and refIdxL1 candidate. There are many rules about combining that need to be considered to generate these additional Merge candidates.

Zero motion candidates are inserted to fill the remaining entries in the Merge candidate list, thus reaching the capacity of MaxNumMergeCand. These candidates have zero spatial displacement and a reference picture index that starts at zero and is incremented every time a new zero motion candidate is added to the manifest. The number of reference frames used by these candidates is 1 frame and 2 frames for unidirectional prediction and bidirectional prediction respectively. Finally, no redundancy checks are performed on these candidates.

2.2.1.5 Parallel processing of motion estimation areas

To speed up the encoding process, motion estimation can be performed concurrently, resulting in simultaneous derivation of motion vectors for all prediction units within a given region. Deriving Merge candidates from spatial neighborhoods may interfere with parallel processing because one prediction unit cannot derive motion parameters from neighboring PUs until the associated motion estimation is completed. To ease the trade-off between coding efficiency and processing latency, HEVC defines Motion Estimation Regions (MERs). The size of the MER can be signaled in the picture parameter set using the syntax element "log2_parallel_merge_level_minus2" as described below. When MER is defined, Merge candidates that fall into the same region are marked as unavailable, and are therefore not considered in list construction.

Picture parameters set raw byte sequence payload ( RBSP ) syntax

Common picture parameter set RBSP Grammar pic_paraMeter_set_rbsp( ) { Descriptor pps_pic_paraMeter_set_id ue(v) pps_seq_paraMeter_set_id ue(v) depeNdeNt_slice_segMeNts_eNabled_flag u(1) … ​ pps_scaliNg_list_data_preseNt_flag u(1) if(pps_scaliNg_list_data_preseNt_flag) ​ scaliNg_list_data() ​ lists_ModificatioN_preseNt_flag u(1) log2_parallel_Merge_level_MiNus2 ue(v) slice_segMeNt_header_exteNsioN_preseNt_flag u(1) pps_exteNsioN_preseNt_flag u(1) … ​ rbsp_trailiNg_bits( ) ​ } ​

log2_parallel_Merge_level_MiNus2 plus 2 specifies the value of the variable Log2ParMrgLevel, which is used in the derivation of the Merge mode luminance motion vector specified in clause 8.5.3.2.2, and in the derivation of the spatial Merge candidate specified in clause 8.5.3.2.3. The value of log2_parallel_Merge_level_MiNus2 should be in the range of 0 to CtbLog2SizeY − 2, inclusive.

The variable Log2ParMrgLevel is derived as follows:

Log2ParMrgLevel = log2_parallel_Merge_level_MiNus2 + 2 (7-37)

Note 3 – The value of Log2ParMrgLevel represents the built-in functionality of Merge candidate list parallel inference. For example, when Log2ParMrgLevel is equal to 6, the Merge candidate list for all prediction units (PUs) and coding units (CUs) contained in a 64x64 block can be derived in parallel.

2.2.2 AMVP Motion vector prediction in mode

Motion vector prediction exploits the spatial-temporal correlation of motion vectors with neighboring PUs, which is used for explicit transmission of motion parameters. A motion vector candidate list is first constructed by checking the availability of temporally adjacent PU positions in the upper left, removing redundant candidate positions, and adding zero vectors to make the candidate list constant length. The encoder can then select the best predictor from the candidate list and send the corresponding index indicating the selected candidate. Similar to Merge index signaling, the index of the best motion vector candidate is encoded using a truncated unary. The maximum value to encode in this case is 2 (e.g., Figures 2 to 8). In the following sections, the derivation process of motion vector prediction candidates will be introduced in detail.

2.2.2.1 Derivation of motion vector prediction candidates

Figure 16 summarizes the derivation process of motion vector prediction candidates.

In motion vector prediction, two types of motion vector candidates are considered: spatial motion vector candidates and temporal motion vector candidates. For the derivation of spatial motion vector candidates, two motion vector candidates are ultimately derived based on the motion vectors of each PU located at five different locations shown in Figure 11.

For the derivation of temporal motion vector candidates, one motion vector candidate is selected from two candidates, which are derived based on two different collocated positions. After making the first empty selection list, remove duplicate motion vector candidates from the list. If the number of potential candidates is greater than two, motion vector candidates with a reference picture index greater than 1 in the associated reference picture list are removed from the list. If the number of space-time motion vector candidates is less than two, additional zero motion vector candidates are added to the list.

2.2.2.2 Spatial motion vector candidate

When deriving spatial motion vector candidates, a maximum of two candidates are considered out of the five potential candidates from the PU at the locations depicted in Figure 11, which are the same locations as the motion merge. The derivation sequence for the left side of the current PU is defined as A ₀ , A ₁ , and scaled A ₀ , scaled A ₁ . The derivation sequence on the current PU is defined as B ₀ , B ₁ , B ₂ , scaled B ₀ , scaled B ₁ , scaled B ₂ . Therefore, there are four cases per side that can be used as motion vector candidates, two of which do not require the use of spatial scaling, and two of which use spatial scaling. The four different situations are summarized as follows:

--No space scaling

(1) The same reference picture list, and the same reference picture index (same POC)

(2) Different reference picture lists, but the same reference pictures (same POC)

--Spatial scaling

(3) The same reference picture list, but different reference pictures (different POC)

(4) Different reference picture lists, and different reference pictures (different POC)

Check first for the case of no spatial scaling, then for spatial scaling. Whenever the POC differs between the reference picture of a neighboring PU and the reference picture of the current PU, spatial scaling is considered regardless of the reference picture list. If all PUs of the left candidate are unavailable or internally coded, the above motion vectors are allowed to be scaled to aid in the parallel derivation of the left and upper MV candidates. Otherwise, spatial scaling of the above motion vectors is not allowed.

In the spatial scaling process, the motion vectors of neighboring PUs are scaled in a similar manner to temporal scaling, as shown in Figure 17. The main difference is that the reference picture list and index of the current PU are given as input, and the actual scaling process is the same as the time domain scaling process.

2.2.2.3 time domain motion vector candidate

Except for the derivation of the reference picture index, all derivation processes for temporal Merge candidates are identical to those for spatial motion vector candidates (see e.g. Figure 6). The reference picture index is signaled to the decoder.

2.2.2.4 AMVP signaling of information

For AMVP mode, four parts can be signaled in the bit stream, including prediction direction, reference index, MVD and MV prediction candidate index.

Grammar table: predictionN_uNit( x0, y0, NPbW, NPbH ) { Descriptor if( cu_skip_flag[ x0 ][ y0 ] ) { if(MaxNuMMergeCaNd > 1) Merge_idx [x0][y0] ae(v) } else { /* MODE_INTER */ Merge_flag [x0][y0] ae(v) if( Merge_flag[ x0 ][ y0 ] ) { if(MaxNuMMergeCaNd > 1) Merge_idx [x0][y0] ae(v) } else { if( slice_type == B ) iNter_pred_idc [x0][y0] ae(v) if( iNter_pred_idc[ x0 ][ y0 ] != PRED_L1 ) { if( NuM_ref_idx_l0_active_MiNus1 > 0 ) ref_idx_l0 [x0][y0] ae(v) Mvd_codiNg(x0, y0, 0) Mvp_l0_flag [x0][y0] ae(v) } if( iNter_pred_idc[ x0 ][ y0 ] != PRED_L0 ) { if( NuM_ref_idx_l1_active_MiNus1 > 0 ) ref_idx_l1 [x0][y0] ae(v) if( Mvd_l1_zero_flag && iNter_pred_idc[ x0 ][ y0 ] = = PRED_BI ) { MvdL1[x0][y0][0] = 0 MvdL1[x0][y0][1] = 0 } else Mvd_codiNg(x0, y0, 1) Mvp_l1_flag [x0][y0] ae(v) } } } }

Motion vector difference syntax Mvd_codiNg(x0, y0, refList) { Descriptor abs_Mvd_greater0_flag [ 0 ] ae(v) abs_Mvd_greater0_flag [ 1 ] ae(v) if( abs_Mvd_greater0_flag[ 0 ] ) abs_Mvd_greater1_flag [ 0 ] ae(v) if( abs_Mvd_greater0_flag[ 1 ] ) abs_Mvd_greater1_flag [ 1 ] ae(v) if( abs_Mvd_greater0_flag[ 0 ] ) { if( abs_Mvd_greater1_flag[ 0 ] ) abs_Mvd_MiNus2 [0] ae(v) Mvd_sigN_flag [ 0 ] ae(v) } if( abs_Mvd_greater0_flag[ 1 ] ) { if( abs_Mvd_greater1_flag[ 1 ] ) abs_Mvd_MiNus2 [1] ae(v) Mvd_sigN_flag [1] ae(v) } }

2.3 joint exploration model ( JEM ) new inter-frame prediction method

2.3.1 Based on sub CU motion vector prediction

In JEM with QTBT, each CU can have at most one set of motion parameters for each prediction direction. By dividing a large CU into sub-CUs and deriving the motion information of all sub-CUs of the large CU, two sub-CU-level motion vector prediction methods are considered in the encoder. The Alternative Temporal Motion Vector Prediction (ATMVP) method allows each CU to obtain multiple sets of motion information from multiple blocks smaller than the current CU in the collocated reference picture. In the space-time motion vector prediction (STMVP) method, the motion vector of a sub-CU is recursively derived by utilizing a temporal motion vector predictor and a spatial adjacent motion vector.

In order to maintain a more accurate motion field for sub-CU motion prediction, motion compression of reference frames is currently disabled.

2.3.1.1 Optional temporal motion vector prediction

In the alternative temporal motion vector prediction (ATMVP) method, the motion vector temporal motion vector prediction (TMVP) is modified by extracting multiple sets of motion information (including motion vectors and reference indices) from blocks smaller than the current CU. As shown in Figure 18, the sub-CU is a square N×N block (default N is set to 4).

ATMVP predicts the motion vectors of sub-CUs within a CU in two steps. The first step is to identify the corresponding blocks in the reference image using so-called temporal vectors. The reference picture is called the motion source picture. The second step is to divide the current CU into sub-CUs and obtain the motion vector and the reference index of each sub-CU from the block corresponding to each sub-CU, as shown in Figure 18.

In the first step, the reference picture and the corresponding block are determined from the motion information of the spatial neighboring blocks of the current CU. In order to avoid repeated scanning processing of adjacent blocks, the first Merge candidate in the Merge candidate list of the current CU is used. The first available motion vector and its associated reference index are set to the temporal vector and the index of the motion source picture. In this way, in ATMVP, the corresponding block can be identified more accurately than in TMVP, where the corresponding block (sometimes called a collocated block) is always located in the lower right corner or center position relative to the current CU. In one example, if the first Merge candidate is from the left neighboring block (ie, A ₁ in Figure 19 ), the relevant MV and reference picture are used to identify the source block and source picture.

Figure 19 shows an example of identification of source blocks and source pictures.

In the second step, the corresponding block of the sub-CU is identified through the temporal vector in the motion source picture by adding the temporal vector to the coordinates of the current CU. For each sub-CU, the motion information of its corresponding block (the minimum motion grid covering the center sample) is used to derive the motion information of the sub-CU. After identifying the motion information corresponding to the N×N block, it is converted into the motion vector and reference index of the current sub-CU, which is the same as the TMVP method of HEVC, in which motion scaling and other processing are applied. For example, the decoder checks whether the low-latency condition is met (i.e., the POC of all reference pictures of the current picture is smaller than the POC of the current picture), and may use the motion vector MVx (the motion vector corresponding to the reference picture list CU predicts motion vector MVy (X equals 0 or 1 and Y equals 1−X).

2.3.1.2 Space-time motion vector prediction

In this method, the motion vectors of sub-CUs are derived recursively in raster scan order. Figure 20 illustrates this concept. Consider an 8×8 CU, which contains four 4×4 sub-CUs A, B, C and D. The adjacent 4×4 blocks in the current frame are labeled a, b, c, and d.

The motion derivation of sub-CU A starts by identifying its two spatial neighbors. The first neighbor is the N×N block above sub-CU A (block c). If block c is unavailable or internally encoded, other N×N blocks above sub-CU A are checked (from left to right, starting at block c). The second neighbor is a block to the left of sub-CU A (block b). If block b is not available or is internally coded, check other blocks to the left of sub-CU A (from top to bottom, starting from block b). The motion information obtained from neighboring blocks for each manifest is scaled to the first reference frame of the given manifest. Next, the temporal motion vector prediction (TMVP) of sub-block A is derived according to the same procedure as TMVP specified in HEVC. Extract the motion information of the collocated block at position D and scale accordingly. Finally, after retrieving and scaling the motion information, all available motion vectors (up to 3) are averaged separately for each reference list. Specify the average motion vector as the motion vector of the current sub-CU.

Figure 20 shows an example of one CU with four sub-blocks (A-D) and their neighboring blocks (a-d).

2.3.1.3 son CU Motion prediction mode signaling notification

Sub-CU mode is enabled as an additional Merge candidate mode, and no additional syntax elements are required to signal this mode. Two additional Merge candidates are added to each CU's Merge candidate list to represent ATMVP mode and STMVP mode. If the sequence parameter set indicates that ATMVP and STMVP are enabled, up to seven Merge candidates are used. The encoding logic of additional Merge candidates is the same as that of Merge candidates in HM, which means that for each CU in a P-slice or B-slice, two additional RD checks are required for the two additional Merge candidates.

In JEM, all bin files indexed by Merge are contextually encoded by CABAC. However, in HEVC, only the first bin file is context encoded, and the remaining biN files are context bypass encoded.

2.3.2 Self-adjusting motion vector difference resolution

In HEVC, when use_integer_mv_flag is equal to 0 in the slice header, the motion vector difference (MVD) (between the PU's motion vector and the predicted motion vector) is signaled in units of quarter luma samples. In JEM, Local Self-Adjusting Motion Vector Resolution (LAMVR) is introduced. In JEM, MVD can be encoded in units of quarter luma samples, integer luma samples, or quad luma samples. MVD resolution is controlled at the coding unit (CU) level, and the MVD resolution flag is conditionally signaled for each CU that has at least one non-zero MVD component.

For CUs with at least one non-zero MVD component, the first flag will signal whether quarter-luma sample MV precision is used in the CU. While the first flag (equal to 1) indicates not to use quarter luma sample MV precision, the other flag signals whether to use integer luma sample MV precision or quad luma sample MV precision.

When the first MVD resolution flag of the CU is zero or is not encoded for the CU (meaning all MVDs in the CU are zero), the CU uses quarter-luma sample MV resolution. When a CU uses integer luma sample MV precision or quad luma sample MV precision, the MVPs in the CU's AMVP candidate list will be rounded to the corresponding precision.

In the encoder, a CU-level RD check is used to determine which MVD resolution will be used for the CU. That is, three CU-level RD checks are performed for each MVD resolution. To speed up the encoder, the following encoding scheme is applied in JEM.

During RD inspection of a CU with normal quarter luma sampling MVD resolution, motion information for the current CU (integer luma sampling resolution) is stored. When performing RD inspection on the same CU with integer luma samples and 4 luma sample MVD resolution, the stored motion information (after rounding) is used as a starting point for further small-scale motion vector refinement, thus making the time-consuming The motion estimation process is not repeated three times.

Conditionally calls the RD check for CUs with 4 luminance sample MVD resolution. For a CU, when the RD check cost for the integer luma sample MVD resolution is much greater than the RD check cost for the quarter luma sample MVD resolution, the RD check for the CU's 4-luma sample MVD resolution will be skipped.

2.3.3 Pattern matching motion vector derivation

Pattern Matching Motion Vector Derivation (PMMVD) mode is a special Merge mode based on Frame Rate Up Conversion (FRUC) technology. In this mode, the motion information of the block is not signaled but derived on the decoder side.

For a CU, the FRUC flag is signaled when its Merge flag is true. When the FRUC flag is false, the Merge search is signaled and regular Merge mode is used. When the FRUC flag is true, another FRUC mode flag is signaled to indicate which mode (bilateral matching or template matching) will be used to derive motion information for this block.

On the encoder side, it is decided whether to use FRUC Merge mode for the CU based on the RD cost selection made on the normal Merge candidates. That is, two matching modes (bilateral matching and template matching) of CU are checked by using RD cost selection. The mode that resulted in the lowest cost was further compared with other CU modes. If the FRUC match mode is the most efficient mode, then for CU the FRUC flag is set to true and the associated match mode is used.

The motion derivation process in FRUC Merge mode has two steps: first perform CU-level motion search, and then perform sub-CU-level motion optimization. At the CU level, the initial motion vector of the entire CU is derived based on bilateral matching or template matching. First, an MV candidate list is generated, and the candidate that results in the lowest matching cost is selected as the starting point for further optimization at the CU level. Then a local search based on bilateral matching or template matching is performed near the starting point, and the MV result of the minimum matching cost is used as the MV value of the entire CU. Then, taking the derived CU motion vector as the starting point, the motion information is further refined at the sub-CU level.

For example, for W×H CU motion information derivation, the following derivation process is performed. In the first stage, the MV of the entire W × H CU is derived. In the second stage, the CU is further divided into M×M sub-CUs. The value of M is calculated according to (16), and D is the predefined division depth, which is set to 3 by default in JEM. Then the MV value of each sub-CU is derived.

(3)

As shown in Figure 21, by finding the closest match between two blocks in two different reference pictures along the motion trajectory of the current CU, bilateral matching is used to derive the motion information of the current CU. Under the assumption of continuous motion trajectories, the motion vectors MV0 and MV1 pointing to the two reference blocks are proportional to the time distance between the current picture and the two reference pictures (i.e., TD0 and TD1). As a special case, when the current picture is temporarily located between When there are two reference pictures and the time distance between the current picture and the two reference pictures is the same, bilateral matching becomes a two-way MV based on mirroring.

As shown in Figure 22, template matching is used by finding the closest match between the template in the current picture (the top and/or left adjacent block of the current CU) and the block in the reference picture (same size as the template). Derive the current CU's motion information. In addition to the above-mentioned FRUC Merge mode, template matching is also applied to AMVP mode. In JEM, as in HEVC, there are two candidates for AMVP. Using the template matching method, new candidates are derived. If a newly derived candidate by template matching is different from the first existing AMVP candidate, it is inserted at the very beginning of the AMVP candidate list, and the list size is then set to 2 (i.e., the second existing AMVP candidate is removed). When applied to AMVP mode, only CU-level searches are applied.

2.3.3.1 CU level MV candidate set

The CU-level MV candidate set includes:

(i) Original AMVP candidate, if the current CU is in AMVP mode,

(ii) all Merge candidates,

(iii) Interpolate several MVs in the MV field.

(iv) Top and left adjacent motion vectors

When using bilateral matching, each valid MV of the Merge candidate is used as input to generate pairs of MVs that are assumed to be bilaterally matched. For example, a valid MV of the Merge candidate at reference list A is (MVa, ref _a ). The reference picture ref _b of its paired bilateral MV is then found in another reference list B, so that ref _a and ref _b are temporally located on different sides of the current picture. If reference ref _b in reference list B is not available, reference ref _b is determined to be a reference different from reference ref _a , and its temporal distance to the current picture is the minimum distance in list B. After the reference ref _b is determined, MVb is derived by scaling the MVa based on the time distance between the current picture and the reference ref _a and the reference ref _b .

Four MVs from the interpolated MV field are also added to the CU-level candidate list. More specifically, the interpolated MVs at the positions (0, 0), (W/2, 0), (0, H/2) and (W/2, H/2) of the current CU are added.

When applying FRUC in AMVP mode, the original AMVP candidates are also added to the CU-level MV candidate set.

At the CU level, up to 15 MVs for AMVP CUs and up to 13 MVs for Merge CUs can be added to the candidate list.

2.3.3.2 son CU level MV candidate set

MV candidates set at the sub-CU level include:

(i) Search the determined MV from the CU level,

(ii) Top, left, upper left and upper right adjacent MVs,

(iii) a scaled version of the juxtaposed MV from the reference image,

(iv) Up to 4 ATMVP candidates,

(v) Up to 4 STMVP candidates.

The scaled MV from the reference image is derived as follows. All reference images in both lists are traversed. The MV at the collocated position of the sub-CU in the reference picture is scaled to the reference of the starting CU-level MV.

ATMVP and STMVP candidates are limited to the top four. At the sub-CU level, up to 17 MVs are added to the candidate list.

2.3.3.3 interpolation MV field generation

Before encoding the frame, an interpolated motion field of the entire picture is generated based on unidirectional ME. This sports field can then be subsequently used as an MV candidate at the CU level or sub-CU level.

First, the motion field of each reference picture in the two reference lists is traversed at the 4 × 4 block level. For each 4×4 block, if the motion associated with the block passes through the 4×4 block in the current picture (as shown in Figure 23), and the block is not assigned any interpolated motion, then according to the temporal distance TD0 and TD1 Scale the motion of the reference block to the current picture (same as MV scaling of TMVP in HEVC), and assign that scaled motion to the block in the current frame. If no scaled MV is assigned to a 4×4 block, the block's motion is marked as unavailable in the interpolated motion field.

2.3.3.4 interpolation matching cost

When motion vectors point to fractional sampling locations, motion compensated interpolation is required. In order to reduce complexity, bilinear interpolation is used instead of conventional 8-tap HEVC interpolation for both bilateral matching and template matching.

The calculation of matching costs is a little different at different steps. When selecting candidates from a CU-level candidate set, the matching cost is the sum of absolute differences (SAD) for bilateral matching or exemplar matching. After determining the starting MV, the matching cost C of bilateral matching at the sub-CU level is calculated as follows:

(4)

Here, w is the weight coefficient, which is empirically set to 4. MV and Indicate the current MV and starting MV respectively. Still using SAD as the matching cost for pattern matching at the sub-CU level search.

In FRUC mode, MV is derived by using only luminance samples. The derived motion will be used for MC inter prediction of luma and chroma. After determining the MV, a final MC is performed using an 8-tap interpolation filter for luma and a 4-tap interpolation filter for chroma.

2.3.3.5 MV refine

MV refinement is a pattern-based MV search, using bilateral cost or template matching cost as the criterion. In JEM, two search modes are supported—unrestricted center-biased diamond search (UCBDS) and self-adjusting cross search, which perform MV refinement at the CU level and sub-CU level respectively. For both CU-level and sub-CU-level MV refinement, MV is searched directly at one-quarter luminance sample precision, followed by one-eighth luminance sample MV refinement. Set the search range for MV refinement of CU and sub-CU steps to 8 luma samples.

2.3.3.6 Template matching FRUC Merge Selection of prediction direction in mode

In bilateral Merge mode, bidirectional prediction is always applied because the motion information of the CU is obtained in two different reference pictures based on the closest match between the two blocks on the current CU motion trajectory. Template matching Merge mode does not have this limitation. In template matching Merge mode, the encoder can choose for the CU from unidirectional prediction in List 0, unidirectional prediction in List 1, or bidirectional prediction. This selection is based on the following template matching costs:

If costBi>=factor*min（cost0,cost1）

Then use bidirectional prediction;

Otherwise, if cost0>=cost1

Then use the one-way prediction in list 0;

Otherwise,

Use the one-way prediction from Listing 1;

Where cost0 is the SAD matching the list 0 template, cost1 is the SAD matching the list 2 template, and costBi is the SAD matching the bidirectional prediction template. The value of factor is equal to 1.25, which means that the selection process is shifted towards bidirectional prediction. Inter prediction direction selection can be applied only to CU-level template matching processing.

2.3.4 Decoder side motion vector refinement

In the bidirectional prediction operation, for prediction of one block region, two prediction blocks formed by the motion vector (MV) of list 0 and the MV of list 1 respectively are combined to form a single prediction signal. In the decoder-side motion vector refinement (DMVR) method, the two motion vectors of bidirectional prediction are further refined through bilateral template matching processing. Bilateral template matching applied in the decoder is used to perform a distortion-based search between bilateral templates and reconstructed samples in reference pictures to obtain refined MVs without transmitting additional motion information.

In DMVR, the bilateral template is generated as a weighted combination (i.e., average) of two prediction blocks, where the two prediction blocks are from the initial MV0 of list 0 and the MV1 of list 1. The template matching operation consists of computing a cost metric between the generated template and the sample region in the reference picture (around the initial prediction block). For each of the two reference pictures, the MV that yields the minimum template cost is considered as the updated MV of the list, replacing the original MV. In JEM, nine MV candidates are searched for each list. The nine MV candidates include the original MV and eight peripheral MVs that have an offset of one luminance sample from the original MV in the horizontal or vertical direction or both. Finally, the two new MVs (i.e., MV0′ and MV1′) shown in Figure 24 are used to generate the final bidirectional prediction results. The sum of absolute differences (SAD) was used as the cost measure.

Without transmitting additional syntax elements, DMVR is applied to bidirectionally predicted Merge mode, where one MV comes from a past reference picture and the other MV comes from a future reference picture. In JEM, DMVR is not applied when LIC, Affine Motion, FRUC, or Sub-CU Merge candidates are enabled for a CU.

2.3.5 local illumination compensation

Local illumination compensation (IC) is based on a linear model for illumination changes, using a scaling factor a and an offset b. And local illumination compensation is self-adjustedly enabled or disabled for each inter-mode encoding coding unit (CU).

When IC is applied to a CU, the least square error method is adopted to derive the parameters a and b by using the adjacent sample points of the current CU and their corresponding reference sample points. More specifically, as shown in Figure 25, subsampled (2:1 subsampled) adjacent samples of the CU and corresponding samples (identified by the motion information of the current CU or sub-CU) in the reference picture are used. IC parameters are derived and applied separately to each prediction direction.

When a CU is encoded in Merge mode, the IC flag is copied from neighboring blocks in a similar manner to the motion information copy in Merge mode; otherwise, the IC flag is signaled to the CU to indicate whether LIC is applicable.

When IC is enabled for a picture, an additional CU level RD check is required to determine whether to apply LIC to the CU. When IC is enabled for CU, Mean-Removed Sum of Absolute Diffefference (MR-SAD) and Mean-Removed Sum of Absolute Diffefference (MR-SAD) are used for integer pixel motion search and fractional pixel motion search respectively. Removed Sum of Absolute Hadamard-Transformed Difference, MR-SATD) instead of SAD and SATD.

To reduce coding complexity, the following coding scheme is applied in JEM. When there are no significant lighting changes between the current picture and its reference picture, IC is disabled for all pictures. To identify this situation, a histogram of the current picture and each reference picture of the current picture is calculated at the encoder. If the bar chart difference between the current picture and each reference picture of the current picture is less than a given threshold, IC is disabled for the current picture; otherwise, IC is enabled for the current picture.

2.3.6 with two-way match refinement Merge/ skip mode

The Merge candidate list is first constructed by inserting the motion vectors and reference indices of spatially adjacent and temporally adjacent blocks into the candidate list using redundancy checking until the number of available candidates reaches the maximum candidate size of 19. By inserting spatial candidates (Figure 26), temporal candidates, affine candidates, Advanced Temporal MVP (ATMVP) candidates, spatiotemporal MVP (Spatial Temporal, STMVP) candidates and HEVC candidates according to the predefined insertion order. Append candidates (combined candidates and zero candidates) to construct a Merge candidate list for Merge/Skip mode:

(1) Spatial candidates for blocks 1-4

(2) Extrapolated affine candidates for blocks 1-4

(3)ATMVP

(4) STMVP

(5) Virtual affine candidates

(6) Spatial Candidates (Block 5) (only used when the number of available candidates is less than 6)

(7) Extrapolate affine candidates (Block 5)

(8) Temporal candidates (as derived in HEVC)

(9) Non-contiguous spatial candidates, followed by extrapolated affine candidates (blocks 6 to 49)

(10) Combination candidates

(11) Zero candidate

Note that in addition to STMVP and affine, the IC flag is also inherited from the Merge candidate. Furthermore, for the first four spatial candidates, bidirectional prediction candidates are inserted before candidates with unidirectional prediction.

2.3.7 JVET-K0161

In this proposal, sub-block STMVP is not proposed as a space-time Merge mode. The proposed method uses co-located blocks, which is the same as HEVC/JEM (only 1 picture, no time vector here). The proposed method also examines the upper and left spatial positions, which are regulated in this proposal. Specifically, to check adjacent inter prediction information, up to two positions are checked for each upper and left side. The exact location is shown in Figure 27.

Afar: (nPbW * 5 / 2, -1), Amid (nPbW / 2, -1) (Note: the offset of the upper space block above the current block)

Lfar: (-1, nPbH * 5 / 2), Lmid (-1, nPbH/2) (Note: the offset of the left space block above the current block)

The average of the motion vectors of the upper block, left block and time block is calculated in the same way as the BMS software implementation. If 3 reference inter prediction blocks are available.

mvLX[0] = ((mvLX_A[0] + mvLX_L[0] + mvLX_C[0]) * 43) / 128

mvLX[1] = ((mvLX_A[1] + mvLX_L[1] + mvLX_C[1]) * 43) / 128

If only two or one inter prediction block is available, the average of the two is used or only one mv is used.

2.3.8 JVT-K0135

In order to generate smooth and subtle motion fields, Figure 28 gives a brief description of the planar motion vector prediction process.

Planar motion vector prediction is achieved by averaging horizontal and vertical linear interpolation on a 4×4 block basis as follows.

W and H represent the width and height of the block. (x, y) is the coordinate of the current sub-block relative to the upper-left sub-block. All distances are expressed as pixel distance divided by 4. is the motion vector of the current sub-block.

Level prediction at position (x, y) and vertical forecasting The calculation is as follows:

in and are the motion vectors of the 4x4 blocks to the left and right of the current block. and are the motion vectors of the 4x4 blocks above and below the current block.

The reference motion information of the left column and upper row neighboring blocks is derived from the spatial neighboring blocks of the current block.

The reference motion information for adjacent blocks in the right column and bottom row is derived as follows.

Derive the motion information of the temporally adjacent 4×4 blocks in the lower right corner

Using the derived motion information of the adjacent 4×4 block in the lower right and the motion information of the adjacent 4×4 block in the upper right, the motion vector of the adjacent 4×4 block in the right column is calculated, as described in Equation K1.

The motion vector of the bottom row adjacent 4×4 block is calculated using the derived motion information of the lower right adjacent 4×4 block and the motion information of the lower left adjacent 4×4 block, as described in Equation K2.

​ R(W,y) = ((Hy-1) AR + (y+1) BR)/H Formula K1 ​ B(x,H) = ((Wx-1) BL+ (x+1) BR)/W Formula K2

where AR is the motion vector of the spatially adjacent 4×4 block in the upper right, BR is the motion vector of the temporally adjacent 4×4 block in the lower right, and BL is the motion vector of the spatially adjacent 4×4 block in the lower left.

For each list the motion information obtained from adjacent blocks is scaled to the first reference picture for the given list.

3. Examples of problems solved by embodiments disclosed herein

The inventors have previously proposed a lookup table-based motion vector prediction technique, which uses one or more lookup tables storing at least one motion candidate to predict motion information of a block, which can be implemented in various embodiments to provide better performance. High coding efficiency video encoding. Each LUT may include one or more motion candidates, and each motion candidate is associated with corresponding motion information. The motion information of the motion candidate may include prediction direction, reference index/picture, motion vector, LIC flag, affine flag, motion vector difference (MVD) accuracy and/or MVD value. Motion information may also include block location information to indicate where the motion information comes from.

LUT-based motion vector prediction based on the disclosed techniques can enhance existing and future video coding standards, which is illustrated in the examples described below for various implementations. Because LUT allows the encoding/decoding process to be performed based on historical information (eg, blocks that have been processed), LUT-based motion vector prediction may also be called a history-based motion vector prediction (HMVP) method. In LUT-based motion vector prediction methods, one or more tables with motion information from previously encoded blocks are maintained during the encoding/decoding process. These motion candidates stored in the LUT are named HMVP candidates. During encoding/decoding of a block, associated motion information in the LUT can be added to a motion candidate list (eg, Merge/AMVP candidate list), and after encoding/decoding of a block, the LUT can be updated. The updated LUT is then used to encode subsequent blocks. That is, the motion candidates in the LUT are updated based on the encoding/decoding order of the blocks. The following examples should be considered as examples to explain the general concept. These examples should not be interpreted in a narrow way. Furthermore, these instances can be combined in any way.

Some embodiments may use one or more lookup tables storing at least one motion candidate to predict motion information for a block. Embodiments may use motion candidates to indicate a set of motion information stored in a lookup table. For traditional AMVP or Merge modes, embodiments may use AMVP or Merge candidates to store motion information.

Although current LUT-based motion vector prediction techniques overcome the shortcomings of HEVC by using historical data, only information from spatially adjacent blocks is considered.

When using motion candidates from a LUT for AMVP or Merge list building processes, inherit it directly without making any changes.

JVET-K0161 is designed to benefit encoding performance. However, it requires additional derivation of TMVP, which increases computational complexity and memory bandwidth.

4. some examples

The following examples should be considered as examples to explain the general concept. These examples should not be interpreted in a narrow way. Furthermore, these instances can be combined in any way.

Some embodiments using currently disclosed techniques may jointly use motion candidates from LUTs and motion information from temporal neighboring blocks. Additionally, complexity reduction of JVET-K0161 is proposed.

Exploiting from LUT Candidates for sports

1. Propose to construct new AMVP/Merge candidates by utilizing motion candidates from LUT.

a. In one example, new candidates can be derived by adding/subtracting an offset (or offsets) to the motion vector of the motion candidate from the LUT.

b. In one example, new candidates can be derived by averaging the motion vectors of selected motion candidates from the LUT.

i. In one embodiment, averaging can be achieved approximately without division operations. For example, MVa, MVb and MVc can be averaged as (MVa+MVb+MVc)× /2 ^N or (MVa+MVb+MVc)× / ^2N . For example, when N = 7, the average value is (MVa+MVb+MVc) ×42/128 or (MVa+MVb+MVc) ×43/128. Note that precomputing or and store it in the lookup table.

ii. In one example, only motion vectors with the same reference picture (in both prediction directions) are selected.

iii. In one example, the reference picture in each prediction direction is predetermined, and if necessary, the motion vector is scaled to the predetermined reference picture.

1. In one example, the first entry in the reference picture list X (X = 0 or 1) is selected as the reference picture.

2. Alternatively, for each prediction direction, select the most frequently used reference picture in the LUT as the reference picture.

c. In one example, for each prediction direction, first select a motion vector with the same reference picture as the predetermined reference picture, and then select other motion vectors.

2. Propose to construct new AMVP/Merge candidates as a function of one or more motion candidates from LUT and motion information from temporal adjacent blocks.

a. In one example, similar to STMVP or JVET-K0161, new candidates can be derived by averaging the motion candidates from LUT and TMVP.

b. In one example, the above blocks (eg, Amid and Afar in Figure 27) can be replaced by one or more candidates from the LUT. Alternatively, furthermore, other procedures can remain unchanged, as already implemented in JVET-K0161.

3. Propose to construct a new AMVP as a function of one or more motion candidates from the LUT, AMVP/Merge candidates from spatial adjacent blocks and/or spatially non-immediate adjacent blocks, and motion information from temporal blocks /Merge candidate.

a. In one example, one or more of the above blocks (eg, Amid and Afar in Figure 27) can be replaced by candidates from the LUT. Alternatively, furthermore, other procedures can remain unchanged, as already implemented in JVET-K0161.

b. In one example, one or more of the left blocks (e.g., Amid and Afar in Figure 27) can be replaced by candidates from the LUT. Alternatively, furthermore, other procedures can remain unchanged, as already implemented in JVET-K0161.

4. It is proposed that when a block's motion information is inserted into a LUT, whether existing entries in the LUT are pruned can depend on the coding mode of the block.

a. In one example, if a block is encoded in Merge mode, no pruning is performed.

b. In one example, if the block is encoded in AMVP mode, no pruning is performed.

c. In one example, if the block is encoded in AMVP/Merge mode, only the latest M entries of the LUT are pruned.

d. In one example, pruning is always disabled when encoding blocks in sub-block mode (e.g., affine or ATMVP).

5. Propose adding motion information from time blocks to the LUT.

a. In one example, motion information can come from co-bit blocks.

b. In one example, the motion information can come from one or more patches from different reference pictures.

and STMVP Related

1. It is proposed to always use spatial Merge candidates to derive new Merge candidates, regardless of TMVP candidates.

a. In one example, the average of two motion Merge candidates can be used.

b. In one example, new candidates can be derived using a combination of spatial Merge candidates and motion candidates from LTU.

2. Propose that STMVP candidates can be derived using non-immediately adjacent blocks (which are not right or left adjacent blocks).

a. In one example, the upper block used for STMVP candidate derivation remains unchanged, while the left block used changes from adjacent blocks to non-immediate blocks.

b. In one example, the left block used for STMVP candidate derivation remains unchanged, while the upper block used changes from adjacent blocks to non-immediate blocks.

c. In one example, new candidates can be derived using candidates from non-immediate blocks jointly with motion candidates from the LUT.

3. Proposed to always use spatial Merge candidates to derive new Merge candidates, regardless of TMVP candidates.

a. In one example, the average of two motion Merge candidates can be used.

b. Alternatively, the average of two, three or more MVs from different positions adjacent or not adjacent to the current block can be utilized.

i. In one embodiment, the MV can only be obtained from the location in the current LCU (also called CTU).

ii. In one embodiment, the MV can only be obtained from the position in the current LCU row.

iii. In one embodiment, the MV can only be obtained from the position in or next to the current LCU row. An example is shown in Figure 29. Blocks A, B, C, E, E and F are next to the current LCU row.

iv. In one embodiment, the MV can only be obtained from a position in the current LCU row or next to the current LCU row but not to the left of the upper left adjacent block. An example is shown in Figure 29. Block T is the upper left adjacent block. Blocks B, C, E, E, and F are next to the current LCU row, but not to the left of the adjacent block in the upper left corner.

c. In one embodiment, spatial Merge candidates can be used in conjunction with operations from LTU

moving candidates to derive new candidates

4. It is proposed that the MV of the BR block used for plane motion prediction in Figure 28 is not obtained from temporal MV prediction, but from an entry of the LUT.

5. It is proposed that motion candidates from LUTs can be used jointly with other types of Merge/AMVP candidates (e.g., spatial Merge/AMVP candidates, temporal Merge/AMVP candidates, default motion candidates) to derive new candidates.

In various implementations of this example and other examples disclosed in this patent document, pruning may include: a) comparing the motion information to an existing entry for uniqueness, and b) if unique, adding the motion information to the inventory, or c) If not unique, either c1) do not add sports information, or c2) add sports information and delete matching existing entries. In some implementations, the pruning operation is not called when adding motion candidates from the table to the candidate list.

30 is a schematic diagram illustrating an example of the structure of a computer system or other control device 3000 that may be used to implement various portions of the disclosed technology. In Figure 30, computer system 3000 includes one or more processors 3005 and memory 3010 connected by internal connections 3025. Internal connection 3025 may represent any one or more individual physical busses, point-to-point connections, or both connected by an appropriate bridge, adapter, or controller. Thus, internal connections 3025 may include, for example, a system bus, a Peripheral Component Connector (PCI) bus, a HyperTransport or Industry Standard Architecture (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB) , IIC (I2C) bus or Institute of Electrical and Electronics Engineers (IEEE) Standard 674 bus (sometimes called "FireWire").

Processor 3005 may include a central processing unit (CPU) to control, for example, the overall operation of the host computer. In some embodiments, processor 3005 does this by executing software or firmware stored in memory 3010 . Processor 3005 may be or may include one or more programmable general or special purpose microprocessors, digital signal processors (DSPs), programmable controllers, application specific integrated circuits (ASICs), programmable logic devices ( PLD), etc., or a combination of these devices.

Memory 3010 may be or include the main memory of the computer system. Memory 3010 represents any suitable form of random access memory (RAM), read only memory (ROM), flash memory, etc., or a combination of these devices. In use, memory 3010 may contain, among other things, a set of machine instructions that, when executed by processor 3005, cause processor 3005 to perform operations to implement embodiments of the disclosed technology.

Also connected to processor 3005 via internal connection 3025 is an (optional) network interface card 3015. Network interface card 3015 provides computer system 3000 with the ability to communicate with remote devices, such as storage clients and/or other storage servers, and may be, for example, an Ethernet adapter or a Fiber Channel adapter.

31 illustrates a block diagram of an example embodiment of a mobile device 3100 that may be used to implement various portions of the disclosed technology. Mobile device 3100 may be a laptop, smartphone, tablet, video camera, or other device capable of processing video. Mobile device 3100 includes a processor or controller 3101 to process data, and a memory 3102 in communication with processor 3101 to store and/or buffer data. For example, processor 3101 may include a central processing unit (CPU) or a microcontroller unit (MCU). In some implementations, processor 3101 may include a field programmable gate array (FPGA). In some implementations, the mobile device 3100 includes or communicates with a graphics processing unit (GPU), a video processing unit (VPU), and/or a wireless communication unit to implement various visual and/or communication data processing functions of the smartphone device. For example, memory 3102 may include and store processor-executable code that, when executed by processor 3101, configures mobile device 3100 to perform various operations, such as receiving information, commands, and/or information, processing information and information, and Send or provide processed information/data to another data device, such as an actuator or external display. In order to support various functions of the mobile device 3100, the memory 3102 can store information and data, such as instructions, software, values, images, and other data processed or referenced by the processor 3101. For example, various types of random access memory (RAM) devices, read only memory (ROM) devices, flash memory devices, and other suitable storage media may be used to implement the storage function of the memory 3102. In some implementations, mobile device 3100 includes an input/output (I/O) unit 3103 to interface processor 3101 and/or memory 3102 with other modules, units, or devices. For example, I/O unit 3103 can interface with processor 3101 and memory 3102 to utilize various wireless interfaces compatible with typical data communications standards, such as between one or more computers and user devices in the cloud. between. In some implementations, mobile device 3100 may interface with other devices using wired connections through I/O unit 3103 . Mobile device 3100 may also interface with other external interfaces (such as data storage) and/or visual or audio display device 3104 to retrieve and transmit information that may be processed by a processor, stored by memory, or output by display device 3104 or an external device. The data and information displayed on. For example, display device 3104 may display a video frame including a block (CU, PU, or TU) for which intra block copying is applied based on whether the block was encoded using a motion compensation algorithm in accordance with the disclosed techniques.

In some embodiments, a video decoder device that can implement methods of sub-block based prediction as described herein can be used for video decoding.

In some embodiments, the video decoding method may be implemented using a decoding device implemented on the hardware platform as shown in FIG. 30 and FIG. 31 .

Various embodiments and techniques disclosed in this document may be described in the following list of examples.

32 is a flow diagram of an example method 3200 for video processing in accordance with currently disclosed technology. Method 3200 includes, at operation 3202, determining new candidates for video processing by averaging two or more selected motion candidates. Method 3200 includes, at operation 3204, adding the new candidate to a candidate list. Method 3200 includes, at operation 3206, performing a conversion between a first video block of the video and a bitstream representation of the video using the determined new candidate in the candidate list.

In some embodiments, the candidate list is a Merge candidate list, and the determined new candidates are Merge candidates.

In some embodiments, the Merge candidate list is an inter prediction Merge candidate list or an intra block copy prediction Merge candidate list.

In some embodiments, the one or more tables include motion candidates derived from previously processed video blocks in the video material that were processed before the first video block.

In some embodiments, there are no spatial candidates and no temporal candidates available in the candidate list.

In some embodiments, the selected motion candidates are from one or more tables.

In some embodiments, the averaging is performed without division operations.

In some embodiments, the averaging is achieved by multiplying the sum of the motion vectors of the selected motion candidates by a scaling factor.

In some embodiments, the horizontal components of the motion vectors of the selected motion candidates are averaged to derive the horizontal components of the new candidate.

In some embodiments, the vertical components of the motion vectors of the selected motion candidates are averaged to derive the vertical components of the new candidate.

In some embodiments, the scaling factors are pre-computed and stored in a lookup table.

In some embodiments, only motion vectors with the same reference picture are selected.

In some embodiments, only motion vectors having the same reference picture in both prediction directions are selected.

In some embodiments, the target reference picture in each prediction direction is predetermined, and the motion vector is scaled to the predetermined reference picture.

In some embodiments, the first entry in the reference picture list X is selected as the target reference picture for the reference picture list, with X being 0 or 1.

In some embodiments, for each prediction direction, the most commonly used reference picture in the table is selected as the target reference picture.

In some embodiments, for each prediction direction, a motion vector having the same reference picture as the predetermined target reference picture is first selected, and then other motion vectors are selected.

In some embodiments, motion candidates from the table are associated with motion information including at least one of the following: prediction direction, reference picture index, motion vector value, intensity compensation flag, affine flag, motion vector difference accuracy, or Motion vector difference.

In some embodiments, method 3200 further includes updating one or more tables based on the transformation.

In some embodiments, the updating of the one or more tables includes updating the one or more tables based on the motion information of the first video block of the video after performing the conversion.

In some embodiments, method 3200 further includes performing a conversion between subsequent video blocks of the video and a bitstream representation of the video based on the updated table.

In some embodiments, the conversion includes an encoding process and/or a decoding process.

In some embodiments, a video encoding device may perform method 2900 and other methods described herein during reconstruction of video for subsequent videos.

In some embodiments, an apparatus in a video system may include a processor configured to perform the methods described herein.

In some embodiments, the described methods may be embodied as computer-executable code stored on a computer-readable program medium.

33 is a flow diagram of an example method 3300 for video processing in accordance with currently disclosed technology. Method 3300 includes, at operation 3302, determining new motion candidates for video processing by using one or more motion candidates from one or more tables, wherein the table includes the one or more motion candidates and each motion candidate is associated sports information. Method 3300 includes, at operation 3304, performing a conversion between the video block and a coded representation of the video block based on the new candidate.

In some embodiments, the new motion candidates are derived by adding or subtracting offsets to motion vectors associated with motion candidates from the one or more tables.

In some embodiments, determining the new motion candidate includes determining the new motion candidate as a function of one or more motion candidates from one or more tables and motion information from temporally adjacent blocks.

In some embodiments, determining new motion candidates includes averaging motion candidates from one or more tables and a temporal motion vector predictor.

In some embodiments, averaging the selected motion candidates includes a weighted averaging or averaging of motion vectors associated with the selected motion candidates.

In some embodiments, the averaging is performed without division operations.

In some embodiments, the averaging is achieved by multiplying the sum of motion vectors of motion candidates from the one or more tables with the temporal motion vector predictor and a scaling factor.

In some embodiments, horizontal components of motion vectors of motion candidates from the one or more tables are averaged with a temporal motion vector predictor to derive horizontal components of new motion candidates.

In some embodiments, averaging the selected level components includes a weighted average or average of the level components associated with the selected motion candidate.

In some embodiments, averaging the selected vertical components includes a weighted average or average of the vertical components associated with the selected motion candidates.

In some embodiments, as one or more motion candidates from one or more tables, Merge candidates from spatial neighboring blocks and/or spatially non-immediate neighboring blocks, and motion information from temporal neighboring blocks. function to determine new motion candidates.

In some embodiments, determining the new candidate includes: advanced motion vector prediction (AMVP) as one or more motion candidates from one or more tables, from spatially adjacent blocks, and/or from spatially non-immediate adjacent blocks. Candidate functions, as well as motion information from temporal adjacent blocks, are used to determine new motion candidates.

In some embodiments, determining the new candidate includes being one or more motion candidates from one or more tables and an AMVP candidate in an advanced motion vector prediction (AMVP) candidate list or a Merge candidate in a Merge candidate list. function to determine new motion candidates.

In some embodiments, the new motion candidate is added to the Merge candidate list.

In some embodiments, the new motion candidate is added to the AMVP candidate list.

In some embodiments, each of the one or more tables includes a set of motion candidates, where each motion candidate is associated with corresponding motion information.

In some embodiments, motion candidates are associated with motion information including at least one of: prediction direction, reference picture index, motion vector value, intensity compensation flag, affine flag, motion vector difference accuracy, or motion vector difference.

In some embodiments, the method further includes updating one or more tables based on the transformation.

In some embodiments, updating the one or more tables includes updating the one or more tables based on the motion information of the first video block after performing the conversion.

In some embodiments, the method further includes performing a conversion between subsequent video blocks of the video and a bitstream representation of the video based on the updated table.

34 is a flow diagram of an example method 3400 for video processing in accordance with currently disclosed technology. Method 3400 includes, at operation 3402, by always using motion information from more than one spatially neighboring block of a first video block in the current picture and not using motion information from temporal blocks in a different picture than the current picture. Identifying new candidates for video processing. Method 3400 includes, at operation 3404, performing a conversion between a first video block in a current picture of the video and a bitstream representation of the video using the determined new candidate.

In some embodiments, the determined new candidates are added to a candidate list, including a Merge candidate list or an Advanced Motion Vector Prediction (AMVP) candidate list.

In some embodiments, motion information from more than one spatial neighbor is a candidate derived from a predefined spatial neighbor relative to the first video block in the current picture, or from one or more tables. Sports Candidates.

In some embodiments, the one or more tables include motion candidates derived from previously processed video blocks processed before the first video block in the video material.

In some embodiments, candidates derived from predefined spatial neighboring blocks relative to the first video block in the current picture are spatial Merge candidates.

In some embodiments, the new candidates are derived by averaging at least two spatial Merge candidates.

In some embodiments, the new candidates are derived by jointly using spatial Merge candidates and motion candidates from one or more tables.

In some embodiments, the new candidate is derived by averaging at least two motion vectors associated with candidates derived from different locations.

In some embodiments, the different locations are adjacent to the first video block.

In some embodiments, the motion vector is obtained only from the position in the current largest coding unit to which the first video block belongs.

In some embodiments, the motion vector is obtained only from the position in the current largest coding unit row.

In some embodiments, the motion vector is obtained only from a position in or next to the current maximum coding unit row.

In some embodiments, the motion vector is obtained only from a position in or next to the current maximum coding unit row but not to the left of the upper left neighboring block.

In some embodiments, the motion vector for the lower right block used for planar motion prediction is not obtained from the temporal motion vector prediction candidates, but is obtained from an entry of the table.

In some embodiments, the new candidates are derived by jointly using motion candidates from one or more tables and other kinds of Merge/AMVP candidates.

In some embodiments, the motion candidates in the one or more tables are associated with motion information, the motion information including at least one of the following: prediction direction, reference picture index, motion vector value, intensity compensation flag, simulation shot flag, motion vector difference accuracy, or motion vector difference value.

In some embodiments, the method further includes updating one or more tables based on the transformation.

In some embodiments, updating the one or more tables includes updating the one or more tables based on the motion information of the first video block after performing the conversion.

In some embodiments, the method further includes performing a conversion between subsequent video blocks of the video and a bitstream representation of the video based on the updated table.

In some embodiments, the conversion includes an encoding process and/or a decoding process.

35 is a flow diagram of an example method 3500 for video processing in accordance with currently disclosed technology. Method 3500 includes at operation 3502, by using motion information from at least one spatially non-immediately adjacent block of the first video block in the current picture, and from a spatially non-immediately adjacent block of the first video block or from a spatially non-immediately adjacent block of the first video block. Other candidates derived from immediately adjacent blocks are used to determine new candidates for video processing. Method 3500 includes, at operation 3504, performing a conversion between a first video block of the video and a bitstream representation of the video using the determined new candidate.

In some embodiments, the determined new candidates are added to a candidate list, including a Merge or Advanced Motion Vector Prediction (AMVP) candidate list.

In some embodiments, motion information from more than one spatially non-immediate block is a candidate derived from a predefined spatially non-immediate block relative to the first video block in the current picture.

In some embodiments, candidates derived from predefined spatially non-immediate blocks relative to the first video block in the current picture are spatial-temporal motion vector prediction (STMVP) candidates.

In some embodiments, a non-immediate neighbor of the video block is not a right neighbor or a left neighbor of the first video block.

In some embodiments, the upper block of the first video block used for STMVP candidate derivation remains unchanged, while the left block used changes from a neighboring block to a non-immediate block.

In some embodiments, the left block of the first video block used for STMVP candidate derivation remains unchanged, while the upper block used changes from a neighboring block to a non-immediate block.

36 is a flow diagram of an example method 3600 for video processing in accordance with currently disclosed technology. Method 3600 includes, at operation 3602, determining a time block for video processing by using motion information from one or more tables of a first video block in the current picture and motion information from a temporal block in a picture different from the current picture. New candidate. Method 3600 includes, at operation 3604, performing a conversion between a first video block in a current picture of the video and a bitstream representation of the video using the determined new candidate.

In some embodiments, the determined new candidates are added to a candidate list, including a Merge or AMVP candidate list.

In some embodiments, motion information from one or more tables in the current picture is associated with one or more historical motion vector prediction (HMVP) candidates selected from the one or more tables and is from a different source than the current picture. The motion information of the temporal block in the picture is the temporal motion candidate.

In some embodiments, the new candidates are derived by averaging one or more HMVP candidates with one or more temporal motion candidates.

In some embodiments, the one or more tables include motion candidates derived from previously processed video blocks processed before the first video block in the video material.

37 is a flow diagram of an example method 3700 for video processing in accordance with currently disclosed technology. Method 3700 includes, at operation 3702, determining a new table for video processing by using motion information from one or more tables of the first video block and motion information from one or more spatially adjacent blocks of the first video block. candidate. Method 3700 includes, at operation 3704, performing a conversion between a first video block in a current picture of the video and a bitstream representation of the video using the determined new candidate.

In some embodiments, the determined new candidates are added to a candidate list, including a Merge or AMVP candidate list.

In some embodiments, motion information from one or more tables of the first video block is associated with one or more historical motion vector prediction (HMVP) candidates selected from the one or more tables and from the first video The motion information of one or more spatial neighbors of the block is a candidate derived from a predefined spatial block relative to the first video block.

In some embodiments, the candidates derived from a predefined spatial block relative to the first video block are spatial Merge candidates.

In some embodiments, the new candidates are derived by averaging one or more HMVP candidates and one or more spatial Merge candidates.

In some embodiments, the one or more tables include motion candidates derived from previously processed video blocks processed before the first video block in the video material.

In some embodiments, motion candidates from the table are associated with motion information including at least one of: prediction direction, reference picture index, motion vector value, intensity compensation flag, affine flag, motion vector difference Accuracy or motion vector difference.

In some embodiments, the method further includes updating one or more tables based on the transformation.

In some embodiments, updating the one or more tables includes updating the one or more tables based on motion information of the current video block after performing the conversion.

In some embodiments, the method further includes performing a conversion between subsequent video blocks of the video material and a bitstream representation of the video material based on the updated table.

38 is a flow diagram of an example method 3800 for video processing in accordance with currently disclosed technology. Method 3800 includes, at operation 3802, maintaining a set of tables, wherein each table includes motion candidates, and each motion candidate is associated with corresponding motion information; and at operation 3804, performing a first video block and including the first video block converting between bitstream representations of the video; and at operation 3806, updating the one or more existing motion candidates in the one or more tables by selectively pruning the encoding/decoding mode of the first video block. Multiple tables.

In some embodiments, the conversion between the first video block and a bitstream representation of the video including the first video block is performed based on one or more tables in the set of tables.

In some embodiments, where the first video block is encoded/decoded in Merge mode, pruning is omitted.

In some embodiments, pruning is omitted where the first video block is encoded/decoded in advanced motion vector prediction mode.

In some embodiments, where the first video block is encoded/decoded in Merge mode or advanced motion vector prediction mode, the latest M entries of the table are pruned, where M is a pre-specified integer.

In some embodiments, pruning is disabled where the first video block is encoded/decoded in sub-block mode.

In some embodiments, the sub-block modes include affine mode, selectable temporal motion vector prediction mode.

In some embodiments, the pruning includes checking the table for redundant existing motion candidates.

In some embodiments, the pruning further includes inserting motion information associated with the first video block into the table if there are redundant existing motion candidates in the table, and deleting the The redundant existing motion candidates in the table.

In some embodiments, if there are redundant existing motion candidates in the table, the table is not updated with motion information associated with the first video block.

In some embodiments, the method further includes performing a conversion between subsequent video blocks of the video and a bitstream representation of the video based on the updated table.

39 is a flow diagram of an example method 3900 for video processing in accordance with currently disclosed technology. Method 3900 includes, at operation 3902, maintaining a set of tables, wherein each table includes motion candidates, and each motion candidate is associated with corresponding motion information; and at operation 3904, performing a first video block and including the first video block converting between bitstream representations of the video; and at operation 3906, updating one or more tables to include motion information from one or more temporally adjacent blocks of the first video block as new motion candidates .

In some embodiments, the conversion between the first video block and a bitstream representation of the video including the first video block is performed based on one or more tables in the set of tables.

In some embodiments, the one or more temporally adjacent blocks are co-located blocks.

In some embodiments, the one or more temporal neighboring blocks include one or more blocks from different reference pictures.

In some embodiments, the method further includes performing a conversion between subsequent video blocks of the video and a bitstream representation of the video based on the updated table.

40 is a flow diagram of an example method 4000 for updating a motion candidate table in accordance with currently disclosed technology. Method 4000 includes, at operation 4002, selectively pruning existing motion candidates in the table, each motion candidate being associated with corresponding motion information, based on the encoding/decoding mode of the video block being processed; and at operation 4004 , the table is updated to include the motion information of the video block as the new motion candidate.

In some embodiments, where the video block is encoded/decoded in Merge mode or advanced motion vector prediction mode, the latest M entries of the table are pruned, where M is a pre-specified integer.

In some embodiments, pruning is disabled where the video block is encoded/decoded in sub-block mode.

In some embodiments, the sub-block modes include affine mode, selectable temporal motion vector prediction mode.

In some embodiments, the pruning includes checking the table for redundant existing motion candidates.

In some embodiments, the pruning further includes inserting motion information associated with the video block being processed into the table if redundant motion candidates exist in the table, and deleting the table The redundant motion candidates in .

In some embodiments, if there are redundant existing motion candidates in the table, the table is not updated with motion information associated with the video block being processed.

41 is a flow diagram of an example method 4100 for updating a motion candidate table in accordance with currently disclosed technology. Method 4100 includes, at operation 4102, maintaining a table of motion candidates, each motion candidate being associated with corresponding motion information; and at operation 4104, updating the table to include one or more times from the video block being processed The motion information of adjacent blocks is used as a new motion candidate.

In some embodiments, the one or more temporally adjacent blocks are co-located blocks.

In some embodiments, the one or more temporal neighboring blocks include one or more blocks from different reference pictures.

In some embodiments, motion candidates are associated with motion information including at least one of: prediction direction, reference picture index, motion vector value, intensity compensation flag, affine flag, motion vector difference accuracy, or motion Vector difference.

In some embodiments, the motion candidates correspond to motion candidates for intra prediction mode for intra mode encoding.

In some embodiments, the motion candidate corresponds to a motion candidate that includes brightness compensation parameters for IC parameter encoding.

42 is a flow diagram of an example method 4200 for video processing in accordance with currently disclosed technology. Method 4200 includes, at operation 4202, determining new motion candidates for video processing by using one or more motion candidates from one or more tables, wherein the table includes the one or more motion candidates, and each motion The candidates are associated with motion information; and at operation 4104, a conversion is performed between the video block and a coded representation of the video block based on the new candidate.

In some embodiments, the determined new candidates are added to a candidate list, including a Merge or Advanced Motion Vector Prediction (AMVP) candidate list.

In some embodiments, determining the new candidate includes: being one or more motion candidates from one or more tables and an AMVP candidate in an Advanced Motion Vector Prediction (AMVP) candidate list or a Merge candidate in a Merge candidate list function to determine the new motion candidate.

From the above, it should be understood that specific embodiments of the technology disclosed in the present invention have been described herein for convenience of explanation, but various modifications may be made without departing from the scope of the present invention. Therefore, except for, the technology disclosed in the present invention is not limited to the scope of the patent application.

The embodiments, modules, and functional operations disclosed and otherwise described herein may be implemented in digital electronic circuitry, or computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or one thereof. or a combination of more. The disclosed embodiments and other embodiments may be implemented as one or more computer program products, that is, one or more modules of computer program instructions encoded on a computer-readable medium for execution by a data processing device or to control a data processing device. operation. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a storage device, a material composition that affects a machine-readable propagation signal, or a combination of one or more thereof. The term "data processing device" includes all devices, equipment and machines for processing data, including, for example, a programmable processor, a computer or multi-processor or a group of computers. In addition to hardware, the device may include code that creates an execution environment for computer programs, such as code that makes up the processor's firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these. Propagated signals are artificially generated signals, such as machine-generated electrical, optical, or electromagnetic signals, that are generated to encode information for transmission to an appropriate receiving device.

A computer program (also called a program, software, software application, script, or code) may be written in any form of programming language, including a compiled or interpreted language, and may be deployed in any form, including as a stand-alone program or as a module , components, subroutines, or other units suitable for use in a computing environment. Computer programs do not necessarily correspond to files in the file system. A program can be stored in a portion of a file that holds other programs or data (for example, one or more scripts stored in a markup language file), in a single file dedicated to the program, or in multiple coordinated files (for example, one or more scripts stored in a markup language file). or multiple modules, subroutines, or partial code files). A computer program may be deployed and executed on one or more computers located on a single website or distributed across multiple websites and interconnected by a communications network.

The processing and logic flow described herein may be performed by one or more programmable processors that execute one or more computer programs to perform functions by operating on input data and generating output. Processing and logic flow may also be performed by, and devices may be implemented as, special purpose logic circuits, such as, for example, an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

For example, processors suitable for the execution of computer programs include both general and special purpose microprocessors, and any one or more of any type of digital computer. Typically, a processor will receive instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor that executes instructions and one or more storage devices that store instructions and data. Typically, a computer will also include one or more large storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or be operatively coupled to one or more large storage devices to receive data from or transfer data thereto. Data is transferred to one or more large storage area devices, or both. However, computers do not necessarily have such a facility. Computer-readable media suitable for storage of computer program instructions and data includes all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks , such as internal hard drives or removable disks; magneto-optical disks; and CDROM and DVD-ROM discs. The processor and memory can be supplemented by, or incorporated into, dedicated logic circuits.

It is intended that the specification, together with the drawings, be regarded as illustrative only, where illustrative means example. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. In addition, the use of "or" is intended to include "and/or" unless the context clearly dictates otherwise.

Although this patent document contains many details, they should not be construed as limitations on the scope of any invention or claim, but rather as descriptions of features of particular embodiments of particular inventions. Certain features of this patent document that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various functions that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although the features described above may be described as functioning in certain combinations, or even initially claimed as such, in some cases one or more features in the claimed combination may be deleted from the combination and the claimed scope A combination of can point to a subcombination or a variation of a subcombination.

Likewise, although operations are described in a particular order in the drawings, this should not be understood to imply that such operations must be performed in the specific order or sequence shown, or that all operations illustrated, to obtain desired results. Furthermore, the separation of various system components in the embodiments described in this patent document should not be construed as requiring such separation in all embodiments.

Only some implementations and examples are described, other implementations, enhancements, and variations may be made based on what is described and illustrated in this patent document.

100: Video transcoder 3000: Computer system 3005: Processor 3010: Memory 3015: Network interface card 3025: Internal connection 3100: Mobile device 3101: Processor/controller 3102: Memory 3103: Input/output (I /O) Unit 3104: Display 3200, 3300, 3400, 3500, 3600, 3700, 3800, 3900, 4000, 4100, 4200: Method 3202, 3204, 3206, 3302, 3304, 3402, 3404, 3502, 3504, 3602, 3604, 3702, 3704, 3802, 3804, 3806, 3902, 3904, 3906, 4002, 4004, 4102, 4104, 4202, 4204: Operations A ₀ , A ₁ , B ₁ , B ₀ , B ₂ , C ₀ , C ₁ : Positions a, b, c, d, A, B, C, D, E, F, T: Blocks AL, AR, BL, BR, MV0, MV0′, MV1, MV1′: Motion vectors tb, td: POC distance TD0, TD1: time distance

Figure 1 is a block diagram illustrating an example of a video transcoder implementation. Figure 2 illustrates macroblock partitioning in the H.264 video coding standard. Figure 3 illustrates an example of dividing a coding block (CB) into prediction blocks (PB). Figure 4 illustrates an example implementation of subdividing coding tree blocks (CTBs) into CBs and transform blocks (TBs). Solid lines represent CB boundaries, and dashed lines represent TB boundaries, including example CTBs with partitions and corresponding quadtrees. Figure 5 shows an example of a Quad Binary Tree (QTBT) structure for segmenting video material. Figure 6 shows an example of video block segmentation. Figure 7 shows an example of quadtree partitioning. Figure 8 shows an example of tree signaling. Figure 9 shows an example of the derivation process for Merge candidate list construction. Figure 10 shows example locations of spatial Merge candidates. Figure 11 shows an example of candidate pairs taking into account redundancy checking of spatial Merge candidates. Figure 12 shows an example of the location of the second PU for Nx2N and 2NxN partitions. Figure 13 illustrates example motion vector scaling for temporal Merge candidates. Figure 14 shows the candidate locations of the temporal Merge candidates and their collocated pictures. Figure 15 shows an example of combining bidirectional prediction Merge candidates. FIG. 16 shows an example of the derivation process of motion vector prediction candidates. FIG. 17 shows an example of motion vector scaling of spatial motion vector candidates. Figure 18 shows an example alternative temporal motion vector prediction (ATMVP) for motion prediction of a coding unit (CU). Figure 19 diagrammatically depicts an example of identification of source blocks and source pictures. Figure 20 shows an example of one CU with four sub-blocks and adjacent blocks. Figure 21 illustrates an example of bilateral matching. Figure 22 illustrates an example of template matching. Figure 23 depicts an example of one-sided motion estimation (ME) in frame rate up conversion (FRUC). Figure 24 shows an example of decoder-side motion vector refinement (DMVR) based on bilateral exemplar matching. Figure 25 shows an example of spatial neighbor blocks used to derive illumination compensation IC parameters. Figure 26 shows an example of spatial neighboring blocks used to derive spatial Merge candidates. Figure 27 shows an example of using adjacent inter prediction blocks. FIG. 28 shows an example of the plane motion vector prediction process. Figure 29 shows an example of the position next to the current coding unit (CU) row. 30 is a block diagram illustrating an example of the structure of a computer system or other control device that may be used to implement various portions of the disclosed technology. 31 illustrates a block diagram of an example embodiment of a mobile device that may be used to implement various portions of the disclosed technology. 32 illustrates a flowchart of an example method for video processing in accordance with currently disclosed technology. 33 illustrates a flowchart of an example method for video processing in accordance with currently disclosed technology. 34 illustrates a flowchart of an example method for video processing in accordance with currently disclosed technology. 35 illustrates a flowchart of an example method for video processing in accordance with currently disclosed technology. 36 illustrates a flowchart of an example method for video processing in accordance with currently disclosed technology. 37 illustrates a flowchart of an example method for video processing in accordance with currently disclosed technology. 38 illustrates a flow diagram of an example method for video processing in accordance with currently disclosed technology. 39 illustrates a flowchart of an example method for video processing in accordance with currently disclosed technology. 40 illustrates a flowchart of an example method for updating a motion candidate table in accordance with currently disclosed technology. 41 illustrates a flowchart of an example method for updating a motion candidate table in accordance with currently disclosed technology. 42 illustrates a flow diagram of an example method for video processing in accordance with currently disclosed technology.

3200:Method

3202, 3204, 3206: steps

Claims (23)

A method for video processing, comprising: determining new candidates for video processing by averaging two or more motion candidates selected from one or more lookup tables; adding the new candidates to a candidate list; and performing a conversion between a first video block of the video and a bitstream representation of the video using the determined new candidate in the candidate list; wherein the one or more lookup tables allow encoding or decoding to be performed based on historical data A process for maintaining the one or more lookup tables with motion information from previously encoded blocks during the encoding or decoding process. According to the method described in item 1 of the patent application, the candidate list is a Merge candidate list, and the determined new candidate is a Merge candidate. According to the method described in item 2 of the patent application, the Merge candidate list is an inter-frame prediction Merge candidate list or an intra-frame block copy prediction Merge candidate list. The method of claim 1 , wherein the one or more lookup tables include motion candidates derived from previously processed video blocks in the video material that were processed before the first video block. According to the method described in item 4 of the patent application, there are no available spatial candidates and temporal candidates in the candidate list. The method according to claim 1, wherein said averaging is achieved without division operations. The method according to claim 1, wherein the averaging is achieved by multiplication of a sum of motion vectors of the selected motion candidates and a scaling factor. The method according to claim 1, wherein horizontal components of motion vectors of the selected motion candidates are averaged to derive horizontal components of new candidates. The method according to claim 1, wherein vertical components of motion vectors of the selected motion candidates are averaged to derive vertical components of new candidates. According to the method described in claim 6 of the patent application, the scaling factor is pre-calculated and stored in a lookup table. According to the method described in any one of items 1 to 10 of the patent application, only motion vectors with the same reference picture are selected. The method according to claim 11, wherein only motion vectors having the same reference picture in the two prediction directions are selected in the two prediction directions. The method according to any one of items 1 to 10 of the patent scope, wherein the target reference picture in each prediction direction is predetermined, and the motion vector is scaled to the predetermined reference picture. According to the method described in item 13 of the patent application, the first entry in the reference picture list X is selected as the target reference picture for the reference picture list, and X is 0 or 1. According to the method described in item 13 of the patent application, for each prediction direction, the most commonly used reference picture in the table is selected as the target reference picture. The method according to claim 13, wherein for each prediction direction, a motion vector having the same reference picture as a predetermined target reference picture is first selected, and then other motion vectors are selected. The method according to any one of items 1 to 10 of the patent application scope, wherein the motion candidates from the one or more lookup tables are associated with motion information, the motion information includes at least one of the following: predicted direction , reference picture index, motion vector value, intensity compensation flag, affine flag, motion vector difference precision or motion vector difference value. The method according to any one of items 1 to 10 of the patent scope further includes: updating the one or more lookup tables based on the conversion. The method according to claim 18, wherein the updating of the one or more lookup tables includes updating the one or more lookup tables based on motion information of the first video block of the video after performing the conversion. The method of claim 19, further comprising performing a conversion between subsequent video blocks of the video and a bitstream representation of the video based on the updated one or more lookup tables. According to the method described in any one of items 1 to 10 of the patent application scope, the conversion includes encoding processing and/or decoding processing. A device in a video system includes a processor configured to implement the method described in one or more of items 1 to 21 of the patent application scope. A non-transitory computer-readable program medium having code stored thereon, the code including instructions that, when executed by a processor, cause the processor to implement one of items 1 to 21 of the patent scope or multiple methods described in.

Images