TW202032991A - Extension of look-up table based motion vector prediction with temporal information - Google Patents

Abstract

The application provides a method for video processing. This method includes: determining a new candidate for video processing by averaging motion vectors of two or more selected motion candidates; adding the new candidate to a candidate list; and performing a conversion between a first video block of a video and a bitstream representation of the video by using the determined new candidate in the candidate list.

Description

Extend motion vector prediction based on look-up table with time information

The document of the present invention relates to video coding technology, equipment and system.

Cross references to related applications According to the applicable patent law and/or the Paris Convention, the present invention timely requires the international patent application number PCT/CN2018/095716 filed on July 14, 2018 and the international patent application number PCT/ Priorities and benefits of CN2018/095719. The entire disclosures of International Patent Application Nos. PCT/CN2018/095716 and PCT/CN2018/095719 are incorporated herein by reference as part of the disclosure of the present invention.

Despite advances in video compression, digital video still uses the largest bandwidth on the Internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, it is expected that the demand for bandwidth used by digital video will continue to grow.

This document discloses methods, systems and equipment for encoding and decoding digital video using the Merge list of motion vectors.

In an exemplary aspect, a video processing method is disclosed. The video processing method includes: determining a new candidate for video processing by averaging two or more selected motion candidates; adding the new candidate to a candidate list; and by using the determined new candidate in the candidate list, Perform conversion between the first video block of the video and the bitstream representation of the video.

In an exemplary aspect, a video processing method is disclosed. The video processing method includes: determining a new motion candidate for video processing by using one or more motion candidates from one or more tables, where the table includes one or more motion candidates, and each motion candidate is associated Motion information; based on the new candidate, the conversion between the video block and the coded representation of the video block is performed.

In an exemplary aspect, a video processing method is disclosed. The video processing method includes: always using the motion information of more than one spatial neighboring block from the first video block in the current picture, and not using the motion information from a time block in a picture different from the current picture to determine A new candidate for video processing; the conversion between the first video block in the current picture of the video and the bitstream representation of the video is performed by using the determined new candidate.

In an exemplary aspect, a video processing method is disclosed. The video processing method includes: by using the motion information of at least one spatially non-neighboring block from the first video block in the current picture, and the spatially non-neighboring block from the first video block or the spatially non-neighboring block Other candidates derived from the block are used to determine a new candidate for video processing; by using the determined new candidate, the conversion between the first video block of the video and the bitstream representation of the video is performed.

In an exemplary aspect, a video processing method is disclosed. The video processing method includes: determining a new video processing method by using motion information from one or more tables of a first video block in the current picture and motion information from a time block in a picture different from the current picture. Candidate; by using the determined new candidate, the conversion between the first video block in the current picture of the video and the bitstream representation of the video is performed.

In an exemplary aspect, a video processing method is disclosed. The video processing method includes: determining a new candidate for video processing by using motion information from one or more tables of a first video block and motion information from one or more spatial neighboring blocks of the first video block ; By using the determined new candidate, the conversion between the first video block in the current picture of the video and the bitstream representation of the video is performed.

In an exemplary aspect, a video processing method is disclosed. The video processing method includes: maintaining a set of tables, where each table includes motion candidates, and each motion candidate is associated with corresponding motion information; executing a bit stream of a first video block and a video including the first video block The conversion between representations; and updating the one or more tables by selectively trimming the existing motion candidates in the one or more tables based on the encoding/decoding mode of the first video block.

In an exemplary aspect, a video processing method is disclosed. The video processing method includes: maintaining a set of tables, where each table includes motion candidates, and each motion candidate is associated with corresponding motion information; executing a bit stream of a first video block and a video including the first video block Indicates the conversion between; and updating one or more tables to include the motion information of one or more temporal neighboring blocks from the first video block as a new motion candidate.

In an exemplary aspect, a method of updating a motion candidate table is disclosed. The method includes: selectively trimming the existing motion candidates in the table based on the encoding/decoding mode of the video block being processed, and each motion candidate is associated with corresponding motion information; and updating the table to include The motion information of the video block is used as a new motion candidate.

In an exemplary aspect, a method of updating a motion candidate table is disclosed. The method includes: maintaining a table of motion candidates, each of which is associated with corresponding motion information; and updating the table to include motion information from one or more temporal neighbors of the video block being processed as the new Motion candidates.

In an exemplary aspect, a video processing method is disclosed. The video processing method includes: determining a new motion candidate for video processing by using one or more motion candidates from one or more tables, wherein the table includes one or more motion candidates, and each motion candidate and Motion information is associated; and based on the new candidate, a conversion is performed between the video block and the coded representation of the video block.

In an exemplary aspect, an apparatus in a video system is disclosed. The device includes a processor and a non-transitory memory with instructions thereon, where the instructions when executed by the processor cause the processor to implement the various methods described herein.

The various technologies described herein can be implemented as computer program products stored on non-transitory computer-readable media. The computer program product includes program code for executing the method described herein.

One or more implementation details are set forth in the appendix, drawings, and the following description. Other features will be apparent from the specification and drawings and the scope of patent application.

In order to improve the compression ratio of video, researchers are constantly looking for new technologies to encode video.

1. Introduction

This file is related to video coding technology. Specifically, it is related to motion information coding (for example, Merge mode, AMVP mode) in video coding. It can be applied to the existing video coding standard HEVC, or the standard multifunctional video coding (VVC) to be finalized. It may also be suitable for future video coding standards or video transcoders.

Brief discussion

Video coding standards are mainly developed through the development of well-known ITU-T and ISO/IEC standards. ITU-T developed H.261 and H.263, ISO/IEC developed MPEG-1 and MPEG-4 vision, and the two organizations jointly developed H.262/MPEG-2 video, H.264/MPEG-4 Advanced Video Coding (AVC) and H.265/HEVC standard. Since H.262, the video coding standard is based on a hybrid video coding structure, which uses time-domain prediction plus transform coding. An example of a typical HEVC encoder framework is shown in Figure 1.

2.1 Split structure

2.1.1 H.264/AVC Split tree structure in

The core of the coding layer in the previous standard is the macro block, which contains 16×16 luma sample blocks, and in the case of conventional 4:2:0 color sampling, it contains two corresponding 8×8 chroma sample blocks.

Intra-coding blocks use spatial prediction to explore the spatial correlation between pixels. Two divisions are defined: 16x16 and 4x4.

Inter-coding blocks use temporal prediction instead of spatial prediction by estimating the motion between pictures. The motion of a 16x16 macroblock or any sub-macroblock partition can be estimated separately: 16x8, 8x16, 8x8, 8x4, 4x8, 4x4 (see Figure 2). Only one motion vector (MV) is allowed for each sub-macroblock partition.

2.1.2 HEVC Split tree structure in

In HEVC, the CTU is divided into CUs by using a quadtree structure (represented as a coding tree) to adapt to various local characteristics. It is decided at the CU level whether to use inter (temporal) prediction or intra (spatial) prediction to encode picture regions. According to the PU partition type, each CU can be further divided into one, two, or four PUs. In a PU, the same prediction process is applied, and relevant information is transmitted to the decoder on the PU basis. After the residual block is obtained by applying prediction processing based on the PU partition type, the CU may be partitioned into transformation units (TU) according to another quad-tree structure similar to the coding tree of the CU. An important feature of the HEVC structure is that it has multiple partition concepts, including CU, PU and TU.

In the following, various features involved in hybrid video coding using HEVC are highlighted as follows.

1) Coding tree unit and coding tree block (CTB) structure: A similar structure in HEVC is a coding tree unit (CTU), which has a size selected by the encoder and can be larger than a traditional macroblock. CTU is composed of luminance CTB and corresponding saturation CTB and syntax elements. The size L×L of the brightness CTB can be selected as L=16, 32, or 64 samples. A larger size usually achieves better compression. Then, HEVC supports the use of tree structure and quadtree signaling to divide CTB into smaller blocks.

2) Coding Unit (CU) and Coding Block (CB): The quadtree syntax of CTU specifies the size and location of its brightness and chroma CB. The root of the quadtree is associated with the CTU. Therefore, the size of the brightness CTB is the maximum size supported by the brightness CB. The brightness and chroma of CTU and the division of CB are signaled jointly. One luma CB and usually two chroma CBs and related syntax together form a coding unit (CU). The CTB can contain only one CU, or it can be divided to form multiple CUs, and each CU has an associated division, which is divided into a prediction unit (PU) and a conversion unit tree (TU).

3) Prediction unit (PU) and prediction block (PB): It is decided at the CU level whether to use inter prediction or intra prediction to encode the picture area. The root of the PU partition structure is at the CU level. Depending on the basic prediction type decision, the brightness and chroma CB can be further divided in size, and the brightness and chroma CB can be predicted from the brightness and chroma prediction block (PB). HEVC supports variable PB size from 64×64 to 4×4 samples. Figure 3 shows an example of allowed PB for MxM CU.

4) Transform unit (TU) and transform block (TB): Use block transform to encode prediction residuals. The root of the TU tree structure is at the CU level. The luminance CB residual may be the same as the luminance TB, or it may be further divided into smaller luminance transformation blocks TB. The same applies to chroma TB. For 4×4, 8×8, 16×16, and 32×32 square TBs, an integer basis function similar to the discrete cosine transform (DCT) is defined. For the 4×4 transform of the luma intra-prediction residual, an integer transform derived from the discrete sine transform (DST) form can also be specified.

Fig. 4 shows an example of subdividing CTB into CB (and transformation blocks (TB)). The solid line indicates the CB boundary, and the dashed line indicates the TB boundary. (A) CTB with segmentation. (B) The corresponding quadtree.

2.1.2.1 Divide into a tree structure division of transform blocks and units

For residual coding, CB can be recursively divided into conversion blocks (TB). The segmentation is signaled by the residual quadtree. Only the square CB and TB partitions are specified, where the blocks can be recursively divided into four quadrants, as shown in Figure 4. For a given luminance CB of size M×M, the flag indicates whether it is divided into four blocks of size M/2×M/2. If it can be further divided, as indicated by the maximum depth of the residual quadtree indicated in the sequence parameter set (SPS), each quadrant will be assigned a flag indicating whether to divide it into four quadrants. The leaf node blocks generated from the residual quadtree are transformed blocks that are further processed by transform coding. The encoder indicates the maximum and minimum brightness TB size it will use. When the CB size is larger than the maximum TB size, the division is implied. When the division will result in a smaller luminance TB size than the indicated minimum value, it implies no division. The chroma TB size is half of the luminance TB size in each dimension, except when the luminance TB size is 4×4, in this case, the area covered by four 4×4 luminance TBs uses a single 4×4 Saturation TB. In the case of an intra-predicted CU, the decoded samples of the nearest neighboring TB (in or out of CB) are used as reference material for intra prediction.

Unlike previous standards, for inter-predicted CUs, the HEVC design allows TB to span multiple PBs to maximize the potential coding efficiency benefiting from the quadtree structure of TB partitioning.

2.1.2.2 Parent node and child node

The CTB is divided according to the quadtree structure, and its nodes are coding units. Multiple nodes in the quadtree structure include leaf nodes and non-leaf nodes. Leaf nodes have no child nodes in the tree structure (that is, leaf nodes will not be further divided). Non-leaf nodes include the root node of the tree structure. The root node corresponds to the initial video block (for example, CTB) of the video material. For each respective non-root node of the plurality of nodes, the respective non-root node corresponds to a video block, which is a child block of the video block corresponding to the parent node in the tree structure of the respective non-root node. Each of the plurality of non-leaf nodes has one or more child nodes in the tree structure.

2.1.3 Joint Exploration Model ( JEM ) Has a larger CTU Quadtree plus binary tree block structure

In order to explore future video coding technologies beyond HEVC, VCEG and MPEG jointly established a Joint Video Exploration Team (JVET) in 2015. Since then, JVET has adopted many new methods and applied them to a reference software called Joint Exploration Model (JEM).

2.1.3.1 QTBT block partition structure

Unlike HEVC, the QTBT structure eliminates the concept of multiple partition types, that is, it eliminates the separation of the concepts of CU, PU, and TU, and supports more flexibility of CU partition shape. In the QTBT block structure, the CU can be square or rectangular. As shown in Figure 5, first the coding tree unit (CTU) is segmented with a quadtree structure. The quad leaf node is further divided by the binary tree structure. There are two types of divisions in binary tree division: symmetrical horizontal division and symmetrical vertical division. The binary leaf node is called a coding unit (CU), and the division is used for prediction and transformation processing without further division. This means that CU, PU, and TU have the same block size in the QTBT coding block structure. In JEM, a CU is sometimes composed of coding blocks (CB) of different color components. For example, in the P-strip and B-strip in the 4:2:0 chroma format, a CU contains a luminance CB and two color CU is sometimes composed of a single component of CB, for example, in the case of I stripe, one CU contains only one luma CB or only two chroma CBs.

The following parameters are defined for the QTBT segmentation scheme.

-CTU size: the size of the root node of the quadtree, which is the same as the concept in HEVC.

– MiNQTSize : the minimum allowable quadrilateral leaf node size

– MaxBTSize : the maximum allowable size of the root node of the binary tree

– MaxBTDePTh : the maximum allowable depth of the binary tree

– MiNBTSize : the minimum allowable size of a binary leaf node

In an example of the QTBT segmentation structure, the CTU size is set to 128×128 luma samples with two corresponding 64×64 chroma sample blocks, MiNQTSize is set to 16×16, MaxBTSize is set to 64×64, MiNBTSize (width and height) is set to 4×4, and MaxBTSize is set to 4. The quadtree division is first applied to the CTU to generate quad-leaf nodes. The size of the quad leaf node may have a size from 16×16 (ie, MiNQTSize ) to 128×128 (ie, CTU size). If the leaf quadtree node is 128×128, it will not be further divided by the binary tree because its size exceeds MaxBTSize (ie, 64×64). Otherwise, the leaf quadtree node can be further divided by the binary tree. Therefore, the quad leaf node is also the root node of the binary tree, and its binary tree depth is zero. When the depth of the binary tree reaches MaxBTDePTh (ie, 4), no further division is considered. When the width of the binary tree node is equal to MiNBTSize (ie, 4), no further leveling is considered. Similarly, when the height of the binary tree node is equal to MiNBTSize , no further vertical division is considered. The leaf nodes of the binary tree are further processed through prediction and transformation processing without further segmentation. In JEM, the maximum CTU size is 256×256 luminance samples.

Figure 5 (left side) illustrates an example of block segmentation by using QTBT, and Figure 5 (right side) illustrates the corresponding tree representation. The solid line represents the quadtree division, and the dashed line represents the binary tree division. In each partition (ie, non-leaf) node of the binary tree, a flag is signaled to indicate which partition type (ie, horizontal or vertical) is used, where 0 represents horizontal partition and 1 represents vertical partition. For quadtree division, there is no need to specify the division type, because quadtree division always divides one block horizontally and vertically to generate 4 sub-blocks of the same size.

In addition, the QTBT solution supports the ability to have separate QTBT structures for brightness and chroma. Currently, for P-strip and B-strip, the luminance and chroma CTB in a CTU share the same QTBT structure. However, for I-slice, the QTBT structure is used to divide the luminance CTB into CUs, and another QTBT structure is used to divide the chroma CTB into chroma CUs. This means that the CU in the I slice consists of coding blocks of the luma component or two chroma component coding blocks, and the CU in the P slice or the B slice consists of coding blocks of all three color components.

In HEVC, in order to reduce memory access for motion compensation, inter-frame prediction of small blocks is restricted, so that 4×8 and 8×4 blocks do not support bidirectional prediction, and 4×4 blocks do not support inter-frame prediction. In JEM's QTBT, these restrictions are removed.

2.1.4 Multifunctional video coding ( VVC ) Trinomial tree

As proposed in JVET-D0117, tree types other than quadtree and binary tree are also supported. In the implementation, two additional trinomial tree (TT) divisions are introduced, namely, horizontal and vertical center-side trinomial trees, as shown in Figure 6(d) and (e).

Figure 6 shows: (a) quadtree partition, (b) vertical binary tree partition (c) horizontal binary tree partition (d) vertical center side trinomial tree partition, and (e) horizontal center side trinomial tree partition.

In some implementations, there are two levels of trees: regional trees (quadtrees) and prediction trees (binary or trinomial trees). First use the regional tree (RT) to divide the CTU. The prediction tree (PT) can be further used to divide the RT leaves. PT can also be used to further divide the PT leaf until the maximum PT depth is reached. The PT leaf is the basic coding unit. For convenience, it is still called CU. CU cannot be further divided. Both prediction and transformation are applied to CU in the same way as JEM. The whole partition structure is called "multi-type tree".

2.1.5 JVET-J0021 Split structure

The tree structure called multi-tree (MTT) is a generalization of QTBT. In QTBT, as shown in Figure 5, the coding tree unit (CTU) is divided first with a quadtree structure. Then use the binary tree structure to further divide the quad leaf nodes.

The basic structure of MTT consists of two types of tree nodes: regional tree (RT) and prediction tree (PT), which supports nine types of divisions, as shown in Figure 7.

Figure 7 illustrates: (a) quadtree division, (b) vertical binary tree division, (c) horizontal binary tree division, (d) vertical trinomial tree division, (e) horizontal trinomial tree division, (f) horizontal upward non Symmetrical binary tree partition, (g) horizontal downward asymmetric binary tree partition, (h) vertical left asymmetric binary tree partition, and (i) vertical right asymmetric binary tree partition.

The regional tree can recursively divide the CTU into square blocks up to 4x4 size regional leaf nodes. At each node of the area tree, a prediction tree can be formed from one of three tree types: Binary Tree (BT), Trinomial Tree (TT) and Asymmetric Binary Tree (ABT). In the PT division, quadtree division is prohibited in the branches of the prediction tree. Like JEM, the luminance tree and the chromaticity tree are separated in the I strip. The signaling method of RT and PT is shown in Figure 8.

2.2 Inter prediction in HEVC/H.265

Each inter-predicted PU has one or two motion parameters of the reference picture list. The motion parameters include motion vectors and reference picture indexes. The use of one of the two reference picture lists can also be signaled using inter_pred_idc . Motion vectors can be explicitly coded as increments relative to the predictor. This coding mode is called advanced motion vector prediction (AMVP) mode.

When the CU is coded in skip mode, a PU is associated with the CU, and there is no significant residual coefficient, no coded motion vector increment or reference picture index. A Merge mode is specified, through which the motion parameters of the current PU can be obtained from adjacent PUs (including spatial and temporal candidates). The Merge mode can be applied to any inter-predicted PU, not just the skip mode. Another option for the Merge mode is the explicit transmission of motion parameters, in which the motion vector, the reference picture index corresponding to each reference picture list, and the use of the reference picture list will be explicitly signaled in each PU.

When the signaling indicates that one of the two reference picture lists is to be used, the PU is generated from one sample block. This is called "one-way prediction". One-way prediction is available for both P-band and B-band.

When the signaling indicates that two reference picture lists are to be used, the PU is generated from the two sample blocks. This is called "bidirectional prediction". Bi-directional prediction is only available for band B.

The details of the inter prediction modes specified in HEVC are provided below. The description will start in Merge mode.

2.2.1 Merge mode

2.2.1.1Merge Derivation of model candidates

When using the Merge mode to predict the PU, the index pointing to the entry in the Merge candidate list is analyzed from the bit stream and used to retrieve motion information. The structure of the list is specified in the HEVC standard and can be summarized in the following sequence of steps:

Step 1: Initial candidate derivation

Step 1.1: Airspace candidate derivation

Step 1.2: Redundancy check of airspace candidates

Step 1.3: Time domain candidate derivation

Step 2: Additional candidate insertion

Step 2.1: Creation of bidirectional prediction candidates

Step 2.2: Insertion of zero motion candidates

These steps are also schematically depicted in FIG. 9. For the derivation of spatial Merge candidates, at most four Merge candidates are selected among candidates located at five different positions. For the time-domain Merge candidate derivation, at most one Merge candidate is selected from the two candidates. Since the number of candidates for each PU is assumed to be constant at the decoder, when the number of candidates does not reach the maximum number of Merge candidates ( MaxNumMergeCand ) signaled in the slice header , additional candidates are generated. Since the number of candidates is constant, the index of the best Merge candidate is encoded using truncated unary binarization (TU). If the size of the CU is equal to 8, all PUs of the current CU share a Merge candidate list, which is the same as the Merge candidate list of 2N×2N prediction units.

The following describes the operations related to the above steps in detail.

2.2.1.2 Airspace candidate derivation

In the derivation of spatial Merge candidates, at most four Merge candidates are selected among the candidates located at the positions shown in FIG. 10. The derivation sequence is A1, B1, B0, A0 and B2. Position B2 is considered only when any PU at positions A1, B1, B0, A0 is not available (for example, because it belongs to another strip or slice) or is internally coded. After the candidates at the A1 position are added, a redundancy check is performed on the addition of the remaining candidates, which ensures that candidates with the same motion information are excluded from the list, thereby improving coding efficiency. In order to reduce the computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. On the contrary, only the pair connected with the arrow in Fig. 11 will be considered, and only when the corresponding candidate for redundancy check does not have the same motion information, the candidate is added to the list. Another source of copied motion information is the "second PU" related to the 2N×2N different partition. For example, Fig. 12 depicts the second PU in the case of N×2N and 2N×N, respectively. When the current PU is divided into N×2N, the A1 position candidate is not considered for list construction. In some embodiments, adding this candidate may result in two prediction units with the same motion information, which is redundant for having only one PU in the coding unit. Likewise, when the current PU is divided into 2N×N, the position B1 is not considered.

2.2.1.3 Time Domain Candidate Derivation

In this step, only one candidate is added to the list. In particular, in the derivation of this time-domain Merge candidate, the scaling motion vector is derived based on the collocated PU that has the smallest picture order count POC difference with the current picture in the given reference picture list. The reference picture list used to derive the collocated PU is explicitly signaled in the slice header. The dotted line in FIG. 13 shows the acquisition of the scaled motion vector of the time-domain Merge candidate, which uses POC distances tb and td to scale from the motion vector of the collocated PU, where tb is defined as the distance between the reference picture of the current picture and the current picture. POC difference, and td is defined as the POC difference between the reference picture of the collocated picture and the collocated picture. The reference picture index of the time domain Merge candidate is set to zero. The actual processing of the scaling process is described in the HEVC specification. For slice B, two motion vectors (one for reference picture list 0 and the other for reference picture list 1) are obtained and combined to make them a bi-directional predictive Merge candidate. Illustrates the motion vector scaling for temporal Merge candidates.

In the collocated PU (Y) belonging to the reference frame, the position of the time domain candidate is selected between the candidates C ₀ and C ₁ , as shown in FIG. 14. If the PU at position C ₀ is unavailable, internally coded or outside the current CTU, position C ₁ is used. Otherwise, the position C ₀ is used for the derivation of the time-domain Merge candidate.

2.2.1.4 Additional candidate insertion

In addition to space-time Merge candidates, there are two additional types of Merge candidates: combined bidirectional prediction Merge candidates and zero Merge candidates. The combined bidirectional prediction Merge candidate is generated using space-time Merge candidates. The combined bidirectional prediction Merge candidate is only used for the B band. The combined bidirectional prediction candidate is generated by combining the first reference picture list motion parameter of the initial candidate with the second reference picture list motion parameter of another candidate. If these two tuples provide different motion hypotheses, they will form a new bidirectional prediction candidate. Figure 15 shows the case where two candidates in the original list (on the left) are used to create a combined bidirectional prediction Merge candidate added to the final list (on the right), which has two of MvL0 and refIdxL0 or MvL1 and refIdxL1 Candidate. There are many rules about combination that need to be considered to generate these additional Merge candidates.

Insert zero motion candidates to fill the remaining entries in the Merge candidate list to reach the capacity of MaxNumMergeCand. These candidates have a zero spatial displacement and a reference picture index that starts from zero and increases every time a new zero motion candidate is added to the list. The number of reference frames used by these candidates is 1 frame and 2 frames for unidirectional prediction and bidirectional prediction, respectively. Finally, no redundancy check is performed on these candidates.

2.2.1.5 Motion estimation area processed in parallel

In order to speed up the encoding process, motion estimation can be performed concurrently, thereby deriving the motion vectors of all prediction units in a given area at the same time. Deriving Merge candidates from the spatial neighborhood may interfere with parallel processing, because a prediction unit cannot derive motion parameters from neighboring PUs before completing related motion estimation. In order to ease the balance between coding efficiency and processing delay, HEVC defines a motion estimation region (MER). The following syntax element "log2_parallel_merge_level_minus2" can be used to signal the size of MER in the picture parameter set. When MER is defined, Merge candidates that fall into the same area are marked as unavailable, and therefore are not considered in the list construction.

The picture parameter sets the original byte sequence payload ( RBSP )grammar

General picture parameter set RBSP grammar pic_paraMeter_set_rbsp() { Descriptor pps_pic_paraMeter_set_id ue(v) pps_seq_paraMeter_set_id ue(v) depeNdeNt_slice_segMeNts_eNabled_flag u(1) … To pps_scaliNg_list_data_preseNt_flag u(1) If( pps_scaliNg_list_data_preseNt_flag) To ScaliNg_list_data() To lists_ModificatioN_preseNt_flag u(1) log2_parallel_Merge_level_MiNus2 ue(v) slice_segMeNt_header_exteNsioN_preseNt_flag u(1) pps_exteNsioN_preseNt_flag u(1) … To Rbsp_trailiNg_bits() To } To

log2_parallel_Merge_level_MiNus2 plus 2 specifies the value of the variable Log2ParMrgLevel, which is used in the derivation process of the brightness motion vector of the Merge mode specified in Article 8.5.3.2.2, and the derivation process of the spatial Merge candidate specified in Article 8.5.3.2.3. The value of log2_parallel_Merge_level_MiNus2 should be in the range of 0 to CtbLog2SizeY − 2, including 0 and CtbLog2SizeY − 2.

The variable Log2ParMrgLevel is derived as follows:

Log2ParMrgLevel = log2_parallel_Merge_level_MiNus2 + 2 (7-37)

Note 3-The value of Log2ParMrgLevel represents the built-in function of parallel derivation of the Merge candidate list. For example, when Log2ParMrgLevel is equal to 6, the Merge candidate list of all prediction units (PU) and coding units (CU) contained in the 64x64 block can be derived in parallel.

2.2.2 AMVP Motion vector prediction in mode

Motion vector prediction uses the space-time correlation between motion vectors and adjacent PUs, which is used for explicit transmission of motion parameters. First, the motion vector candidate list is constructed by checking the availability of the temporally adjacent PU positions in the upper left, removing redundant candidate positions and adding a zero vector to make the length of the candidate list constant. Then, the encoder can select the best predictor from the candidate list and send a corresponding index indicating the selected candidate. Similar to Merge index signaling, the index of the best motion vector candidate is encoded using truncated unary. The maximum value to be encoded in this case is 2 (for example, Figure 2 to Figure 8). In the following chapters, the derivation process of motion vector prediction candidates will be introduced in detail.

2.2.2.1 Derivation of motion vector prediction candidates

Figure 16 summarizes the derivation process of motion vector prediction candidates.

In motion vector prediction, two types of motion vector candidates are considered: spatial motion vector candidates and temporal motion vector candidates. For the derivation of the spatial motion vector candidates, two motion vector candidates are finally derived based on the motion vector of each PU located in the five different positions shown in FIG. 11.

For the derivation of time-domain motion vector candidates, one motion vector candidate is selected from two candidates, and the two candidates are derived based on two different juxtaposed positions. After making the first empty time selection list, remove the repeated motion vector candidates from the list. If the number of potential candidates is greater than two, the motion vector candidates whose reference picture index is greater than 1 in the associated reference picture list are removed from the list. If the number of space-time motion vector candidates is less than two, additional zero motion vector candidates will be added to the list.

2.2.2.2 Spatial motion vector candidate

When deriving the spatial motion vector candidates, at most two candidates are considered among the five potential candidates. These five candidates are from the PU at the positions depicted in FIG. 11, and these positions are the same as the positions of the motion merge. The derivation sequence on the left side of the current PU is defined as A ₀ , A ₁ , and scaled A ₀ , scaled A ₁ . The derivation sequence above the current PU is defined as B ₀ , B ₁ , B ₂ , scaled B ₀ , scaled B ₁ , scaled B ₂ . Therefore, there are four cases on each side that can be used as motion vector candidates, two cases do not need to use spatial scaling, and two cases use spatial scaling. The four different situations are summarized as follows:

--Zoom without space

(1) The same reference picture list and the same reference picture index (same POC)

(2) Different reference picture lists, but the same reference picture (same POC)

--Space zoom

(3) The same reference picture list, but different reference pictures (different POC)

(4) Different reference picture lists, and different reference pictures (different POC)

First check the situation without space scaling, and then check the space scaling. When the POC is different between the reference picture of the neighboring PU and the reference picture of the current PU, spatial scaling will be considered instead of the reference picture list. If all the PUs of the left candidate are unavailable or are internally coded, the aforementioned motion vector is allowed to be scaled to help parallel derivation of the left and upper MV candidates. Otherwise, spatial scaling of the above motion vector is not allowed.

In the spatial scaling process, the motion vectors of neighboring PUs are scaled in a similar manner to temporal scaling, as shown in FIG. 17. The main difference is that the reference picture list and index of the current PU are given as input, and the actual scaling process is the same as the time domain scaling process.

2.2.2.3 Time Domain Motion vector candidate

Except for the derivation of the reference picture index, all the derivation processes of the temporal Merge candidates are the same as those of the spatial motion vector candidates (see, for example, FIG. 6). Signal the reference picture index to the decoder.

2.2.2.4 AMVP Information signaling

For AMVP mode, four parts can be signaled in the bit stream, including prediction direction, reference index, MVD and MV prediction candidate index.

Syntax table: predictioN_uNit( x0, y0, NPbW, NPbH) { Descriptor if( cu_skip_flag[ x0 ][ y0]) { if( MaxNuMMergeCaNd> 1) Merge_idx [x0 ][ y0] ae(v) } else {/* MODE_INTER */ Merge_flag [x0 ][ y0] ae(v) if( Merge_flag[ x0 ][ y0]) { if( MaxNuMMergeCaNd> 1) Merge_idx [x0 ][ y0] ae(v) } else { if( slice_type = = B) iNter_pred_idc [x0 ][ y0] ae(v) if( iNter_pred_idc[ x0 ][ y0] != PRED_L1) { if( NuM_ref_idx_l0_active_MiNus1> 0) ref_idx_l0 [x0 ][ y0] ae(v) Mvd_codiNg( x0, y0, 0) Mvp_l0_flag [x0 ][ y0] ae(v) } if( iNter_pred_idc[ x0 ][ y0] != PRED_L0) { if( NuM_ref_idx_l1_active_MiNus1> 0) ref_idx_l1 [x0 ][ y0] ae(v) if( Mvd_l1_zero_flag && iNter_pred_idc[ x0 ][ y0] == PRED_BI) { MvdL1[ x0 ][ y0 ][ 0] = 0 MvdL1[ x0 ][ y0 ][ 1] = 0 } else Mvd_codiNg( x0, y0, 1) Mvp_l1_flag [x0 ][ y0] ae(v) } } } }

Motion vector difference grammar Mvd_codiNg( x0, y0, refList) { Descriptor abs_Mvd_greater0_flag [0] ae(v) abs_Mvd_greater0_flag [1] ae(v) if( abs_Mvd_greater0_flag[ 0]) abs_Mvd_greater1_flag [0] ae(v) if( abs_Mvd_greater0_flag[ 1]) abs_Mvd_greater1_flag [1] ae(v) if( abs_Mvd_greater0_flag[ 0]) { if( abs_Mvd_greater1_flag[ 0]) abs_Mvd_MiNus2 [0] ae(v) Mvd_sigN_flag [0] ae(v) } if( abs_Mvd_greater0_flag[ 1]) { if( abs_Mvd_greater1_flag[ 1]) abs_Mvd_MiNus2 [1] ae(v) Mvd_sigN_flag [1] ae(v) } }

2.3 Joint Exploration Model ( JEM ) New inter prediction method

2.3.1 Based on sub CU Motion vector prediction

In JEM with QTBT, each CU can have at most one set of motion parameters for each prediction direction. By dividing a large CU into sub-CUs and deriving the motion information of all sub-CUs of the large CU, two sub-CU-level motion vector prediction methods are considered in the encoder. The optional temporal motion vector prediction (ATMVP) method allows each CU to obtain multiple sets of motion information from multiple blocks smaller than the current CU in the collocated reference picture. In the space-time motion vector prediction (STMVP) method, the motion vector of the sub-CU is recursively derived by using a time-domain motion vector predictor and a spatially adjacent motion vector.

In order to maintain a more accurate motion field for sub-CU motion prediction, motion compression of reference frames is currently disabled.

2.3.1.1 Optional temporal motion vector prediction

In the optional temporal motion vector prediction (ATMVP) method, the motion vector temporal motion vector prediction (TMVP) is modified by extracting multiple sets of motion information (including motion vectors and reference indexes) from blocks smaller than the current CU. As shown in Figure 18, the sub-CU is a square N×N block (N is set to 4 by default).

ATMVP predicts the motion vectors of sub-CUs in the CU in two steps. The first step is to use so-called time-domain vectors to identify the corresponding blocks in the reference image. The reference picture is called the motion source picture. The second step is to divide the current CU into sub-CUs, and obtain the motion vector and the reference index of each sub-CU from the block corresponding to each sub-CU, as shown in FIG. 18.

In the first step, the reference picture and the corresponding block are determined by the motion information of the spatial neighboring blocks of the current CU. In order to avoid repeated scanning processing of adjacent blocks, the first Merge candidate in the Merge candidate list of the current CU is used. The first available motion vector and its associated reference index are set as the index of the time domain vector and the motion source picture. In this way, in ATMVP, compared with TMVP, the corresponding block can be identified more accurately, where the corresponding block (sometimes called a collocated block) is always located at the lower right corner or center position relative to the current CU. In one example, if the first Merge candidate is from the left neighboring block (ie, A _{1 in} FIG. 19), the related MV and reference picture are used to identify the source block and the source picture.

Fig. 19 shows an example of identification of source blocks and source pictures.

In the second step, by adding the time domain vector to the coordinates of the current CU, the corresponding block of the sub-CU is identified by the time domain vector in the motion source picture. For each sub-CU, the motion information of its corresponding block (the smallest motion grid covering the center sample) is used to derive the motion information of the sub-CU. After identifying the motion information corresponding to the N×N block, it is converted into the motion vector and reference index of the current sub-CU, which is the same as the TMVP method of HEVC, in which motion scaling and other processing are applied. For example, the decoder checks whether the low-delay condition is met (that is, the POC of all reference pictures of the current picture is less than the POC of the current picture), and may use the motion vector MVx (the motion vector corresponding to the reference picture list X) to assign each sub CU predicts the motion vector MVy (X equals 0 or 1 and Y equals 1−X).

2.3.1.2 Space-time motion vector prediction

In this method, the motion vector of the sub-CU is derived recursively in the raster scan order. Figure 20 illustrates this concept. Consider an 8×8 CU, which contains four 4×4 sub-CUs A, B, C, and D. The adjacent 4×4 blocks in the current frame are labeled a, b, c, and d.

The motion derivation of sub CU A starts by identifying its two spatial neighbors. The first neighbor is the N×N block (block c) above the sub CU A. If the block c is not available or internally coded, then check the other N×N blocks above the sub-CU A (from left to right, starting from block c). The second neighbor is a block to the left of the sub CU A (block b). If block b is not available or is internally coded, check other blocks on the left of sub-CU A (from top to bottom, starting from block b). The motion information obtained from adjacent blocks in each list is scaled to the first reference frame of a given list. Next, according to the same formula as TMVP stipulated in HEVC, the temporal motion vector prediction (TMVP) of sub-block A is derived. Extract the motion information of the collocated block at position D and perform corresponding scaling. Finally, after retrieving and scaling the motion information, all available motion vectors (up to 3) are averaged for each reference list. The average motion vector is designated as the motion vector of the current sub-CU.

Fig. 20 shows an example of one CU with four sub-blocks (A-D) and its neighboring blocks (a-d).

2.3.1.3 child CU Motion prediction mode signaling

The sub-CU mode is enabled as an additional Merge candidate mode, and no additional syntax elements are required to signal the mode. The other two Merge candidates are added to the Merge candidate list of each CU to indicate the ATMVP mode and the STMVP mode. If the sequence parameter set indicates that ATMVP and STMVP are enabled, a maximum of seven Merge candidates are used. The coding logic of the additional Merge candidate is the same as the coding logic of the Merge candidate in the HM, which means that for each CU in the P slice or the B slice, two additional RD checks need to be performed on the two additional Merge candidates.

In JEM, all bin files indexed by Merge are context-encoded by CABAC. However, in HEVC, only the first bin file is context coded, and the remaining biN files are context bypass coded.

2.3.2 Self-adjusting motion vector difference resolution

In HEVC, when use_integer_mv_flag is equal to 0 in the slice header, the motion vector difference (MVD) (between the motion vector of the PU and the predicted motion vector) is signaled in units of a quarter luminance sample. In JEM, local self-adjusting motion vector resolution (LAMVR) is introduced. In JEM, MVD can be coded in units of one-quarter luminance samples, integer luminance samples, or four luminance samples. The MVD resolution is controlled at the coding unit (CU) level, and the MVD resolution flag conditionally signals each CU with at least one non-zero MVD component.

For a CU with at least one non-zero MVD component, the first flag will be signaled to indicate whether a quarter-luminance sample MV accuracy is used in the CU. When the first flag (equal to 1) indicates that one-quarter luminance sample MV accuracy is not used, the other flag signals whether to use integer luminance sample MV accuracy or four luminance sample MV accuracy.

When the first MVD resolution flag of the CU is zero or is not coded for the CU (meaning that all MVDs in the CU are zero), the CU uses a quarter luminance sample MV resolution. When a CU uses integer luma sample MV precision or four luma sample MV precision, the MVP in the AMVP candidate list of the CU will be rounded to the corresponding precision.

In the encoder, the CU-level RD check is used to determine which MVD resolution will be used for the CU. That is, three CU-level RD checks are performed for each MVD resolution. In order to speed up the encoder, the following coding scheme is applied in JEM.

During the RD inspection of a CU with a normal quarter luminance sampling MVD resolution, the motion information of the current CU (integer luminance sampling accuracy) is stored. When performing RD inspection on the same CU with integer luminance samples and 4 luminance sample MVD resolutions, the stored motion information (rounded) is used as the starting point for further refinement of small-range motion vectors, which makes it time-consuming The motion estimation process will not be repeated three times.

Conditionally invoke the RD check of the CU with the MVD resolution of 4 luminance samples. For the CU, when the RD check cost of the MVD resolution of the integer luminance sample is much greater than the RD check cost of the quarter luminance sample MVD resolution, the RD check of the 4 luminance sample MVD resolution of the CU will be skipped.

2.3.3 Pattern matching motion vector derivation

Pattern matching motion vector derivation (PMMVD) mode is a special Merge mode based on frame rate up-conversion (FRUC) technology. In this mode, the motion information of the block will not be signaled, but is derived at the decoder side.

For CU, when its Merge flag is true, the FRUC flag is signaled. When the FRUC flag is false, the Merge index is signaled and the regular Merge mode is used. When the FRUC flag is true, another FRUC mode flag is signaled to indicate which mode (bilateral matching or template matching) will be used to derive the motion information of the block.

On the encoder side, it is determined whether to use the FRUC Merge mode for the CU based on the RD cost selection made for the normal Merge candidates. That is, the two matching modes of CU (bilateral matching and template matching) are checked by using RD cost selection. The mode that results in the lowest cost is further compared with other CU modes. If the FRUC matching mode is the most effective mode, then for the CU, the FRUC flag is set to true and the relevant matching mode is used.

The motion derivation process in FRUC Merge mode has two steps: first, perform CU-level motion search, and then perform sub-CU-level motion optimization. At the CU level, based on bilateral matching or template matching, the initial motion vector of the entire CU is derived. First, a list of MV candidates is generated, and the candidate that leads to the lowest matching cost is selected as the starting point for further optimization of the CU level. Then a local search based on bilateral matching or template matching is performed near the starting point, and the MV result with the smallest matching cost is taken as the MV value of the entire CU. Then, using the derived CU motion vector as a starting point, the motion information is further refined at the sub-CU level.

For example, for W×H CU motion information derivation, the following derivation process is performed. In the first stage, the MV of the entire W×H CU was derived. In the second stage, the CU is further divided into M×M sub-CUs. The value of M is calculated according to (16), and D is the predefined division depth, which is set to 3 by default in JEM. Then the MV value of each sub-CU is derived.

（3）

(3)

As shown in FIG. 21, by finding the closest match between two blocks in two different reference pictures along the motion trajectory of the current CU, bilateral matching is used to derive the motion information of the current CU. Under the assumption of continuous motion trajectories, the motion vectors MV0 and MV1 pointing to the two reference blocks are proportional to the time distance between the current picture and the two reference pictures (ie, TD0 and TD1 are proportional. As a special case, when the current picture is temporarily located between When the time distance between two reference pictures and the current picture to the two reference pictures is the same, bilateral matching becomes a mirror-based bidirectional MV.

As shown in Figure 22, by finding the closest match between the template in the current picture (the top and/or left adjacent block of the current CU) and the block in the reference picture (the same size as the template), the template matching is used to Derive the current CU sports information. In addition to the aforementioned FRUC Merge mode, template matching is also applied to AMVP mode. In JEM, just as in HEVC, AMVP has two candidates. Using the template matching method, new candidates are derived. If the newly derived candidate from the template matching is different from the first existing AMVP candidate, insert it at the very beginning of the AMVP candidate list, and then set the list size to 2 (ie remove the second existing AMVP candidate). When applied to AMVP mode, only CU-level search is applied.

2.3.3.1 CU level MV Candidate set

The CU-level MV candidate set includes:

(I) Original AMVP candidate, if the current CU is in AMVP mode,

(Ii) All Merge candidates,

(Iii) Interpolate several MVs in the MV field.

(Iv) Top and left adjacent motion vectors

When using bilateral matching, each valid MV of the Merge candidate is used as input to generate MV pairs that are assumed to be bilateral matching. For example, a valid MV of the Merge candidate at the reference list A is (MVa, ref _a ). Then find the reference picture ref _{b of} its paired bilateral MV in another reference list B, so that ref _a and ref _b are located on different sides of the current picture in time. If the reference ref _b in the reference list B is not available, the reference ref _{b is} determined to be _a different reference from the reference ref _a , and its time distance to the current picture is the smallest distance in the list B. After the reference ref _{b is} determined, the MVb is derived by scaling MVa based on the time distance between the current picture and the reference ref _a and the reference ref _b .

Four MVs from the interpolated MV field are also added to the CU-level candidate list. More specifically, the interpolated MVs at positions (0, 0), (W/2, 0), (0, H/2), and (W/2, H/2) of the current CU are added.

When FRUC is applied in AMVP mode, the original AMVP candidates are also added to the MV candidate set at the CU level.

At the CU level, up to 15 MVs of AMVP CU and up to 13 MVs of Merge CU can be added to the candidate list.

2.3.3.2 child CU level MV Candidate set

The MV candidates set at the sub-CU level include:

(I) MV determined from CU-level search,

(Ii) MVs adjacent to the top, left, top left, and top right,

(Iii) A scaled version of the collocated MV from the reference picture,

(Iv) Up to 4 ATMVP candidates,

(V) Up to 4 STMVP candidates.

The zoomed MV from the reference picture is derived as follows. All reference pictures in the two lists are traversed. The MV at the collocated position of the sub-CU in the reference picture is scaled to the reference of the starting CU-level MV.

The ATMVP and STMVP candidates are limited to the first four. At the sub-CU level, up to 17 MVs are added to the candidate list.

2.3.3.3 Interpolation MV Field generation

Before encoding the frame, an interpolated motion field of the entire picture is generated based on the one-way ME. Then, the sports field can be subsequently used as a CU-level or sub-CU-level MV candidate.

First, the motion field of each reference picture in the two reference lists is traversed at the 4×4 block level. For each 4×4 block, if the motion associated with the block passes through a 4×4 block in the current picture (as shown in Figure 23), and the block is not assigned any interpolated motion, then according to the time distance TD0 and TD1 Scale the motion of the reference block to the current picture (the same as the MV scaling of TMVP in HEVC), and assign the scaling motion to the block in the current frame. If an unscaled MV is assigned to a 4×4 block, the motion of the block is marked as unusable in the interpolated motion field.

2.3.3.4 Imputation matching cost

When the motion vector points to the fractional sampling position, motion compensation interpolation is required. In order to reduce the complexity, bilinear interpolation is used for both bilateral matching and template matching instead of conventional 8-tap HEVC interpolation.

The calculation of the matching cost is a bit different at different steps. When selecting candidates from the CU-level candidate set, the matching cost is the absolute sum difference (SAD) of bilateral matching or template matching. After the initial MV is determined, the matching cost C for bilateral matching at the sub-CU level is calculated as follows:

（4）

(4)

分別指示當前MV和起始MV。仍然將SAD用作模式匹配在子CU級搜索的匹配成本。Here, w is the weight coefficient, which is empirically set to 4. MV and

Indicate the current MV and start MV respectively. SAD is still used as the matching cost of pattern matching in the sub-CU level search.

In FRUC mode, MV is derived by using only luminance samples. The derived motion will be used for MC inter prediction of luma and chroma. After determining the MV, an 8-tap (8-taps) interpolation filter is used for luminance and a 4-tap (4-taps) interpolation filter is used for chroma to perform the final MC.

2.3.3.5 MV Refine

MV refinement is a pattern-based MV search, using bilateral cost or template matching cost as the standard. In JEM, two search modes are supported—unlimited center-biased diamond search (UCBDS) and self-adjusting cross search. MV refinement is performed at the CU level and the sub-CU level respectively. For the MV refinement at the CU level and the sub-CU level, the MV is directly searched at the accuracy of a quarter of the luminance sample, followed by the MV refinement of the one-eighth luminance sample. Set the search range of MV refinement in the CU and sub-CU steps to 8 luminance samples.

2.3.3.6 Template matching FRUC Merge Selection of prediction direction in mode

In the bilateral Merge mode, two-way prediction is always applied, because the motion information of the CU is obtained based on the closest match between two blocks on the current CU motion trajectory in two different reference pictures. There is no such limitation for the template matching Merge mode. In the template matching Merge mode, the encoder can make a choice for the CU from the one-way prediction of List 0, the one-way prediction of List 1, or the two-way prediction. This selection is based on the following template matching costs:

If costBi>=factor*min(cost0,cost1)

Bidirectional prediction is used;

Otherwise, if cost0>=cost1

Then use the one-way prediction in list 0;

otherwise,

Use the one-way forecast in Listing 1;

Where cost0 is the SAD matched by the template in Listing 0, cost1 is the SAD matched by the template in Listing 2, and costBi is the SAD matched by the bidirectional prediction template. The value of factor is equal to 1.25, which means that the selection process is shifted towards bidirectional prediction. Inter-frame prediction direction selection can only be applied to CU-level template matching processing.

2.3.4 Decoder side motion vector refinement

In the bidirectional prediction operation, for the prediction of a block region, two prediction blocks respectively formed by the motion vector (MV) of list 0 and the MV of list 1 are combined to form a single prediction signal. In the decoder-side motion vector refinement (DMVR) method, the two motion vectors of bidirectional prediction are further refined through bilateral template matching processing. The bilateral template matching applied in the decoder is used to perform a distortion-based search between the bilateral template and the reconstructed samples in the reference picture in order to obtain a refined MV without transmitting additional motion information.

In DMVR, the bilateral template is generated as a weighted combination (ie, average) of two prediction blocks, where the two prediction blocks are from the initial MV0 in list 0 and MV1 in list 1. The template matching operation includes calculating the cost metric between the generated template and the sample area (around the initial prediction block) in the reference picture. For each of the two reference pictures, the MV that produces the smallest template cost is regarded as the updated MV of the list to replace the original MV. In JEM, nine MV candidates are searched for each list. The nine MV candidates include the original MV and 8 peripheral MVs, and the eight peripheral MVs have a luminance sample offset from the original MV in the horizontal or vertical direction or both. Finally, use the two new MVs shown in Figure 24 (namely MV0' and MV1') to generate the final bidirectional prediction result. The sum of absolute differences (SAD) is used as a cost metric.

Without transmitting additional syntax elements, DMVR is applied to the Merge mode of bidirectional prediction, where one MV is from a reference picture in the past and the other MV is from a reference picture in the future. In JEM, when LIC, affine motion, FRUC, or sub-CU Merge candidates are enabled for CU, DMVR is not applied.

2.3.5 Local illumination compensation

Local illumination compensation (IC) is based on a linear model for illumination changes, using a scaling factor a and an offset b. And for each inter-mode coding unit (CU), the local illumination compensation is enabled or disabled by self-adjustment.

When the IC is applied to the CU, the least square error method is adopted to derive the parameters a and b by using the neighboring samples of the current CU and the corresponding reference samples. More specifically, as shown in FIG. 25, the neighboring samples of the sub-sampling (2:1 sub-sampling) of the CU and the corresponding samples (identified by the motion information of the current CU or sub-CU) in the reference picture are used. IC parameters are derived and applied to each prediction direction separately.

When the CU is encoded in the Merge mode, the IC flag is copied from adjacent blocks in a similar manner to the motion information copy in the Merge mode; otherwise, the IC flag is signaled to the CU to indicate whether the LIC is applicable.

When IC is enabled for pictures, an additional CU level RD check is required to determine whether to apply LIC to the CU. When the IC is enabled for CU, for integer-pixel motion search and fractional pixel motion search were used and the mean absolute difference is removed (Mean-Removed Sum of Absolute Diffefference, MR-SAD) and Mean Absolute Hadamard transform and remove the difference (Mean- Removed Sum of Absolute Hadamard-Transformed Difference, MR-SATD) instead of SAD and SATD.

In order to reduce coding complexity, the following coding scheme is applied in JEM. When there is no obvious lighting change between the current picture and its reference picture, IC is disabled for all pictures. In order to recognize this situation, the bar graph of the current picture and each reference picture of the current picture are calculated at the encoder. If the bar graph difference between the current picture and each reference picture of the current picture is less than a given threshold, IC is disabled for the current picture; otherwise, IC is enabled for the current picture.

2.3.6 With two-way matching refinement Merge/ Skip mode

First, the Merge candidate list is constructed by inserting the motion vectors and reference indexes of the spatially adjacent and temporally adjacent blocks into the candidate list by using redundancy check, until the number of available candidates reaches the maximum candidate size of 19. By inserting spatial candidates (Figure 26), temporal candidates, affine candidates, advanced temporal MVP (Advanced Temporal MVP, ATMVP) candidates, spatiotemporal MVP (Spatial Temporal, STMVP) candidates and HEVC according to the predefined insertion order, Additional candidates (combined candidates and zero candidates) are added to construct the Merge candidate list of Merge/Skip mode:

(1) Spatial candidates for blocks 1-4

(2) Extrapolated affine candidates for blocks 1-4

(3) ATMVP

(4) STMVP

(5) Virtual affine candidate

(6) Spatial candidates (block 5) (only used when the number of available candidates is less than 6)

(7) Extrapolate Affine Candidates (Block 5)

(8) Time candidate (as derived in HEVC)

(9) Non-adjacent spatial candidates, followed by extrapolated affine candidates (blocks 6 to 49)

(10) Combination candidates

(11) Zero candidates

Note that in addition to STMVP and affine, the IC flag is also inherited from the Merge candidate. Also, for the first four spatial candidates, bi-directional prediction candidates are inserted before the candidates with uni-directional prediction.

2.3.7 JVET-K0161

In this proposal, the sub-block STMVP is not proposed as the space-time Merge mode. The proposed method uses co-located blocks, which is the same as HEVC/JEM (only 1 picture, no time vector here). The proposed method also checks the upper and left spatial positions, which are adjusted in the proposal. Specifically, in order to check adjacent inter-frame prediction information, at most two positions are checked for each upper and left side. The exact location is shown in Figure 27.

Afar: (nPbW * 5 / 2, -1), Amid (nPbW / 2, -1) (Note: the offset of the upper space block above the current block)

Lfar: (-1, nPbH * 5 / 2), Lmid (-1, nPbH/2) (Note: the offset of the left space block above the current block)

The average value of the motion vectors of the upper block, the left block and the time block are calculated in the same way as the BMS software implementation. If 3 reference inter prediction blocks are available.

mvLX[0] = ((mvLX_A[0] + mvLX_L[0] + mvLX_C[0]) * 43) / 128

mvLX[1] = ((mvLX_A[1] + mvLX_L[1] + mvLX_C[1]) * 43) / 128

If only two or one inter prediction blocks are available, the average of the two is used or only one mv is used.

2.3.8 JVT-K0135

In order to generate a smooth and subtle motion field, Figure 28 gives a brief description of the plane motion vector prediction process.

The plane motion vector prediction is realized by averaging the horizontal and vertical linear interpolation on the basis of 4×4 blocks as follows.

是當前子塊的運動向量。 W and H represent the width and height of the block. (x,y) is the coordinates of the current sub-block relative to the upper-left sub-block. All distances are represented by the pixel distance divided by 4.

Is the motion vector of the current sub-block.

和垂直預測

計算如下：Level prediction of location (x,y)

And vertical prediction

The calculation is as follows:

和

是當前塊左側和右側的4x4塊的運動向量。

和

是當前塊上方和底部的4x4塊的運動向量。among them

with

Is the motion vector of the 4x4 block on the left and right of the current block.

with

Is the motion vector of the 4x4 block above and below the current block.

The reference motion information of the adjacent blocks in the left column and the upper row is derived from the spatial adjacent blocks of the current block.

The reference motion information of adjacent blocks in the right column and bottom row is derived as follows.

Derive motion information of adjacent 4×4 blocks at the bottom right

Use the derived motion information of the lower right adjacent 4×4 block and the motion information of the upper right adjacent 4×4 block to calculate the motion vector of the adjacent 4×4 block in the right column, as described in formula K1.

Use the derived motion information of the lower right adjacent 4×4 block and the lower left adjacent 4×4 block to calculate the motion vector of the bottom row adjacent 4×4 block, as described in formula K2.

To

AR + (y+1)

BR)/HR(W,y) = ((Hy-1)

AR + (y+1)

BR)/H Formula K1 To B(x,H) = ((Wx-1)

BL+ (x+1)

BR)/W Formula K2

AR is the motion vector of the adjacent 4×4 block in the upper right space, BR is the motion vector of the adjacent 4×4 block in the lower right space, and BL is the motion vector of the adjacent 4×4 block in the lower left space.

For each list, the motion information obtained from adjacent blocks is scaled to the first reference picture of a given list.

3. Examples of problems solved by the embodiments disclosed herein

The inventor has previously proposed a motion vector prediction technology based on a look-up table, which uses one or more look-up tables storing at least one motion candidate to predict the motion information of a block. It can be implemented in various embodiments to provide more Video coding with high coding efficiency. Each LUT may include one or more motion candidates, and each motion candidate is associated with corresponding motion information. The motion information of the motion candidate may include prediction direction, reference index/picture, motion vector, LIC flag, affine flag, motion vector difference (MVD) accuracy and/or MVD value. The motion information may also include block position information to indicate where the motion information comes from.

The LUT-based motion vector prediction based on the disclosed technology can enhance existing and future video coding standards, which are illustrated in the examples described below for various implementations. Because the LUT allows the encoding/decoding process to be performed based on historical data (for example, blocks that have been processed), the LUT-based motion vector prediction may also be referred to as a history-based motion vector prediction (HMVP) method. In the LUT-based motion vector prediction method, one or more tables with motion information from previously encoded blocks are maintained during the encoding/decoding process. These motion candidates stored in the LUT are named HMVP candidates. During encoding/decoding of a block, the associated motion information in the LUT may be added to the motion candidate list (for example, Merge/AMVP candidate list), and after encoding/decoding a block, the LUT may be updated. The updated LUT is then used to encode subsequent blocks. That is, the update of the motion candidate in the LUT is based on the encoding/decoding order of the block. The following examples should be regarded as examples explaining general concepts. These examples should not be interpreted in a narrow way. In addition, these examples can be combined in any manner.

Some embodiments may use one or more lookup tables storing at least one motion candidate to predict the motion information of the block. The embodiment may use motion candidates to indicate a set of motion information stored in the lookup table. For the traditional AMVP or Merge mode, embodiments may use AMVP or Merge candidates to store motion information.

Although the current LUT-based motion vector prediction technology overcomes the shortcomings of HEVC by using historical data, it only considers information from spatially adjacent blocks.

When the motion candidate from the LUT is used in the AMVP or Merge list building process, it is directly inherited without any change.

The design of JVET-K0161 is beneficial to coding performance. However, it requires additional derivation of TMVP, which increases computational complexity and memory bandwidth.

4. Some examples

The following examples should be regarded as examples explaining general concepts. These examples should not be interpreted in a narrow way. In addition, these examples can be combined in any manner.

Some embodiments using the currently disclosed technology may jointly use motion candidates from LUTs and motion information from temporal neighboring blocks. In addition, the complexity of JVET-K0161 is reduced.

Use from LUT Motion candidates

1. Propose to construct new AMVP/Merge candidates by using motion candidates from LUT.

a. In an example, a new candidate can be derived by adding/subtracting an offset (or multiple offsets) to the motion vector of the motion candidate from the LUT.

b. In an example, a new candidate can be derived by averaging the motion vectors of the selected motion candidates from the LUT.

/2 ^N 或(MVa+MVb+MVc)×

/2 ^N 。例如，當N = 7時，平均值是(MVa+MVb+MVc) ×42/128或(MVa+MVb+MVc) ×43/128。請注意，預先計算

或

並將其存儲在查閱資料表中。i. In one embodiment, averaging can be achieved approximately without a division operation. For example, MVa, MVb and MVc can be averaged to (MVa+MVb+MVc)×

/2 ^N or (MVa+MVb+MVc)×

/2 ^N. For example, when N = 7, the average value is (MVa+MVb+MVc) ×42/128 or (MVa+MVb+MVc) ×43/128. Note that pre-calculated

or

And store it in the lookup table.

ii. In one example, only motion vectors with the same reference picture (in two prediction directions) are selected.

iii. In one example, the reference picture in each prediction direction is predetermined, and if necessary, the motion vector is scaled to the predetermined reference picture.

1. In an example, the first entry (X=0 or 1) in the reference picture list X is selected as the reference picture.

2. Alternatively, for each prediction direction, the most frequently used reference picture in the LUT is selected as the reference picture.

c. In one example, for each prediction direction, first select a motion vector having the same reference picture as a predetermined reference picture, and then select other motion vectors.

2. Propose to construct a new AMVP/Merge candidate through a function of one or more motion candidates from the LUT and motion information from temporal neighboring blocks.

a. In an example, similar to STMVP or JVET-K0161, new candidates can be derived by averaging the motion candidates from LUT and TMVP.

b. In an example, the above-mentioned blocks (for example, Amid and Afar in Figure 27) can be replaced by one or more candidates from the LUT. Alternatively, in addition, other processes can remain unchanged, as already implemented in JVET-K0161.

3. Propose a function of one or more motion candidates from the LUT, AMVP/Merge candidates from spatial neighboring blocks and/or spatially non-neighboring neighboring blocks, and motion information from temporal blocks to construct a new AMVP /Merge candidate.

a. In one example, one or more of the above blocks (for example, Amid and Afar in Figure 27) can be replaced by candidates from the LUT. Alternatively, in addition, other processes can remain unchanged, as already implemented in JVET-K0161.

b. In one example, one or more of the blocks on the left (for example, Amid and Afar in Figure 27) can be replaced by candidates from the LUT. Alternatively, in addition, other processes can remain unchanged, as already implemented in JVET-K0161.

4. It is proposed that when the motion information of the block is inserted into the LUT, whether to trim the existing entries in the LUT may depend on the coding mode of the block.

a. In one example, if the block is encoded in Merge mode, no trimming is performed.

b. In one example, if the block is coded in AMVP mode, no trimming is performed.

c. In an example, if the block is coded in AMVP/Merge mode, only the latest M entries of the LUT are trimmed.

d. In one example, when a block is encoded in a sub-block mode (for example, affine or ATMVP), trimming is always disabled.

5. Propose adding motion information from the time block to the LUT.

a. In one example, the motion information can come from a co-located block.

b. In one example, the motion information can come from one or more blocks from different reference pictures.

versus STMVP Related

1. It is proposed to always use spatial Merge candidates to derive new Merge candidates, regardless of TMVP candidates.

a. In one example, the average of two motion Merge candidates can be used.

b. In an example, the spatial Merge candidate and the motion candidate from the LTU can be jointly used to derive a new candidate.

2. Propose that STMVP candidates can be derived using non-neighboring blocks (which are not right or left neighboring blocks).

a. In one example, the upper block used for STMVP candidate derivation remains unchanged, while the left block used is changed from a neighboring block to a non-neighboring block.

b. In one example, the left block used for STMVP candidate derivation remains unchanged, while the upper block used is changed from a neighboring block to a non-neighboring block.

c. In one example, a new candidate can be derived using the candidate of the non-neighboring block and the motion candidate from the LUT jointly.

3. It is proposed to always use spatial Merge candidates to derive new Merge candidates without considering TMVP candidates.

a. In one example, the average of two motion Merge candidates can be used.

b. Alternatively, the average value of two, three or more MVs from different positions adjacent or not adjacent to the current block can be used.

i. In one embodiment, the MV can only be obtained from the location in the current LCU (also called CTU).

ii. In one embodiment, the MV can only be obtained from the position in the current LCU row.

iii. In one embodiment, the MV can only be obtained from the current LCU row or the position next to the current LCU row. An example is shown in Figure 29. Blocks A, B, C, E, E, and F are next to the current LCU row.

iv. In one embodiment, the MV can only be obtained from the current LCU row or a position next to the current LCU row but not to the left of the adjacent block in the upper left corner. An example is shown in Figure 29. Block T is the adjacent block in the upper left corner. Blocks B, C, E, E, and F are next to the current LCU row, but not to the left of the adjacent block in the upper left corner.

c. In one embodiment, the spatial Merge candidate and the operation from the LTU can be used jointly

Move candidates to derive new candidates

4. Propose that the MV of the BR block used for plane motion prediction in Fig. 28 is not obtained from temporal MV prediction, but from an entry in the LUT.

5. Proposed motion candidates from the LUT can be used in conjunction with other types of Merge/AMVP candidates (for example, spatial Merge/AMVP candidates, temporal Merge/AMVP candidates, default motion candidates) to derive new candidates.

In various implementations of this example and other examples disclosed in this patent file, trimming may include: a) uniquely comparing sports information with existing entries, and b) if unique, adding sports information to the list, or c) If it is not unique, either c1) do not add sports information, or c2) add sports information and delete matching existing entries. In some implementations, when a motion candidate is added from the table to the candidate list, the trim operation is not invoked.

FIG. 30 is a schematic diagram illustrating an example of the structure of a computer system or other control device 3000 that can be used to implement various parts of the disclosed technology. In FIG. 30, the computer system 3000 includes one or more processors 3005 and a memory 3010 connected through an internal connection 3025. The internal connection 3025 may represent any one or more individual physical buses, point-to-point connections, or both connected by appropriate bridges, adapters, or controllers. Therefore, the internal connection 3025 may include, for example, a system bus, a peripheral component connection (PCI) bus, a hypertransmission or industry standard architecture (ISA) bus, a small computer system interface (SCSI) bus, and a universal serial bus (USB) , IIC (I2C) bus or Institute of Electrical and Electronics Engineers (IEEE) standard 674 bus (sometimes called "FireWire").

The processor 3005 may include a central processing unit (CPU) to control, for example, the overall operation of the host. In some embodiments, the processor 3005 implements this by executing software or firmware stored in the memory 3010. The processor 3005 may be or may include one or more programmable general-purpose or special-purpose microprocessors, digital signal processors (DSP), programmable controllers, special integrated circuits (ASICs), programmable logic devices ( PLD), etc., or a combination of these devices.

The memory 3010 may be or include the main memory of the computer system. The memory 3010 represents any suitable form of random access memory (RAM), read-only memory (ROM), flash memory, etc., or a combination of these devices. In use, the memory 3010 may contain, among other things, a set of machine instructions, and when the processor 3005 executes the instructions, the processor 3005 executes operations to implement the embodiments of the disclosed technology.

Also connected to the processor 3005 via the internal connection 3025 is an (optional) network interface card 3015. The network interface card 3015 provides the computer system 3000 with the ability to communicate with remote devices (such as storage clients and/or other storage servers), and may be, for example, an Ethernet adapter or a fiber channel adapter.

FIG. 31 shows a block diagram of an example embodiment of a mobile device 3100 that can be used to implement various parts of the disclosed technology. The mobile device 3100 may be a notebook computer, a smart phone, a tablet computer, a video camera, or other devices capable of processing video. The mobile device 3100 includes a processor or controller 3101 to process data, and a memory 3102 in communication with the processor 3101 to store and/or buffer data. For example, the processor 3101 may include a central processing unit (CPU) or a microcontroller unit (MCU). In some implementations, the processor 3101 may include a field programmable gate array (FPGA). In some implementations, the mobile device 3100 includes or communicates with a graphics processing unit (GPU), a video processing unit (VPU), and/or a wireless communication unit to implement various visual and/or communication data processing functions of the smartphone device. For example, the memory 3102 may include and store processor executable code. When the processor 3101 executes the code, the mobile device 3100 is configured to perform various operations, such as receiving information, commands and/or data, processing information and data, and Send or provide the processed information/data to another data device, such as an actuator or an external display. In order to support various functions of the mobile device 3100, the memory 3102 can store information and data, such as instructions, software, values, images, and other data processed or referenced by the processor 3101. For example, various types of random access memory (RAM) devices, read-only memory (ROM) devices, flash memory devices, and other suitable storage media may be used to implement the storage function of the memory 3102. In some implementations, the mobile device 3100 includes an input/output (I/O) unit 3103 to interface the processor 3101 and/or the memory 3102 with other modules, units, or devices. For example, the I/O unit 3103 can interface with the processor 3101 and the memory 3102 to utilize various wireless interfaces compatible with typical data communication standards, for example, among one or more computers and user equipment in the cloud. between. In some implementations, the mobile device 3100 can interface with other devices through the I/O unit 3103 using a wired connection. The mobile device 3100 can also be connected with other external interfaces (such as data memory) and/or a visual or audio display device 3104 for retrieval and transmission, which can be processed by the processor, stored by the memory, or output by the display device 3104 or external device. Data and information displayed on For example, the display device 3104 may display a video frame including a block (CU, PU, or TU) to which intra block copy is applied based on whether the block is encoded using a motion compensation algorithm according to the disclosed technology.

In some embodiments, a video decoder device that can implement the sub-block-based prediction method as described herein can be used for video decoding.

In some embodiments, a decoding device implemented on a hardware platform as described in FIG. 30 and FIG. 31 may be used to implement the video decoding method.

The various embodiments and techniques disclosed in this document can be described in the list of examples below.

Figure 32 is a flowchart of an example method 3200 for video processing in accordance with the currently disclosed technology. The method 3200 includes, in operation 3202, determining a new candidate for video processing by averaging two or more selected motion candidates. The method 3200 includes, at operation 3204, adding the new candidate to a candidate list. The method 3200 includes, in operation 3206, performing a conversion between the first video block of the video and the bitstream representation of the video by using the determined new candidate in the candidate list.

In some embodiments, the candidate list is a Merge candidate list, and the determined new candidate is a Merge candidate.

In some embodiments, the Merge candidate list is an inter prediction Merge candidate list or an intra block copy prediction Merge candidate list.

In some embodiments, the one or more tables include motion candidates derived from previously processed video blocks that were processed before the first video block in the video material.

In some embodiments, there are no spatial and temporal candidates available in the candidate list.

In some embodiments, the selected motion candidates are from one or more tables.

In some embodiments, the averaging is achieved without a division operation.

In some embodiments, the averaging is achieved through multiplication of the sum of the motion vectors of the selected motion candidates and the scaling factor.

In some embodiments, the level components of the motion vectors of the selected motion candidates are averaged to derive the level components of the new candidate.

In some embodiments, the vertical components of the motion vectors of the selected motion candidates are averaged to derive the vertical components of the new candidate.

In some embodiments, the zoom factor is pre-calculated and stored in the lookup table.

In some embodiments, only motion vectors with the same reference picture are selected.

In some embodiments, only motion vectors with the same reference picture in the two prediction directions are selected in the two prediction directions.

In some embodiments, the target reference picture in each prediction direction is predetermined, and the motion vector is scaled to the predetermined reference picture.

In some embodiments, the first entry in the reference picture list X is selected as the target reference picture for the reference picture list, and X is 0 or 1.

In some embodiments, for each prediction direction, the most frequently used reference picture in the table is selected as the target reference picture.

In some embodiments, for each prediction direction, a motion vector having the same reference picture as the predetermined target reference picture is selected first, and then other motion vectors are selected.

In some embodiments, the motion candidates from the table are associated with motion information, and the motion information includes at least one of the following: prediction direction, reference picture index, motion vector value, intensity compensation flag, affine flag, motion vector difference accuracy, or Motion vector difference.

In some embodiments, the method 3200 further includes updating one or more tables based on the conversion.

In some embodiments, the updating of the one or more tables includes updating the one or more tables based on the motion information of the first video block of the video after performing the conversion.

In some embodiments, the method 3200 further includes performing a conversion between subsequent video blocks of the video and the bitstream representation of the video based on the updated table.

In some embodiments, the conversion includes encoding processing and/or decoding processing.

In some embodiments, the video encoding device may perform the method 2900 described herein and other methods during reconstruction of the video for subsequent videos.

In some embodiments, the device in the video system may include a processor configured to perform the methods described herein.

In some embodiments, the described method may be embodied as computer executable code stored on a computer readable program medium.

FIG. 33 is a flowchart of an example method 3300 for video processing according to the currently disclosed technology. The method 3300 includes at operation 3302, determining a new motion candidate for video processing by using one or more motion candidates from one or more tables, where the table includes one or more motion candidates, and each motion candidate is associated Sports information. The method 3300 includes, at operation 3304, performing a conversion between the video block and the encoded representation of the video block based on the new candidate.

In some embodiments, the new motion candidate is derived by adding or subtracting an offset to the motion vector associated with the motion candidate from the one or more tables.

In some embodiments, determining the new motion candidate includes determining the new motion candidate as a function of one or more motion candidates from one or more tables and motion information from temporal neighboring blocks.

In some embodiments, determining the new motion candidate includes averaging the motion candidates from one or more tables and the temporal motion vector predictor.

In some embodiments, averaging the selected motion candidates includes a weighted average or average of the motion vectors associated with the selected motion candidates.

In some embodiments, the averaging is achieved without a division operation.

In some embodiments, the averaging is achieved by the sum of the motion vectors of the motion candidates from the one or more tables and the multiplication of the temporal motion vector predictor and the scaling factor.

In some embodiments, the level component of the motion vector of the motion candidate from the one or more tables is averaged with the temporal motion vector predictor to derive the level component of the new motion candidate.

In some embodiments, averaging the selected level components includes a weighted average or average of the level components associated with the selected motion candidate.

In some embodiments, averaging the selected vertical components includes a weighted average or average of the vertical components associated with the selected motion candidate.

In some embodiments, as one or more motion candidates from one or more tables, Merge candidates from spatial neighboring blocks and/or spatially non-neighboring neighboring blocks, and motion information from temporal neighboring blocks Function to determine new motion candidates.

In some embodiments, determining new candidates includes: Advanced Motion Vector Prediction (AMVP) as one or more motion candidates from one or more tables, from spatial neighboring blocks and/or spatially non-neighboring neighboring blocks Candidate functions and motion information from temporal neighboring blocks are used to determine new motion candidates.

In some embodiments, determining new candidates includes: one or more motion candidates from one or more tables, and AMVP candidates in the advanced motion vector prediction (AMVP) candidate list or Merge candidates in the Merge candidate list Function to determine new motion candidates.

In some embodiments, the new motion candidate is added to the Merge candidate list.

In some embodiments, the new motion candidate is added to the AMVP candidate list.

In some embodiments, each of the one or more tables includes a set of motion candidates, where each motion candidate is associated with corresponding motion information.

In some embodiments, the motion candidate is associated with motion information, and the motion information includes at least one of the following: prediction direction, reference picture index, motion vector value, intensity compensation flag, affine flag, motion vector difference accuracy, or motion vector Difference.

In some embodiments, the method further includes updating one or more tables based on the conversion.

In some embodiments, updating the one or more tables includes updating the one or more tables based on the motion information of the first video block after performing the conversion.

In some embodiments, the method further includes performing a conversion between subsequent video blocks of the video and the bitstream representation of the video based on the updated table.

Figure 34 is a flowchart of an example method 3400 for video processing in accordance with the currently disclosed technology. Method 3400 includes in operation 3402, by always using motion information from more than one spatial neighboring block of the first video block in the current picture, and not using motion information from temporal blocks in a picture different from the current picture. Identify new candidates for video processing. The method 3400 includes in operation 3404, performing a conversion between the first video block in the current picture of the video and the bitstream representation of the video by using the determined new candidate.

In some embodiments, the determined new candidate is added to a candidate list, which includes a Merge candidate list or an advanced motion vector prediction (AMVP) candidate list.

In some embodiments, the motion information from more than one spatial neighboring block is a candidate derived from a predefined spatial neighboring block relative to the first video block in the current picture, or from one or more tables Motion candidates.

In some embodiments, the one or more tables include motion candidates derived from previously processed video blocks processed before the first video block in the video material.

In some embodiments, the candidates derived from the predefined spatial neighboring blocks relative to the first video block in the current picture are spatial Merge candidates.

In some embodiments, the new candidate is derived by averaging at least two spatial Merge candidates.

In some embodiments, the new candidate is derived by jointly using spatial Merge candidates and motion candidates from one or more tables.

In some embodiments, the new candidate is derived by averaging at least two motion vectors associated with the candidate derived from different positions.

In some embodiments, the different position is adjacent to the first video block.

In some embodiments, the motion vector is acquired only from the position in the current largest coding unit to which the first video block belongs.

In some embodiments, the motion vector is obtained only from the position in the current largest coding unit row.

In some embodiments, the motion vector is obtained only from a position in or next to the current largest coding unit row.

In some embodiments, the motion vector is obtained only from a position in the current maximum coding unit row or next to the current maximum coding unit row but not to the left of the adjacent block in the upper left corner.

In some embodiments, the motion vector of the lower right block used for planar motion prediction is not obtained from temporal motion vector prediction candidates, but is obtained from an entry in the table.

In some embodiments, the new candidate is derived by jointly using motion candidates from one or more tables and other kinds of Merge/AMVP candidates.

In some embodiments, the motion candidates in the one or more tables are associated with motion information, and the motion information includes at least one of the following: prediction direction, reference picture index, motion vector value, intensity compensation flag, simulation Shooting flag, motion vector difference accuracy or motion vector difference value.

In some embodiments, the method further includes updating one or more tables based on the conversion.

In some embodiments, updating one or more tables includes updating one or more tables based on the motion information of the first video block after performing the conversion.

In some embodiments, the method further includes performing a conversion between subsequent video blocks of the video and the bitstream representation of the video based on the updated table.

In some embodiments, the conversion includes an encoding process and/or a decoding process.

Figure 35 is a flowchart of an example method 3500 for video processing in accordance with the currently disclosed technology. The method 3500 includes in operation 3502, by using motion information from at least one spatially non-neighboring block of the first video block in the current picture, Next to the other candidates derived from the block to determine new candidates for video processing. The method 3500 includes, in operation 3504, performing a conversion between the first video block of the video and the bitstream representation of the video by using the determined new candidate.

In some embodiments, the determined new candidate is added to a candidate list, which includes a Merge or Advanced Motion Vector Prediction (AMVP) candidate list.

In some embodiments, the motion information from more than one spatially non-neighboring block is a candidate derived from a predefined spatially non-neighboring block relative to the first video block in the current picture.

In some embodiments, the candidate derived from the predefined spatially non-neighboring block relative to the first video block in the current picture is a spatial-temporal motion vector prediction (STMVP) candidate.

In some embodiments, the non-neighboring block of the video block is not the right neighboring block or the left neighboring block of the first video block.

In some embodiments, the upper block used for STMVP candidate derivation of the first video block remains unchanged, and the used left block is changed from a neighboring block to a non-neighboring block.

In some embodiments, the left block used for STMVP candidate derivation of the first video block remains unchanged, and the upper block used is changed from a neighboring block to a non-neighboring block.

Figure 36 is a flowchart of an example method 3600 for video processing in accordance with the currently disclosed technology. The method 3600 includes, in operation 3602, determining a video process for video processing by using motion information from one or more tables of a first video block in the current picture and motion information from a time block in a picture different from the current picture. New candidate. The method 3600 includes, in operation 3604, performing a conversion between the first video block in the current picture of the video and the bitstream representation of the video by using the determined new candidate.

In some embodiments, the determined new candidate is added to a candidate list, which includes a Merge or AMVP candidate list.

In some embodiments, the motion information from one or more tables in the current picture is associated with one or more historical motion vector prediction (HMVP) candidates selected from the one or more tables, and is from a different picture than the current picture. The motion information of the temporal block in the picture is a temporal motion candidate.

In some embodiments, the new candidate is derived by averaging one or more HMVP candidates and one or more temporal motion candidates.

In some embodiments, the one or more tables include motion candidates derived from previously processed video blocks processed before the first video block in the video material.

FIG. 37 is a flowchart of an example method 3700 for video processing according to the currently disclosed technology. The method 3700 includes, in operation 3702, determining a new one for video processing by using motion information from one or more tables of a first video block and motion information from one or more spatial neighboring blocks of the first video block. Candidate. The method 3700 includes, in operation 3704, performing a conversion between the first video block in the current picture of the video and the bitstream representation of the video by using the determined new candidate.

In some embodiments, the determined new candidate is added to a candidate list, which includes a Merge or AMVP candidate list.

In some embodiments, the motion information from one or more tables of the first video block is associated with one or more historical motion vector prediction (HMVP) candidates selected from the one or more tables, and is from the first video The motion information of one or more spatial neighboring blocks of a block is a candidate derived from a predefined spatial block relative to the first video block.

In some embodiments, the candidate derived from the predefined spatial block relative to the first video block is a spatial Merge candidate.

In some embodiments, the new candidate is derived by averaging one or more HMVP candidates and one or more spatial Merge candidates.

In some embodiments, the one or more tables include motion candidates derived from previously processed video blocks processed before the first video block in the video material.

In some embodiments, the motion candidates from the table are associated with motion information, and the motion information includes at least one of the following: prediction direction, reference picture index, motion vector value, intensity compensation flag, affine flag, motion vector difference Precision or motion vector difference.

In some embodiments, the method further includes updating one or more tables based on the conversion.

In some embodiments, updating the one or more tables includes updating the one or more tables based on the motion information of the current video block after performing the conversion.

In some embodiments, the method further includes performing conversion between subsequent video blocks of the video material and the bitstream representation of the video material based on the updated table.

FIG. 38 is a flowchart of an example method 3800 for video processing according to the currently disclosed technology. The method 3800 includes, in operation 3802, maintaining a set of tables, where each table includes motion candidates, and each motion candidate is associated with corresponding motion information; in operation 3804, performing a first video block and including the first video block And in operation 3806, by selectively trimming the existing motion candidates in one or more tables based on the encoding/decoding mode of the first video block to update the one or Multiple tables.

In some embodiments, based on one or more tables in the set of tables, the conversion between the first video block and the bitstream representation of the video including the first video block is performed.

In some embodiments, in the case of encoding/decoding the first video block in Merge mode, trimming is omitted.

In some embodiments, in the case of encoding/decoding the first video block in the advanced motion vector prediction mode, trimming is omitted.

In some embodiments, in the case of encoding/decoding the first video block in Merge mode or advanced motion vector prediction mode, the latest M entries of the table are pruned, where M is a pre-designated integer.

In some embodiments, in the case of encoding/decoding the first video block in sub-block mode, pruning is disabled.

In some embodiments, the sub-block mode includes an affine mode and an optional temporal motion vector prediction mode.

In some embodiments, the pruning includes checking whether there are redundant existing motion candidates in the table.

In some embodiments, the pruning further includes: if there are redundant existing motion candidates in the table, inserting the motion information associated with the first video block into the table, and deleting the The redundant existing motion candidates in the table.

In some embodiments, if there are redundant existing motion candidates in the table, the motion information associated with the first video block is not used to update the table.

In some embodiments, the method further includes performing a conversion between subsequent video blocks of the video and the bitstream representation of the video based on the updated table.

Figure 39 is a flowchart of an example method 3900 for video processing in accordance with the currently disclosed technology. The method 3900 includes, in operation 3902, maintaining a set of tables, where each table includes motion candidates, and each motion candidate is associated with corresponding motion information; in operation 3904, performing a first video block and including the first video block Conversion between the bitstream representations of the video; and in operation 3906, update one or more tables to include the motion information of one or more temporal neighbors from the first video block as a new motion candidate .

In some embodiments, based on one or more tables in the set of tables, the conversion between the first video block and the bitstream representation of the video including the first video block is performed.

In some embodiments, the one or more temporally neighboring blocks are co-located blocks.

In some embodiments, the one or more temporal neighboring blocks include one or more blocks from different reference pictures.

In some embodiments, the method further includes performing a conversion between subsequent video blocks of the video and the bitstream representation of the video based on the updated table.

FIG. 40 is a flowchart of an example method 4000 for updating a motion candidate table according to the currently disclosed technology. The method 4000 includes, in operation 4002, selectively trimming existing motion candidates in the table based on the encoding/decoding mode of the video block being processed, each motion candidate being associated with corresponding motion information; and in operation 4004 , Update the table to include the motion information of the video block as a new motion candidate.

In some embodiments, in the case of encoding/decoding the video block in the Merge mode or the advanced motion vector prediction mode, the latest M entries of the table are pruned, where M is a pre-designated integer.

In some embodiments, in the case of encoding/decoding the video block in sub-block mode, pruning is disabled.

In some embodiments, the sub-block mode includes an affine mode and an optional temporal motion vector prediction mode.

In some embodiments, the pruning includes checking whether there are redundant existing motion candidates in the table.

In some embodiments, the pruning further includes: if there are redundant motion candidates in the table, inserting the motion information associated with the video block being processed into the table, and deleting the table The redundant motion candidates in.

In some embodiments, if there are redundant existing motion candidates in the table, the motion information associated with the video block being processed is not used to update the table.

Figure 41 is a flowchart of an example method 4100 for updating a motion candidate table according to the currently disclosed technology. The method 4100 includes, in operation 4102, maintaining a list of motion candidates, each motion candidate being associated with corresponding motion information; and in operation 4104, updating the table to include one or more times from the video block being processed The motion information of neighboring blocks is used as a new motion candidate.

In some embodiments, the one or more temporally neighboring blocks are co-located blocks.

In some embodiments, the one or more temporal neighboring blocks include one or more blocks from different reference pictures.

In some embodiments, the motion candidate is associated with motion information, and the motion information includes at least one of the following: prediction direction, reference picture index, motion vector value, intensity compensation flag, affine flag, motion vector difference accuracy or motion Vector difference.

In some embodiments, the motion candidate corresponds to a motion candidate of an intra prediction mode used for intra mode encoding.

In some embodiments, the motion candidate corresponds to a motion candidate including brightness compensation parameters used for IC parameter encoding.

Figure 42 is a flowchart of an example method 4200 for video processing in accordance with currently disclosed technology. The method 4200 includes, in operation 4202, determining a new motion candidate for video processing by using one or more motion candidates from one or more tables, where the table includes one or more motion candidates, and each motion The candidate is associated with the motion information; and in operation 4104, based on the new candidate, a conversion is performed between the video block and the encoded representation of the video block.

In some embodiments, the determined new candidate is added to a candidate list, which includes a Merge or Advanced Motion Vector Prediction (AMVP) candidate list.

In some embodiments, determining the new candidate includes: being one or more motion candidates from one or more tables, and an AMVP candidate in an advanced motion vector prediction (AMVP) candidate list or a Merge candidate in a Merge candidate list Function to determine the new motion candidate.

From the above point of view, it should be understood that, for ease of description, specific embodiments of the technology disclosed in the present invention have been described herein, but various modifications can be made without departing from the scope of the present invention. Therefore, in addition to the above, the technology disclosed in the present invention is not limited to the limitation of the scope of patent application.

The embodiments, modules, and functional operations disclosed and other described herein can be implemented in digital electronic circuits, or computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or one of them Or a combination of multiple. The disclosed embodiments and other embodiments can be implemented as one or more computer program products, that is, one or more modules of computer program instructions encoded on a computer-readable medium for the data processing device to execute or control the data processing device Operation. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a storage device, a material composition that affects a machine-readable propagation signal, or a combination of one or more of them. The term "data processing device" includes all devices, equipment, and machines used to process data, including, for example, programmable processors, computers, or multi-processors or computer sets. In addition to hardware, the device may also include code for creating an execution environment for computer programs, for example, code that constitutes processor firmware, protocol stack, database management system, operating system or a combination of one or more of them. Propagated signals are artificially generated signals, such as electrical, optical or electromagnetic signals generated by machines. These signals are generated to encode information for transmission to an appropriate receiving device.

Computer programs (also called programs, software, software applications, scripts or codes) can be written in any form of programming language (including compiled language or interpreted language), and can be deployed in any form, including as stand-alone programs or as modules , Components, subprograms or other units suitable for use in a computing environment. Computer programs do not necessarily correspond to files in the file system. The program can be stored in the part of the file that saves other programs or data (for example, one or more scripts stored in a markup language file), in a single file dedicated to the program, or multiple coordination files (for example, storing one Or multiple modules, subprograms or partial code files). Computer programs can be deployed on one or more computers to be executed. These computers are located on one website or distributed on multiple websites and are interconnected by communication networks.

The processing and logic flow described herein can be executed by one or more programmable processors that execute one or more computer programs that perform functions by operating on input data and generating output. Processing and logic flow can also be performed by special-purpose logic circuits, and the device can also be implemented as special-purpose logic circuits, such as FPGA (Field Programmable Gate Array) or ASIC (dedicated integrated circuit).

For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, and any one or more of any type of digital computer. Generally, the processor will receive commands and data from read-only memory or random access memory or both. The basic components of a computer are a processor that executes instructions and one or more storage devices that store instructions and data. Generally, a computer will also include one or more large storage area devices for storing data, such as floppy disks, magneto-optical disks, or optical discs, or be operatively coupled to one or more large storage area devices to receive data or transfer data from them. Data is transferred to one or more large storage area devices, or both. However, computers do not necessarily have such equipment. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM and flash memory devices; magnetic disks , Such as internal hard disks or removable disks; magneto-optical disks; and CDROM and DVD-ROM discs. The processor and memory can be supplemented by dedicated logic circuits or incorporated into dedicated logic circuits.

It is intended to treat the description together with the drawings as merely exemplary, where exemplary means example. As used herein, the singular forms "a", "an" and "the" are intended to also include the plural forms, unless the context clearly dictates otherwise. In addition, the use of "or" is intended to include "and/or" unless the context clearly dictates otherwise.

Although this patent file contains many details, it should not be construed as a limitation on the scope of any invention or patent application, but as a description of the features of a particular embodiment of a particular invention. Certain features described in this patent document in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various functions described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. In addition, although the above-mentioned features can be described as working in certain combinations, even as originally required, in some cases, one or more of the features in the combination can be deleted from the combination, and the scope of the patent application The combination of can point to a sub-combination or a variant of the sub-combination.

Similarly, although the operations are described in a specific order in the drawings, it should not be understood that such operations must be performed in the specific order or sequence shown, or all the operations described, to obtain the desired results. In addition, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

Only some implementations and examples are described, and other implementations, enhancements and variations can be made based on the content described and illustrated in this patent file.

100: Video codec 3000: Computer system 3005: Processor 3010: Memory 3015: Network interface card 3025: Internal connection 3100: Mobile device 3101: Processor/controller 3102: Memory 3103: Input/output (I /O) Unit 3104: displays 3200, 3300, 3400, 3500, 3600, 3700, 3800, 3900, 4000, 4100, 4200: methods 3202, 3204, 3206, 3302, 3304, 3402, 3404, 3502, 3504, 3602 3604, 3702, 3704, 3802, 3804, 3806, 3902, 3904, 3906, 4002, 4004, 4102, 4104, 4202, 4204: Operation A ₀ , A ₁ , B ₁ , B ₀ , B ₂ , C ₀ , C ₁ : Position a, b, c, d, A, B, C, D, E, F, T: block AL, AR, BL, BR, MV0, MV0′, MV1, MV1′: motion vector tb, td: POC distance TD0, TD1: time distance

FIG. 1 is a block diagram showing an example of implementation of a video transcoder. Figure 2 illustrates the macro block partitioning in the H.264 video coding standard. FIG. 3 illustrates an example of dividing a coding block (CB) into prediction blocks (PB). Figure 4 illustrates an example implementation of subdividing the coding tree block (CTB) into CB and transformation blocks (TB). The solid line represents the CB boundary, and the dashed line represents the TB boundary, including the example CTB with segmentation and the corresponding quadtree. FIG. 5 shows an example of a quadtree binary tree (QTBT) structure used to segment video materials. Fig. 6 shows an example of video block division. Fig. 7 shows an example of quadtree division. Figure 8 shows an example of tree signaling. Fig. 9 shows an example of the derivation process of the Merge candidate list construction. Fig. 10 shows example positions of spatial Merge candidates. FIG. 11 shows an example of a candidate pair considering redundancy check of spatial Merge candidates. FIG. 12 shows an example of the position of the second PU divided by Nx2N and 2NxN. Figure 13 illustrates an example motion vector scaling of a temporal Merge candidate. Fig. 14 shows candidate positions of time-domain Merge candidates and their collocated pictures. FIG. 15 shows an example of combining bidirectional prediction Merge candidates. FIG. 16 shows an example of the derivation process of motion vector prediction candidates. Fig. 17 shows an example of motion vector scaling of spatial motion vector candidates. FIG. 18 shows an example optional temporal motion vector prediction (ATMVP) of motion prediction of a coding unit (CU). Fig. 19 graphically depicts an example of identification of source blocks and source pictures. Fig. 20 shows an example of one CU with four sub-blocks and neighboring blocks. Fig. 21 illustrates an example of bilateral matching. Fig. 22 illustrates an example of template matching. Figure 23 depicts an example of unilateral motion estimation (ME) in frame rate up conversion (FRUC). Fig. 24 shows an example of decoder-side motion vector refinement (DMVR) based on bilateral template matching. FIG. 25 shows an example of spatial neighboring blocks used to derive illumination compensation IC parameters. FIG. 26 shows an example of spatial neighboring blocks used to derive spatial Merge candidates. Fig. 27 shows an example of using adjacent inter prediction blocks. Fig. 28 shows an example of a planar motion vector prediction process. Fig. 29 shows an example of the position next to the current coding unit (CU) line. FIG. 30 is a block diagram illustrating an example of the structure of a computer system or other control device that can be used to implement various parts of the disclosed technology. FIG. 31 shows a block diagram of an example embodiment of a mobile device that can be used to implement various parts of the disclosed technology. FIG. 32 shows a flowchart of an example method for video processing according to the currently disclosed technology. FIG. 33 shows a flowchart of an example method for video processing according to the currently disclosed technology. FIG. 34 shows a flowchart of an example method for video processing according to the currently disclosed technology. FIG. 35 shows a flowchart of an example method for video processing according to the currently disclosed technology. FIG. 36 shows a flowchart of an example method for video processing according to the currently disclosed technology. FIG. 37 shows a flowchart of an example method for video processing according to the currently disclosed technology. FIG. 38 shows a flowchart of an example method for video processing according to the currently disclosed technology. FIG. 39 shows a flowchart of an example method for video processing according to the currently disclosed technology. FIG. 40 shows a flowchart of an example method for updating a motion candidate table according to the currently disclosed technology. FIG. 41 shows a flowchart of an example method for updating a motion candidate table according to the currently disclosed technology. FIG. 42 shows a flowchart of an example method for video processing according to the currently disclosed technology.

3200: method

3202, 3204, 3206: steps

Claims (24)

A method for video processing, including: Determine a new candidate for video processing by averaging two or more selected motion candidates; Adding the new candidate to the candidate list; and By using the determined new candidate in the candidate list, the conversion between the first video block of the video and the bitstream representation of the video is performed. The method according to item 1 of the scope of patent application, wherein the candidate list is a Merge candidate list, and the determined new candidate is a Merge candidate. The method according to item 2 of the scope of patent application, wherein the Merge candidate list is an inter prediction Merge candidate list or an intra block copy prediction Merge candidate list. The method according to item 1 of the scope of patent application, wherein the selected motion candidates come from one or more tables. The method according to item 4 of the scope of patent application, wherein the one or more tables include motion candidates derived from a previously processed video block processed before the first video block in the video material. The method according to item 4 or item 5 of the scope of patent application, wherein there are no available space candidates and time candidates in the candidate list. The method according to item 1 of the scope of patent application, wherein the averaging is realized without division operation. The method according to item 1 of the scope of patent application, wherein the averaging is realized by multiplying the sum of the motion vectors of the selected motion candidates and the scaling factor. The method according to item 1 of the scope of patent application, wherein the level components of the motion vectors of the selected motion candidates are averaged to derive the level components of the new candidates. The method according to item 1 of the scope of patent application, wherein the vertical components of the motion vectors of the selected motion candidates are averaged to derive the vertical components of the new candidates. According to the method described in item 7 of the scope of patent application, the zoom factor is pre-calculated and stored in the look-up data table. The method according to any one of items 1 to 11 of the scope of patent application, wherein only motion vectors with the same reference picture are selected. According to the method described in item 12 of the scope of patent application, in the two prediction directions, only the motion vectors having the same reference picture in the two prediction directions are selected. The method according to any one of items 1 to 10 of the scope of patent application, wherein the target reference picture in each prediction direction is predetermined, and the motion vector is scaled to the predetermined reference picture. According to the method described in item 14 of the scope of patent application, the first entry in the reference picture list X is selected as the target reference picture for the reference picture list, and X is 0 or 1. According to the method described in item 14 of the scope of patent application, for each prediction direction, the most frequently used reference picture in the table is selected as the target reference picture. According to the method described in item 14 of the scope of patent application, for each prediction direction, a motion vector having the same reference picture as a predetermined target reference picture is first selected, and then other motion vectors are selected. The method according to any one of items 1 to 17 of the scope of patent application, wherein the motion candidates from the table are associated with motion information, and the motion information includes at least one of the following: prediction direction, reference picture index, motion vector Value, intensity compensation flag, affine flag, motion vector difference accuracy or motion vector difference value. The method according to any one of items 1 to 17 of the scope of patent application, further comprising: updating one or more tables based on the conversion. According to the method of claim 19, the updating of one or more tables includes updating one or more tables based on the motion information of the first video block of the video after the conversion is performed. According to the method described in item 20 of the scope of patent application, it also includes: Based on the updated table, a conversion between subsequent video blocks of the video and the bitstream representation of the video is performed. The method according to any one of items 1 to 21 of the scope of patent application, wherein the conversion includes encoding processing and/or decoding processing. A device in a video system includes a processor configured to implement the method described in one or more of items 1 to 22 in the scope of the patent application. A non-transitory computer-readable program medium on which codes are stored, the codes including instructions, when a processor executes the instructions, the processor realizes one of items 1 to 22 of the scope of patent application Or multiple methods described in.

Images