TWI719524B - Complexity reduction of non-adjacent merge design - Google Patents

Abstract

Devices, systems and methods for complexity reduction of non-adjacent merge design are described. In a representative aspect, a method for video processing includes receiving a current block of video data, selecting, based on a rule, a first non-adjacent block that is not adjacent to the current block, constructing a merge candidate list including a first merge candidate comprising motion information based on the first non-adjacent block, and processing the current block based on the merge candidate list.

Description

Reduce the complexity of non-adjacent Merge design

In accordance with the applicable “Patent Law” and/or the “Paris Convention”, this application promptly requires the international patent application number PCT/CN2018/093944 filed on July 1, 2018, and the international patent application number filed on September 11, 2018. Priority and benefits of patent application number PCT/CN2018/104982. According to US law, the entire disclosures of International Patent Application No. PCT/CN2018/093944 and International Patent Application No. PCT/CN2018/104982 are incorporated herein by reference as part of the disclosure of this application.

The application documents as a whole directly relate to image and video encoding and decoding technologies.

Digital video occupies the largest bandwidth usage on the Internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, it is expected that the demand for bandwidth used by digital video will continue to grow.

The present invention describes a device, system and method for reducing the complexity of non-adjacent Merge design. For example, the currently disclosed technology discloses a rule for selecting non-adjacent Merge candidates to keep the size of the line buffer below a critical value. The described method can be applied to existing video coding standards (for example, High Efficiency Video Coding (HEVC)) and future video coding standards or video News codec.

In a representative aspect, the disclosed technology can be used to provide a method for video processing. The method includes receiving a current block of video data, selecting a first non-adjacent block that is not adjacent to the current block based on a rule, and constructing a Merge candidate list including a first Merge candidate. The first Merge candidate includes a first non-adjacent block based on , And process the current block based on the Merge candidate list.

In yet another representative aspect, the above method is implemented in the form of processor executable code and stored in a computer-readable program medium.

In yet another representative aspect, a device configured or operable to perform the above method is disclosed. The device may contain a processor programmed to implement the method.

In yet another representative aspect, the video decoder device may implement the method as described herein.

The above and other aspects and features of the disclosed technology are described in more detail in the drawings, specification and claims.

1510: initial list

1520: final list

1800, 2000: coding unit (CU)

1801, 1851, 2001 to 2004: sub coding unit (sub CU)

1850, 2110, 2111, 2210: reference pictures

2011 to 2014: block

2100: current picture

2101, 2102, MV0, MV1, MV0’, MV1’: motion vector

2103, 2104, TD0, TD1: time distance

2200: current coding unit (current CU)

2300: unilateral motion estimation

2800: method

2810 to 2840: steps

2900: Video processing device

2902: processor

2904: memory

2906: Video processing hardware

A ₀ , A ₁ , B ₀ , B ₁ , B ₂ , C ₀ , C ₁ : position; candidate

tb, td: POC distance

Figure 1 shows an example block diagram of a typical high efficiency video coding (HEVC) video encoder and decoder.

Figure 2 shows an example of macro block division in H.264/AVC.

Fig. 3 shows an example of dividing a coding block (CB) into prediction blocks (PB).

4A and 4B respectively show examples of subdividing the coding tree block (CTB) into a CB and a transformation block (TB) and each corresponding quaternary tree.

Figures 5A and 5B illustrate examples of subdivision for the largest coding unit (LCU) and the corresponding QTBT (quaternary tree plus binary tree).

6A to 6E show examples of dividing coding blocks.

Figure 7 shows an example subdivision of CB based on QTBT.

8A to 8I illustrate examples of the division of CBs supporting multiple tree types (MTT), which are generalizations of QTBT.

Figure 9 shows an example of constructing a Merge candidate list.

Fig. 10 shows an example of the position of the spatial candidate.

FIG. 11 shows an example of candidate pairs subjected to redundancy check of spatial Merge candidates.

12A and 12B show examples of the position of the second prediction unit (PU) based on the size and shape of the current block.

Figure 13 shows an example of motion vector scaling for temporal Merge candidates.

Fig. 14 shows an example of candidate positions of temporal Merge candidates.

FIG. 15 shows an example of generating combined bidirectional prediction Merge candidates.

Fig. 16 shows an example of constructing motion vector prediction candidates.

Figure 17 illustrates an example of motion vector scaling for spatial motion vector candidates.

FIG. 18 illustrates an example of using the optional temporal motion vector prediction (ATMVP) algorithm for motion prediction of coding units (CU).

Figure 19 shows an example of the identification of source blocks and source pictures.

FIG. 20 illustrates an example of coding units (CU) with sub-blocks and neighboring blocks used by the space-time motion vector prediction (STMVP) algorithm.

FIG. 21 shows an example of bilateral matching in the pattern matching motion vector derivation (PMMVD) mode, which is a special Merge mode based on the frame rate up conversion (FRUC) algorithm.

Figure 22 shows an example of template matching in the FRUC algorithm.

Figure 23 shows an example of unilateral motion estimation in the FRUC algorithm.

Figure 24 shows an example of a decoder-side motion vector refinement (DMVR) algorithm based on bilateral template matching.

FIG. 25 illustrates an example of spatial neighboring blocks used to derive spatial Merge candidates.

Figure 26 shows an exemplary pseudo code for adding non-adjacent Merge candidates.

Fig. 27 shows an example of restricted areas of non-adjacent blocks.

FIG. 28 shows a flowchart of an example method for video processing according to the currently disclosed technology.

FIG. 29 is a block diagram of an example of a hardware platform for implementing the visual media decoding or visual media encoding technology described in the description of the present invention.

Due to the increasing demand for higher-resolution video, video coding methods and techniques are ubiquitous in modern technology. Video codecs usually include electronic circuits or software that compress or decompress digital video, and are constantly being improved to provide higher coding efficiency. Video codecs convert uncompressed video to compressed format and vice versa. Video quality, the amount of data used to represent the video (determined by the bit rate), the complexity of encoding and decoding algorithms, the sensitivity to data loss and errors, the ease of editing, random access, and end-to-end latency (time There is a complicated relationship between delay). The compression format usually complies with standard video compression specifications, such as the High Efficiency Video Coding (HEVC) standard (also known as H.265 or MPEG-H Part 2), the upcoming general video coding standard, or other current and/or future video Coding standards.

The embodiments of the disclosed technology can be applied to existing video coding standards (for example, HEVC, H.265) and future standards to improve compression performance. The chapter titles are used in the description of the present invention to improve the readability of the description, and the discussion or the embodiments (and/or implementations) are not limited to only the corresponding chapters in any way.

1. Example embodiment of video coding

Figure 1 shows an example block diagram of a typical HEVC video encoder and decoder. The coding algorithm for generating a bit stream conforming to HEVC usually proceeds as follows. Each picture is divided into block regions, where the precise block division is passed to the decoder. The first picture of the video sequence (and to each The first picture at the clean random access point (clean random access point) is only coded using intra-frame prediction (some predictions of spatial data from area to area are used in the same picture, but not dependent on other pictures). For all remaining pictures of the sequence or pictures between random access points, the inter-frame temporal prediction coding mode is usually used for most blocks. The encoding process for inter prediction involves selecting motion data containing selected reference pictures and motion vectors (MV), which will be applied to predict the samples of each block. The encoder and decoder generate the same inter prediction signaling by applying motion compensation (MC) using MV and mode decision data, which is sent as an aid.

The residual signal of intra prediction or inter prediction is transformed by linear spatial transformation, which is the difference between the initial block and its prediction. The transform coefficients are then scaled, quantized, entropy coded, and sent along with the prediction information.

The encoder replicates the decoder processing loop (see the gray shaded box in Figure 1) so that both will produce the same prediction for subsequent data. Therefore, the quantized transform coefficients are constructed by inverse scaling and then inversely transformed to replicate the decoded approximation of the residual signal. The residual is then added to the prediction, and the result of this addition can then be fed into one or two loop filters to smooth out artifacts caused by block-by-block processing and quantization. The final picture representation (ie a copy of the decoder's output) is stored in the decoded picture buffer for prediction of subsequent pictures. Generally, the order of encoding or decoding processing of pictures is usually different from the order in which they arrive from the source; it is necessary to distinguish the decoding order (ie, bit stream order) and output order (ie, display order) of the decoder.

It is generally expected that the video material encoded by HEVC is input as a progressive image (because the source video originates from this format or is generated by deinterlacing before encoding). There are no explicit coding features in the HEVC design to support the use of interlaced scanning, because interlaced scanning is no longer used in displays and becomes largely uncommon for distribution. However, the metadata syntax has been provided in HEVC to allow the encoder to indicate that the interlaced video has been transmitted by encoding each field of the interlaced video (that is, the even or odd lines of each video frame) into a separate picture. Or have been encoded by encoding each interlaced frame into HEVC The code image is transmitted with interlaced video. This provides an effective method for encoding interlaced video without requiring the decoder to support a special decoding process for it.

1.1 Example of partition tree structure in H.264/AVC

The core of the coding layer in the previous standard is a macro block, which contains a 16×16 luma sample block and two corresponding 8×8 color samples in the usual case of 4:2:0 color sampling. Chroma sample block.

Intra-coded blocks use spatial prediction to take advantage of the spatial correlation between pixels. Two divisions are defined as: 16x16 and 4x4.

The inter-coding block uses temporal prediction instead of spatial prediction by estimating the motion between pictures. The motion can be estimated independently for the 16x16 macro block or any of the following sub-macro block partitions: 16x8, 8x16, 8x8, 8x4, 4x8, 4x4 (as shown in FIG. 2). Only one motion vector (MV) is allowed per sub-macro block partition.

1.2 Example of partition tree structure in HEVC

In HEVC, a coding tree unit (CTU) is divided into coding units (CU) by using a four-element tree structure expressed as a coding tree to adapt to various local characteristics. The decision to use inter-picture (temporal) prediction or intra-picture (spatial) prediction to encode picture regions is made at the CU level. According to the prediction unit (PU) division type, each CU can be further divided into one, two, or four PUs. Within a PU, the same prediction process is applied, and relevant information is sent to the decoder on the basis of the PU. After the residual block is obtained by applying the prediction process based on the PU division type, the CU may be divided into transformation units (TU) according to another quad tree structure similar to the coding tree of the CU. One of the key features of the HEVC structure is that it has multiple division concepts, including CU, PU, and TU.

Some of the features involved in hybrid video coding using HEVC include:

(1) Coding tree unit (CTU) and coding tree block (CTB) structure: The similar structure in HEVC is the coding tree unit (CTU), which has a size selected by the encoder and can be larger than traditional giants. Set blocks. CTU is composed of luminance CTB and corresponding chrominance CTB and syntax elements. The size L×L of the luminance CTB can be selected as L=16, 32, or 64 samples, and a larger size can usually achieve better compression. Then, HEVC supports the use of tree structure and quad-tree-like signaling to divide the CTB into smaller blocks.

(2) Coding Unit (CU) and Coding Block (CB): The quaternary tree syntax of CTU specifies the size and location of its luminance CB and chrominance CB. The root of the quaternary tree is associated with the CTU. Therefore, the size of the brightness CTB is the maximum supported size of the brightness CB. The division of CTU into luminance CB and chrominance CB is a joint signaling. One luma CB and usually two chroma CBs and the associated syntax together form a coding unit (CU). The CTB may contain only one CU or may be divided to form multiple CUs, and each CU has an associated division into a tree of prediction units (PU) and transformation units (TU).

(3) Prediction unit and prediction block (PB): The decision to use inter-picture prediction or intra-picture prediction to encode a picture region is made at the CU level. The root of the PU partition structure is at the CU level. Depending on the basic prediction type decision, the luminance CB and chrominance CB can then be further divided in size and predicted based on the luminance and chrominance prediction blocks (PB). HEVC supports variable PB size from 64×64 to 4×4 samples. Figure 3 shows an example of PB allowed for M×M CU.

(4) TU and transform block: use block transform to encode prediction residuals. The root of the TU tree structure is at the CU level. The luminance CB residual may be the same as the luminance transform block (TB), or may be further divided into smaller luminance TB. The same applies to chroma TB. For square TB sizes 4×4, 8×8, 16×16, and 32×32, an integer basis function similar to a discrete cosine transform (DCT) is defined. For the 4×4 transform of the luma intra picture prediction residual, an integer transform derived from the form of Discrete Sine Transform (DST) may alternatively be specified.

1.2.1 Example of dividing the tree structure into TB and TU

For residual coding, CB can be recursively divided into transform blocks (TB). The division is notified by residual quadruple tree signaling. Only square CB and TB divisions are specified, where blocks can be recursively divided into quadrants, as shown in Figs. 4A and 4B. For a given brightness CB with a size of M×M, flag Indicate whether to divide the CB into four blocks of size M/2×M/2. If each quadrant can be further divided as signaled by the maximum depth of the residual quadruple tree indicated in the sequence parameter set (SPS), then a flag is assigned to each quadrant, which indicates whether to divide it into four Quadrant. The leaf node block generated by the residual quadruple tree is a transform block, which is further processed by transform coding. The encoder indicates the maximum and minimum brightness TB size it will use. When the CB size is greater than the maximum TB size, the division is implicit. When the division will result in the luminance TB size being smaller than the indicated minimum value, no division is implicit. Except when the luminance TB size is 4×4, the chrominance TB size is half of the luminance TB size in each dimension. When the luminance TB size is 4×4, a single 4×4 chrominance TB is used by Area covered by four 4×4 TBs of brightness. In the case of an intra-predicted CU, the decoded samples of the nearest neighbor TB (within or outside the CB) are used as reference materials for intra-prediction.

Contrary to previous standards, the HEVC design allows TBs to span multiple PBs for inter-predicted CUs, so as to maximize the potential coding efficiency benefits of TB partitioning in a quad-tree structure.

1.2.2 Parent node and child node

The CTB is divided according to the quaternary tree structure, and its nodes are coding units. Multiple nodes in the quaternary tree structure include leaf nodes and non-leaf nodes. The leaf node has no child nodes in the tree structure (that is, the leaf node will not be further divided). Non-leaf nodes include the root node of the tree structure. The root node corresponds to the initial video block (for example, CTB) of the video data. For each respective non-root node of the plurality of nodes, the respective non-root node corresponds to a video block, and the video block is a child block of the video block corresponding to the parent node in the tree structure of the respective non-root node. Each non-leaf node of the plurality of non-leaf nodes has one or more child nodes in the tree structure.

1.3 An example of a quaternary tree with a larger CTU and a binary tree block structure in JEM

In some embodiments, reference software called Joint Exploration Model (JEM) is used to explore future video coding technologies. In addition to the binary tree structure, JEM also describes the quaternary tree plus binary tree (QTBT) and ternary tree (TT) structure.

1.3.1 Example of QTBT block partition structure

Unlike HEVC, the QTBT structure removes the concept of multiple partition types, that is, it removes the separation of the concepts of CU, PU, and TU, and supports greater flexibility in CU partition shapes. In the QTBT block structure, the CU may have a square or rectangular shape. As shown in FIG. 5A, the coding tree unit (CTU) is first divided by the quad tree structure. The quaternary leaf nodes are further divided by the binary tree structure. There are two types of divisions in binary tree division: symmetric horizontal division and symmetric vertical division. The binary leaf node is called a coding unit (CU), and the division is used for prediction and transformation processing without any further division. This means that CU, PU and TU have the same block size in the QTBT coding block structure. In JEM, a CU is sometimes composed of coding blocks (CB) of different color components. For example, in the case of P and B slices in the 4:2:0 chroma format, one CU contains one luminance CB and two And a CU is sometimes composed of a single component CB, for example, in the case of I slice, one CU contains only one luma CB or only two chroma CBs.

Define the following parameters for the QTBT division scheme.

- CTU size: the size of the root node of the quaternary tree, the same concept as in HEVC

- MinQTSize : the minimum allowable size of four-element leaf nodes

- MaxBTSize : the maximum allowable size of the root node of the binary tree

- MaxBTDepth : Maximum allowable binary tree depth

- MinBTSize : the minimum allowable size of a binary leaf node

In an example of the QTBT partition structure, the CTU size is set to 128×128 luma samples with two corresponding 64×64 chroma sample blocks, MinQTSize is set to 16×16, MaxBTSize is set to 64×64, MinBTSize (Width and Height) is set to 4×4, and MaxBTDepth is set to 4. First, the quaternary tree division is applied to the CTU to generate quaternary leaf nodes. The quaternary leaf node may have a size from 16×16 (ie, MinQTSize ) to 128×128 (ie, CTU size). If the leaf quadtree node is 128×128, since the size exceeds MaxBTSize (ie, 64×64), it will not be further divided by the binary tree. Otherwise, the leaf quaternary tree nodes can be further divided by the binary tree. Therefore, the quaternary leaf node is also the root node of the binary tree, and the depth of the binary tree is zero. When the depth of the binary tree reaches MaxBTDepth (ie, 4), no further division is considered. When the width of the binary tree node is equal to MinBTSize (ie, 4), no further horizontal division is considered. Similarly, when the height of the binary tree node is equal to MinBTSize , no further vertical division is considered. The leaf nodes of the binary tree are further processed through prediction and transformation processing without any further division. In JEM, the maximum CTU size is 256×256 luminance samples.

FIG. 5A shows an example of block division by using QTBT, and FIG. 5B shows the corresponding tree representation. The solid line represents the quaternary tree division, and the dashed line represents the binary tree division. In each partition (ie, non-leaf) node of the binary tree, a flag is signaled to indicate which partition type (ie, horizontal or vertical) is used, where 0 represents horizontal partition and 1 represents vertical partition. For the quad-tree division, there is no need to indicate the division type, because the quad-tree division always divides the blocks horizontally and vertically to generate 4 sub-blocks of equal size.

In addition, the QTBT scheme supports the ability to have a separate QTBT structure for luminance and chrominance. Currently, for P and B slices, the luminance CTB and chrominance CTB in a CTU share the same QTBT structure. However, for I slices, the luma CTB is divided into CUs through the QTBT structure, and the chrominance CTB is divided into chrominance CUs through another QTBT structure. This means that a CU in an I slice is composed of coding blocks of a luma component or two chrominance components, and a CU in a P slice or a B slice is composed of coding blocks of all three color components.

In HEVC, the inter-frame prediction of small blocks is restricted by memory access that reduces motion compensation, so that bi-directional prediction is not supported for 4×8 and 8×4 blocks, and inter-frame prediction is not supported for 4×4 blocks. In JEM's QTBT, these restrictions are removed.

1.4 Triple Tree (TT) of Multifunctional Video Coding (VVC)

Fig. 6A shows an example of a quaternary tree (QT) division, and Fig. 6B and Fig. 6C respectively show An example of vertical and horizontal binary tree (BT) division is presented. In some embodiments, in addition to quaternary trees and binary trees, ternary tree (TT) division is also supported, such as horizontal and vertical center-side ternary trees (as shown in FIG. 6D and FIG. 6E).

In some implementations, two levels of trees are supported: regional trees (quaternary trees) and prediction trees (binary trees or ternary trees). First, use the regional tree (RT) to divide the CTU. The prediction tree (PT) can be further used to divide the RT leaves. PT can also be used to further divide the PT leaf until the maximum PT depth is reached. The PT leaf is the basic coding unit. For convenience, it is still called CU. CU cannot be further divided. Both prediction and transformation are applied to CU in the same way as JEM. The whole partition structure is called "multi-type tree".

1.5 Example of the partition structure in optional video coding technology

In some embodiments, a tree structure called multi-tree (MTT) is supported, which is a generalization of QTBT. In QTBT, as shown in Figure 7, the coding tree unit (CTU) is first divided by a quad tree structure. Then use the binary tree structure to further divide the quaternary leaf nodes.

The structure of MTT is composed of two types of tree nodes: regional tree (RT) and prediction tree (PT), which supports nine types of divisions, as shown in Figs. 8A to 8I. The regional tree can recursively divide the CTU into square blocks, up to 4x4 size regional leaf nodes. At each node of the regional tree, a prediction tree can be formed from one of three tree types: binary tree, ternary tree, and asymmetric binary tree. In the PT division, it is forbidden to divide the quaternary tree in the branches of the prediction tree. Like JEM, the luminance tree and the chrominance tree are separated in I slices.

2. Example of inter prediction in HEVC/H.265

Over the years, video coding standards have improved significantly and now partly provide high coding efficiency and support for higher resolutions. Recent standards such as HEVC and H.265 are based on a hybrid video coding structure in which temporal prediction plus transform coding is used.

2.1 Examples of prediction modes

Each inter prediction PU (prediction unit) has motion parameters of one or two reference picture lists. In some embodiments, the motion parameters include motion vectors and reference picture indexes. In other embodiments, inter_pred_idc can also be used to signal the use of one of the two reference picture lists. In yet another embodiment, the motion vector can be explicitly coded as an increment relative to the predictor.

When coding a coding unit in skip mode, one PU is associated with a CU, and there are no significant residual coefficients, no coding motion vector increments, or reference picture indexes. Specify the Merge mode to obtain the motion parameters of the current PU from the neighboring PU, including space and time candidates. The Merge mode can be applied to any inter-predicted PU, not only to the skip mode. The alternative to the Merge mode is the explicit transmission of motion parameters, in which, for each PU, the motion vector, the corresponding reference picture index of each reference picture list, and the reference picture list use are explicitly signaled.

When the signaling indicates that one of the two reference picture lists will be used, the PU is generated from one sample block. This is called "uni-prediction". Unidirectional prediction can be used for both P slices and B slices.

When the signaling indicates that two reference picture lists will be used, the PU is generated from two sample blocks. This is called "bi-prediction". Bidirectional prediction is only applicable to B slices.

2.1.1 Examples of candidates for constructing Merge mode

When using the Merge mode to predict the PU, the index pointing to the entry in the Merge candidate list is parsed from the bit stream and used to retrieve motion information. The construction of this list can be summarized according to the following sequence of steps:

Step 1: Initial candidate derivation

Step 1.1: Spatial candidate derivation

Step 1.2: Redundancy check of spatial candidates

Step 1.3: Time candidate derivation

Step 2: Insert additional candidates

Step 2.1: Creation of bidirectional prediction candidates

Step 2.2: Insertion of zero motion candidates

Figure 9 illustrates an example of constructing a Merge candidate list based on the sequence of steps outlined above. For the derivation of spatial Merge candidates, at most four Merge candidates are selected among candidates located at five different positions. For the derivation of temporal Merge candidates, at most one Merge candidate is selected among the two candidates. Since a constant number of candidates is assumed for each PU at the decoder, when the number of candidates does not reach the maximum number of Merge candidates (MaxNumMergeCand) signaled in the slice header, additional candidates are generated. Since the number of candidates is constant, truncated unary binarization (TU) is used to encode the index of the best Merge candidate. If the size of the CU is equal to 8, all PUs of the current CU share a single Merge candidate list, which is the same as the Merge candidate list of the 2N×2N prediction unit.

2.1.2 Build Space Merge Candidate

In the derivation of spatial Merge candidates, a maximum of four Merge candidates are selected among the candidates located at the positions depicted in FIG. 10. The order of derivation is A ₁ , B ₁ , B ₀ , A ₀ and B ₂ . _{The position B 2 is} only considered when any PU at the positions A ₁ , B ₁ , B ₀ , A ₀ is not available (for example, because it belongs to another slice or block) or is intra-coded. After the addition of the candidate position at A _1, of the remaining candidate is added redundancy check, which ensures that the candidate has the same motion information is excluded from the list, so that to improve coding efficiency.

In order to reduce the computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. In contrast, only the pairs connected by arrows in FIG. 11 are considered, and the candidates are added to the list only when the corresponding candidates for redundancy check have different motion information. Another source of repetitive motion information is the "second PU" associated with a partition other than 2N×2N. As an example, FIGS. 12A and 12B depict the second PU for the cases of N×2N and 2N×N, respectively. When the current PU is partitioned into N×2N, _{the candidate at position A 1} is not considered for list construction. In some embodiments, adding the candidate may result in two prediction units with the same motion information, which is redundant for having only one PU in the coding unit. Similarly, when the current PU is partitioned as 2N×N, the position B _{1 is} not considered.

2.1.3 Build Time Merge Candidate

In this step, only one candidate is added to the list. In particular, in the derivation of this temporal Merge candidate, the scaling motion vector is derived based on the co-located PU with the smallest POC difference from the current picture in the given reference picture list. The reference picture list used to derive the co-located PU is explicitly signaled in the slice header.

Figure 13 shows an example of the derivation of the scaled motion vector for temporal Merge candidates (such as the dashed line), time, which uses the POC distance tb and td to scale from the motion vector of the co-located PU, where tb is defined as the reference of the current picture The POC difference between the picture and the current picture, and td is defined as the POC difference between the reference picture of the co-located picture and the co-located picture. The reference picture index of the temporal Merge candidate is set to zero. For the B slice, two motion vectors (one for reference picture list 0 and the other for reference picture list 1) are obtained and combined to make them into bidirectional prediction Merge candidates.

In the co-located PU (Y) belonging to the reference frame, the position of the _{temporal candidate is selected between the candidates C 0} and C ₁ , as shown in FIG. 14. If the PU at position C ₀ is not available, internally coded, or outside the current CTU, position C ₁ is used. Otherwise, the position C ₀ is used for the derivation of temporal Merge candidates.

In this step, only one candidate is added to the list. Specifically, in the derivation of the temporal Merge candidate, the scaled motion vector is derived based on the co-located PU, which belongs to the picture with the smallest POC difference from the current picture in the given reference picture list. The reference picture list to be used to derive the co-located PU is explicitly signaled in the slice header.

Figure 13 shows an example of deriving a scaled motion vector for temporal Merge candidates (as shown by the dashed line), which is scaled from the motion vector of the co-located PU using POC distances tb and td, where tb is defined as the reference of the current picture The POC difference between the picture and the current picture, td is defined as the POC difference between the reference picture of the co-located picture and the co-located picture. The reference picture index of the temporal Merge candidate is set equal to zero. For the B slice, two motion vectors are obtained, one is used for reference picture list 0 and the other is used for reference picture list 1, and the two motion vectors are combined to obtain bidirectional prediction Merge candidates.

In the co-located PU (Y) belonging to the reference frame, the position of the _{temporal candidate is selected between the candidates C 0} and C ₁ , as shown in FIG. 14. If the PU at position C ₀ is not available, is intra-coded, or is outside the current CTU, position C ₁ is used. Otherwise, the position C _{0 is} used for the derivation of temporal Merge candidates.

2.1.4 Constructing additional types of Merge candidates

In addition to space-time Merge candidates, there are two additional types of Merge candidates: combined bidirectional predictive Merge candidates and zero Merge candidates. The combined bi-directional predictive Merge candidate is generated by using the space-time Merge candidate. The combined bidirectional prediction Merge candidate is only used for B slices. The combined bi-directional prediction candidate is generated by combining the first reference picture list motion parameter of the initial candidate with the second reference picture list motion parameter of another candidate. If these two tuples provide different motion hypotheses, they will form a new bi-prediction candidate.

Figure 15 shows an example of this process, where two candidates with mvL0 and refIdxL0 or mvL1 and refIdxL1 in the initial list (1510, left) are used to create a combined bidirectional prediction Merge that is added to the final list (1520, right) Candidate.

Insert zero motion candidates to fill the remaining entries in the Merge candidate list to reach the MaxNumMergeCand capacity. These candidates have a zero spatial displacement and a reference picture index, which starts at zero and increases every time a new zero motion candidate is added to the list. The number of reference frames used by these candidates is 1 and 2, which are used for unidirectional and bidirectional prediction, respectively. In some embodiments, no redundancy check is performed on these candidates.

2.1.5 Examples of motion estimation regions processed in parallel

In order to speed up the encoding process, motion estimation can be performed in parallel, thereby deriving the motion vectors of all prediction units in a given area at the same time. Deriving Merge candidates from the spatial neighborhood may interfere with parallel processing because a prediction unit cannot derive motion parameters from neighboring PUs until its associated motion estimation is completed. In order to alleviate the trade-off between coding efficiency and processing latency, a motion estimation area (MER) can be defined. The size of MER can be used in the picture parameter set (PPS) The "log2_parallel_merge_level_minus2" syntax element is signaled. When MER is defined, Merge candidates that fall into the same area are marked as unavailable, so they are not considered in the list construction.

Table 1 presents the picture parameter set (PPS) initial byte sequence payload (RBSP) syntax, where log2_parallel_merge_level_minus2 plus 2 specifies the value of the variable Log2ParMrgLevel, which is used for the Merge mode luminance motion vector specified in the existing video coding standards The derivation process and the derivation process of spatial Merge candidates. The value of log2_parallel_merge_level_minus2 should be in the range of 0 to CtbLog2SizeY-2, including 0 and CtbLog2SizeY-2.

The variable Log2ParMrgLevel is derived as follows: Log2ParMrgLevel=log2_parallel_merge_level_minus2+2

Note that the value of Log2ParMrgLevel represents the built-in capability of parallel derivation of the Merge candidate list. For example, when Log2ParMrgLevel is equal to 6, the Merge candidate list of all prediction units (PU) and coding units (CU) contained in a 64×64 block can be derived in parallel.

2.2 Examples of motion vector prediction

Motion vector prediction uses the space-time correlation between motion vectors and adjacent PUs, which is used for explicit transmission of motion parameters. The motion vector candidate list is constructed by first checking the availability of the temporally adjacent PU positions on the upper left, removing redundant candidate positions, and adding a zero vector to make the length of the candidate list constant. Then, the encoder can select the best predictor from the candidate list and send a corresponding index indicating the selected candidate. Similar to Merge index signaling, the index of the best motion vector candidate is coded using truncated unary.

2.2.1 Example of constructing motion vector prediction candidates

Figure 16 summarizes the derivation process of motion vector prediction candidates, and can be implemented with refidx as input for each reference picture list.

In motion vector prediction, two types of motion vector candidates are considered: spatial motion vector candidates and temporal motion vector candidates. For the derivation of the spatial motion vector candidates, two motion vector candidates are finally derived based on the motion vector of each PU located at the five different positions previously shown in FIG. 10.

For the derivation of temporal motion vector candidates, one motion vector candidate is selected from two candidates, which are derived based on two different co-location positions. After the first empty time selection list is made, the duplicate motion vector candidates in the list are removed. If the number of potential candidates is greater than two, the motion vector candidates whose reference picture index is greater than 1 in the associated reference picture list are removed from the list. If the number of space-time motion vector candidates is less than two, additional zero motion vector candidates will be added to the list.

2.2.2 Construction of spatial motion vector candidates

When deriving the spatial motion vector candidates, at most two candidates are considered among the five potential candidates. These five candidates are from the PU at the positions previously shown in FIG. 10, and these positions are the same as the positions of the motion Merge. The derivation sequence on the left side of the current PU is defined as A ₀ , A ₁ , and scaled A ₀ , scaled A ₁ . The derivation sequence above the current PU is defined as B ₀ , B ₁ , B ₂ , scaled B ₀ , scaled B ₁ , scaled B ₂ . Therefore, there are four cases on each side that can be used as motion vector candidates, two cases do not need to use spatial scaling, and two cases use spatial scaling. The four different situations are summarized as follows:

--Zoom without space

(1) The same reference picture list and the same reference picture index (same POC)

(2) Different reference picture lists, but the same reference picture (same POC)

--Space zoom

(3) The same reference picture list, but different reference pictures (different POC)

(4) Different reference picture lists, and different reference pictures (different POCs)

First check the condition of no spatial scaling, and then check the condition of allowing spatial scaling. When the POC is different between the reference picture of the neighboring PU and the reference picture of the current PU, spatial scaling is considered, regardless of the reference picture list. If all the PUs of the left candidate are unavailable or are intra-coded, the above-mentioned motion vector is allowed to be scaled to help the parallel derivation of the left and upper MV candidates. Otherwise, spatial scaling of the above motion vector is not allowed.

As shown in the example in FIG. 17, for the spatial scaling case, the motion vectors of neighboring PUs are scaled in a similar manner to temporal scaling. One difference is that the reference picture list and index of the current PU are given as input, and the actual scaling process is the same as the time scaling process.

2.2.3 Construction of temporal motion vector candidates

Except for the derivation of the reference picture index, all the derivation processes of the temporal Merge candidates are the same as the derivation process of the spatial motion vector candidates (as shown in the example in FIG. 14). In some embodiments, the reference picture index is signaled to the decoder.

2.2.4 Signaling of AMVP Information

For the AMVP mode, four parts can be signaled in the bit stream, such as prediction direction, reference index, MVD and MV prediction candidate index, which are described in the context of the syntax shown in Table 2 and Table 3.

3 Examples of inter prediction methods in the Joint Exploration Model (JEM)

In some embodiments, reference software called Joint Exploration Model (JEM) is used to explore future video coding technologies. In JEM, sub-block-based prediction is used in several coding tools, such as affine prediction, optional temporal motion vector prediction (ATMVP), space-time motion vector prediction (STMVP), Bidirectional optical flow (BIO), frame rate up-conversion (FRUC), local adaptive motion vector resolution (LAMVR), overlapping block motion compensation (OBMC), local illumination compensation (LIC), and decoder-side motion vector refinement (DMVR) ).

3.1 Example of motion vector prediction based on sub-CU

In JEM with a quad tree plus a binary tree (QTBT), each CU can have at most one set of motion parameters for each prediction direction. In some embodiments, by dividing the large CU into sub-CUs and deriving the motion information of all sub-CUs of the large CU, two sub-CU-level motion vector prediction methods are considered in the encoder. The optional temporal motion vector prediction (ATMVP) method allows each CU to extract multiple sets of motion information from multiple blocks smaller than the current CU in the co-located reference picture. In the space-time motion vector prediction (STMVP) method, the motion vector of the sub-CU is recursively derived by using a temporal motion vector predictor and spatial neighboring motion vectors. In some embodiments, in order to retain a more accurate motion field for sub-CU motion prediction, motion compression of reference frames may be disabled.

3.1.1 Sample time for optional temporal motion vector prediction (ATMVP)

In the ATMVP method, the Temporal Motion Vector Prediction (TMVP) method is modified by extracting multiple sets of motion information (including motion vectors and reference indexes) from blocks smaller than the current CU.

FIG. 18 shows an example of the ATMVP motion prediction processing of the CU 1800. The ATMVP method predicts the motion vector of the sub-CU 1801 in the CU 1800 in two steps. The first step is to use the time vector to identify the corresponding block 1851 in the reference picture 1850. The reference picture 1850 is also called a motion source picture. The second step is to divide the current CU 1800 into sub-CU 1801, and obtain the motion vector and the reference index of each sub-CU from the block corresponding to each sub-CU.

In the first step, the reference picture 1850 and the corresponding block are determined from the motion information of the spatial neighboring blocks of the current CU 1800. In order to avoid the repeated scanning process of adjacent blocks, the first Merge candidate in the Merge candidate list of the current CU 1800 is used. The first available motion vector and its associated reference index are set as the index of the time vector and the motion source picture. In this way, compared with TMVP, the corresponding block can be identified more accurately, The corresponding block (sometimes called a co-located block) is always located in the lower right or center position relative to the current CU.

In one example, if the first Merge candidate is from the left neighboring block (ie, A _{1 in} FIG. 19), the related MV and reference picture are used to identify the source block and the source picture.

In the second step, by adding a time vector to the coordinates of the current CU, the corresponding block of the sub-CU 1851 is identified through the time vector in the motion source picture 1850. For each sub-CU, the motion information of its corresponding block (for example, the smallest motion grid covering the center sample) is used to derive the motion information of the sub-CU. After the motion information of the corresponding N×N block is identified, it is converted into the reference index and motion vector of the current sub-CU in the same way as the TMVP of HEVC, where motion scaling and other processes are also applicable. For example, the decoder checks whether the low-delay condition is satisfied (for example, the POC of all reference pictures of the current picture is smaller than the POC of the current picture) and may use the motion vector MV _x (for example, the motion vector corresponding to the reference picture list X) to predict each The motion vector MV _{y of each} sub-CU (for example, where X is equal to 0 or 1 and Y is equal to 1-X).

3.1.2 Example of Space-Time Motion Vector Prediction (STMVP)

In the STMVP method, the motion vector of the sub-CU is derived recursively according to the raster scan order. Figure 20 shows an example of a CU with four sub-blocks and adjacent blocks. Consider an 8×8 CU 2000, which contains four 4×4 sub-CUs A (2001), B (2002), C (2003) and D (2004). The adjacent 4×4 blocks in the current frame are marked as a(2011), b(2012), c(2013) and d(2014).

The motion derivation of sub CU A starts by identifying its two spatial neighbors. The first neighbor is the N×N block above the sub CU A 1101 (block c 2013). If the block c 2013 is not available or is intra-coded, then check the other N×N blocks above the sub CU A (2001) (from left to right, starting at block c 2013). The second neighbor is the block to the left of the sub CU A 2001 (block b 2012). If block b (2012) is not available or is intra-coded, check the other blocks on the left of sub-CU A 2001 (from top to bottom, starting at block b 2012). The motion information obtained from neighboring blocks in each list is scaled to the first reference frame of a given list. Next, follow the same procedure as TMVP specified in HEVC to derive the temporal motion vector of sub-block A 2001 Forecast (TMVP). The motion information of the co-located block at block D 2004 is extracted and scaled accordingly. Finally, after retrieving and scaling the motion information, all available motion vectors are averaged for each reference list. The average motion vector is designated as the motion vector of the current sub-CU.

3.1.3 Example of sub-CU motion prediction mode signaling

In some embodiments, the sub-CU mode is enabled as an additional Merge candidate mode, and no additional syntax elements are required to signal the mode. The other two Merge candidates are added to the Merge candidate list of each CU to indicate ATMVP mode and STMVP mode. In other embodiments, if the sequence parameter set indicates that ATMVP and STMVP are enabled, a maximum of seven Merge candidates can be used. The coding logic of the additional Merge candidate is the same as that of the Merge candidate in the HM, which means that for each CU in the P slice or B slice, two additional RD checks may be required for two additional Merge candidates. In some embodiments, such as JEM, all bins of the Merge index are context-encoded by CABAC (Context-based Adaptive Binary Arithmetic Coding). In other embodiments, such as HEVC, only the first binary bit is context coded, and the remaining binary bits are context bypass coded.

3.2 Example of adaptive motion vector difference resolution

In some embodiments, when the use_integer_mv_flag in the slice header is equal to 0, the motion vector difference (MVD) (between the motion vector of the PU and the predicted motion vector) is signaled in units of a quarter luminance sample. In JEM, local adaptive motion vector resolution (LAMVR) is introduced. In JEM, MVD can be coded in units of one-quarter luminance samples, integer luminance samples, or 4 luminance samples. The MVD resolution is controlled at the coding unit (CU) level, and the MVD resolution flag is conditionally signaled for each CU with at least one non-zero MVD component.

For a CU with at least one non-zero MVD component, the first flag is signaled to indicate whether to use the quarter luma sample MV accuracy in the CU. When the first flag (equal to 1) indicates that the quarter luminance sample MV precision is not used, another flag is signaled to indicate that the integer luminance sample MV is used The accuracy is still 4 luminance samples MV accuracy.

When the first MVD resolution flag of the CU is zero or is not coded for the CU (meaning that all MVDs in the CU are zero), a quarter luma sample MV resolution is used for the CU. When the CU uses integer luma sample MV precision or 4 luma sample MV precision, the MVP in the AMVP candidate list of the CU is rounded to the corresponding precision.

In the encoder, the CU-level RD check is used to determine which MVD resolution to use for the CU. That is, for each MVD resolution, three CU-level RD checks are performed. In order to speed up the encoder, the following coding scheme is applied in JEM.

- During the RD inspection of the CU with the MVD resolution of a normal quarter luminance sample, store the current CU motion information (integer luminance sample accuracy). The stored motion information (after rounding) is used as the starting point for further small-range motion vector refinement for the same CU with integer luma samples and 4 luma sample MVD resolution during the RD inspection, making a time-consuming motion estimation process Do not repeat three times.

- Conditionally invoke the RD check of the CU with the MVD resolution of 4 luminance samples. For the CU, when the MVD resolution of the RD cost integer luminance sample is much larger than the MVD resolution of a quarter luminance sample, skip the RD check for the MVD resolution of 4 luminance samples of the CU.

3.3 Example of Pattern Matching Motion Vector Derivation (PMMVD)

The PMMVD mode is a special Merge mode based on the frame rate up conversion (FRUC) method. Using this mode, the motion information of the block is derived on the decoder side instead of signaling the motion information of the block.

When the Merge flag of the CU is true, the FRUC flag can be signaled to the CU. When the FRUC flag is false, the Merge index can be signaled and the regular Merge mode can be used. When the FRUC flag is true, an additional FRUC mode flag can be signaled to indicate which method (for example, bilateral matching or template matching) will be used to derive the motion information of the block.

On the encoder side, the decision on whether to use the FRUC Merge mode for the CU is based on the RD cost selection made for the normal Merge candidates. For example, by using RD cost selection to check the number of CUs Two kinds of matching modes (for example, bilateral matching and template matching). The matching mode that causes the least cost is further compared with other CU modes. If the FRUC matching mode is the most effective mode, the FRUC flag is set to true for the CU, and the relevant matching mode is used.

Generally, the motion derivation process in the FRUC Merge mode has two steps: first, perform a CU-level motion search, and then perform sub-CU-level motion refinement. At the CU level, based on bilateral matching or template matching, the initial motion vector of the entire CU is derived. First, a list of MV candidates is generated, and the candidate that causes the smallest matching cost is selected as the starting point for further CU-level refinement. Then, a local search based on bilateral matching or template matching is performed near the starting point. The MV result with the smallest matching cost is taken as the MV of the entire CU. Subsequently, the derived CU motion vector is used as a starting point to further refine the motion information at the sub-CU level.

For example, for W × H CU motion information derivation, the following derivation process is performed. In the first stage, the MV of the entire W × H CU is derived. In the second stage, the CU is further divided into M × M sub-CUs. The calculation method of the value of M is shown in equation (3), and D is the predefined division depth, which is preset to 3 in JEM. Then derive the MV of each sub-CU.

FIG. 21 shows an example of bilateral matching used in the frame rate up conversion (FRUC) method. By finding the closest match between two blocks in two different reference pictures (2110, 2111) along the motion trajectory of the current CU, bilateral matching is used to derive the motion information of the current CU (2100). Under the assumption of continuous motion trajectories, the time distance between the motion vectors MV0 (2101) and MV1 (2102) pointing to the two reference blocks and the current picture and the two reference pictures (for example, TD0 (2103) and TD1 (2104) are In some embodiments, when the current picture 2100 is temporarily located between two reference pictures (2110, 2111) and the time distance from the current picture to the two reference pictures is the same, the bilateral matching becomes a mirror-based two-way MV.

FIG. 22 shows an example of template matching used in the frame rate up conversion (FRUC) method. Through the template in the current picture (for example, the top and/or left adjacent block of the current CU) and reference The closest match is found between the blocks in the picture 2210 (for example, the same size as the template), and the template matching is used to derive the current motion information of the CU 2200. In addition to the aforementioned FRUC Merge mode, template matching can also be applied to AMVP mode. In both JEM and HEVC, AMVP has two candidates. Using template matching methods, new candidates can be derived. If the newly derived candidate from template matching is different from the first existing AMVP candidate, insert it at the very beginning of the AMVP candidate list, and then set the list size to 2 (for example, by removing the second existing AMVP candidate) . When applied to AMVP mode, only CU-level search is applied.

The CU-level MV candidate set can include the following: (1) the initial AMVP candidate, if the current CU is in AMVP mode, (2) all Merge candidates, (3) several MVs in the interpolated MV field (described later), and the top And the adjacent motion vector on the left.

When using bilateral matching, each valid MV of the Merge candidate can be used as input to generate MV pairs under the assumption of bilateral matching. For example, in the reference list A, a valid MV of the Merge candidate is (MVa, ref _a ). _{Then, find the reference picture ref b of} its paired bilateral MV in the other reference list B, so that ref _a and ref _b are located on different sides of the current picture in time. If such ref _b is not available in reference list B, then ref _b is determined to be _a different reference from ref a, and its time distance to the current picture is the minimum value in list B. After determining ref _b , MVb is derived by scaling MVa based on _{the temporal distance between ref a} and ref _{b of the current picture.}

In some implementations, four MVs from the interpolated MV field are also added to the CU-level candidate list. More specifically, the interpolated MVs at positions (0, 0), (W/2, 0), (0, H/2), and (W/2, H/2) of the current CU are added. When FRUC is applied in AMVP mode, the initial AMVP candidate is also added to the MV candidate set at the CU level. In some implementations, at the CU level, 15 MVs of AMVP CU and 13 MVs of Merge CU can be added to the candidate list.

The MV candidates set at the sub-CU level include: (1) MVs determined from the CU level search, (2) adjacent MVs on the top, left, top left, and top right, and (3) scaling of co-located MVs from reference pictures Version, (4) one or more ATMVP candidates (e.g., up to 4), and (5) one or more STMVP candidates (e.g., up to 4). The zoomed MV from the reference picture is derived as follows. The reference pictures in the two lists are traversed. The MV at the co-located position of the sub-CU in the reference picture is scaled to the reference of the starting CU-level MV. The ATMVP and STMVP candidates can be the first four. At the sub-CU level, one or more MVs (up to 17) are added to the candidate list.

Generation of interpolated MV field . Before encoding the frame, an interpolated motion field of the entire picture is generated based on the one-way ME. Then, the sports field can be subsequently used as a MV candidate at the CU level or at the sub-CU level.

In some embodiments, the motion field of each reference picture in the two reference lists is traversed at the 4×4 block level. FIG. 23 shows an example of unilateral motion estimation (ME) 2300 in the FRUC method. For each 4×4 block, if the motion associated with the block passes through the 4×4 block in the current picture and the block is not assigned any interpolated motion, the motion of the reference block is based on the temporal distance TD0 and TD1 (to be compared with that in HEVC The MV zoom of TMVP is zoomed to the current picture in the same way, and the zoomed motion is assigned to the block in the current frame. If an unscaled MV is assigned to a 4×4 block, the motion of the block is marked as unavailable in the interpolated motion field.

Interpolation and matching costs. When the motion vector points to the position of the fractional sample, motion compensation interpolation is required. In order to reduce complexity, instead of conventional 8-tap HEVC interpolation, bilinear interpolation can be used for bilateral matching and template matching.

The calculation of the matching cost is a bit different at different steps. When selecting candidates from the candidate set at the CU level, the matching cost may be the absolute sum difference (SAD) of bilateral matching or template matching. After the initial MV is determined, the matching cost C of the bilateral matching of the sub-CU level search is calculated as follows: C = SAD + w . (| MV _x - MV _x ^s |+| MV _y - MV _y ^s |) Equation (4)

Here, w is the weight coefficient. In some embodiments, w is empirically set to 4. MV and MV ^s indicate the current MV and the starting MV, respectively. SAD can still be used as the matching cost of pattern matching for sub-CU-level searches.

In FRUC mode, MV is derived only by using luminance samples. The derived motion will be used for both luma and chroma for MC inter prediction. After the MV is determined, the final MC is performed using an 8-tap interpolation filter for luma and a 4-tap interpolation filter for chroma.

MV refinement is a pattern-based MV search, using bilateral matching costs or template matching costs as the standard. In JEM, two search modes are supported-Unrestricted Center Offset Diamond Search (UCBDS) and Adaptive Cross Search, respectively, for MV refinement at the CU level and the sub-CU level. For both CU and sub-CU-level MV refinement, the MV is directly searched with quarter-luminance sample MV accuracy, and then one-eighth luminance sample MV refinement follows. The search range for MV refinement for the CU and sub-CU steps is set equal to 8 luma samples.

In the bilateral matching Merge mode, bidirectional prediction is applied because the motion information of the CU is derived based on the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures. In the template matching Merge mode, the encoder can select for the CU from one-way prediction in list 0, one-way prediction in list 1, or two-way prediction. The selection can be based on the following template matching cost: if costBi<=factor*min(cost0,cost1), use bidirectional prediction; otherwise, if cost0<=cost1, use the one-way prediction in list 0; otherwise, use the one in list 1. One-way prediction; here, cost0 is the SAD matched by the template in list 0, cost1 is the SAD matched by the template in list 1, and costBi is the SAD matched by the two-way prediction template. For example, when the value of factor is equal to 1.25, it means that the selection process is biased towards bidirectional prediction. Inter-frame prediction direction selection can be applied to CU-level template matching processing.

3.4 Example of Decoder-side Motion Vector Refinement (DMVR)

In the bidirectional prediction operation, for the prediction of a block area, the motion of list0 will be used respectively The two prediction blocks formed by the vector (MV) and the MV of list1 are combined to form a single prediction signal. In the decoder-side motion vector refinement (DMVR) method, the two motion vectors of bidirectional prediction are further refined through bilateral template matching processing. Bilateral template matching is applied in the decoder to perform a distortion-based search between the bilateral template and the reconstructed samples in the reference picture in order to obtain a refined MV without the need to transmit additional motion information.

In DMVR, from the initial MV0 of list 0 and MV1 of list 1, the bilateral template is generated as a weighted combination (ie, average) of two prediction blocks, as shown in FIG. 24. The template matching operation includes calculating the cost metric between the generated template and the sample area (around the initial prediction block) in the reference picture. For each of the two reference pictures, the MV that produces the smallest template cost is considered as the updated MV of the list to replace the original MV. In JEM, each list is searched for nine MV candidates. The nine MV candidates include the initial MV and 8 surrounding MVs with one luminance sample offset from the initial MV in the horizontal or vertical direction or in both directions. Finally, two new MVs, MV0' and MV1' as shown in Fig. 24, are used to generate the final bidirectional prediction result. The sum of absolute difference (SAD) is used as a cost metric.

DMVR is applied to the Merge mode of bidirectional prediction, where one MV comes from a reference picture in the past and the other MV comes from a reference picture in the future, without the need to transmit additional syntax elements. In JEM, when LIC, affine motion, FRUC, or sub-CU Merge candidates are enabled for the CU, DMVR is not applied.

3.5 Example of merge/skip mode with two-sided matching refinement

First, construct the Merge candidate list as follows: Use redundancy check to insert the motion vectors and reference indexes of spatially adjacent and temporally adjacent blocks into the candidate list until the number of available candidates reaches the maximum candidate size of 19. As used in HEVC (combined candidate and zero candidate), according to the predetermined insertion order, by inserting spatial candidates, temporal candidates, affine candidates, advanced temporal MVP (ATMVP) candidates, space-time MVP (STMVP) candidates and additional candidates To build a Merge candidate list for Merge/Skip mode, and in the context of the numbered blocks shown in Figure 25:

(1) Spatial candidates for blocks 1-4

(2) Extrapolated affine candidates for blocks 1-4

(3) ATMVP

(4) STMVP

(5) Virtual affine candidate

(6) Spatial candidates (block 5) (only used when the number of available candidates is less than 6)

(7) Extrapolate affine candidates (block 5)

(8) Temporal candidates (as deduced in HEVC)

(9) Non-adjacent spatial candidates, followed by extrapolated affine candidates (blocks 6 to 49)

(10) Combination candidates

(11) Zero candidates

It can be noted that in addition to STMVP and affine, the IC flag is also inherited from the Merge candidate. In addition, for the first four spatial candidates, the bi-predictive candidate is inserted before the uni-predicted candidate.

3.5.1 Non-adjacent Merge Candidates

If the total number of available Merge candidates has not reached the maximum allowable Merge candidates, non-adjacent Merge candidates can be added to the Merge candidate list. In existing implementations, non-adjacent Merge candidates can be inserted after the TMVP Merge candidates in the Merge candidate list. The process of adding non-adjacent Merge candidates can be performed by the pseudo code shown in FIG. 26.

4 Examples of existing implementations

In existing implementations, using non-adjacent Merge candidates that obtain motion information from non-adjacent blocks may result in sub-optimal performance.

In one example, prediction based on the motion information of non-adjacent blocks located above the CTU line may significantly increase the line buffer size.

In another example, the prediction based on the motion information of non-neighboring blocks can be used to calculate all motions Information (usually on the 4×4 level) stored in the cache brings additional coding gain to the cost (which significantly increases the complexity of the hardware implementation).

5 Examples of methods used to construct non-adjacent Merge candidates

The embodiments of the currently disclosed technology overcome the shortcomings of existing implementations, thereby providing video coding with lower memory and complexity requirements and higher coding efficiency. The selection of non-adjacent Merge candidates based on the disclosed technology can enhance existing and future video coding standards, which are illustrated in the examples described below for various implementations. The examples of the disclosed technology provided below explain general concepts and are not meant to be interpreted as limitations. In the examples, unless explicitly indicated to the contrary, the various features described in these examples may be combined.

The embodiments of the disclosed technology reduce the cache memory/line buffer size required for non-adjacent Merge candidates and methods for further improving the encoding performance of non-adjacent Merge candidates.

For the example discussed below, suppose the coordinates of the upper left sample of the current block are (Cx, Cy), and the coordinates of the upper left sample in a non-adjacent block are (NAx, NAy), and the origin (0,0) is the picture/slice /Piece/LCU line/LCU dot in the upper left corner. The coordinate difference (ie, the offset from the current block) is represented by (offsetX, offsetY), where offsetX=Cx-NAx and offsetY=Cy-NAy.

Example 1 advantageously provides at least a reduction in memory and buffers.

Example 1: In an example, when constructing a Merge candidate, only non-adjacent blocks located at a specific position are accessed.

(a) In an example, x and y should satisfy NAx%M=0 and NAy%N=0, where M and N are two non-zero integers, such as M=N=8 or 16.

(b) In an example, if the upper left sample in a non-adjacent block does not meet a given condition, skip the check of the motion information associated with the block. Therefore, the associated sports information cannot be added to the Merge candidate list.

(c) Alternatively, if the upper left sample in a non-adjacent block does not satisfy the given condition, You can shift, truncate or round the position of the block to ensure that the conditions are met. For example, (NAx, NAy) can be modified to ((NAx/M)*M, (NAy/N)*N), where "/" is integer division.

(d) The size of the restricted area covering all non-adjacent blocks can be pre-defined/signaled. In this case, when a non-adjacent block calculated by a given offset (OffsetX, OffsetY) is outside the area, it is marked as unavailable or regarded as an intra-coding mode. The corresponding motion information can be added to the candidate list as candidates. An example is depicted in Figure 27.

(i) In one example, the area size is defined as one or more CTBs.

(ii) In an example, the area size is defined as W*H (for example, W=64 and H=64). Alternatively, in addition, all non-adjacent blocks with coordinates (NAx, NAy) should satisfy at least one of the following conditions: NAx>=((Cx/W)*W)

NAx<=((Cx/W)* W)+W.

NAy>=((Cy/H)* H)

NAy<=((Cy/H)* H)+H.

Among them, the ">=" and/or "<=" in the above function can be replaced by ">" and/or "<". The function "/" represents an integer division operation, in which the fractional part of the division result is discarded.

(iii) Alternatively, all blocks above the LCU row covering the current block are marked as unavailable or regarded as intra-code mode. The corresponding motion information can be added to the candidate list as candidates.

(iv) Alternatively, suppose that the upper left sample coordinates of the LCU covering the current block are (LX, LY). (LX-NAx), and/or abs(LX-NAx), and/or (LY-NAy), and/or abs(LY-NAy) should be within the critical value.

(v) One or more critical values can be predefined. They may further depend on the minimum size of the CU height/the minimum size of the width/LCU size, etc. For example, (LY-NAy) should be smaller than the minimum size of the CU height, or (LY-NAy) should be smaller than twice the minimum size of the CU height.

(vi) The area size or critical value can be signaled in sequence parameter set (SPS), picture parameter set (PPS), video parameter set (VPS), slice header, slice header, etc.

(vii) In one example, all non-adjacent blocks other than the current slice/slice/other types of units used for parallel encoding are marked as unavailable, and the corresponding motion information should not be added to the candidate list as candidates.

Example 2 advantageously provides at least reduced computational complexity.

Example 2: When a new non-adjacent Merge candidate is inserted, pruning can be applied to some of the available Merge candidates.

(a) In one example, the new non-adjacent Merge candidate is not pruned with other inserted non-adjacent Merge candidates.

(b) In one example, new non-adjacent Merge candidates are not pruned with temporal Merge candidates such as TMVP or ATMVP.

(c) In one example, the new non-adjacent Merge candidates are pruned with some Merge candidates from some specific neighboring blocks, but not with some other Merge candidates from some other specific neighboring blocks.

The examples described above can be incorporated into the context of the methods described below (for example, method 2800), which can be implemented at a video decoder and/or a video encoder.

FIG. 28 shows a flowchart of an exemplary method for video encoding, which can be implemented in a video encoder. Method 2800 includes, in step 2810, receiving the current block of video data.

The method 2800 includes, in step 2820, selecting a first non-adjacent block that is not adjacent to the current block based on a rule.

In some embodiments, and as described in the context of Example 1, the first non-adjacent block is selected from a plurality of non-adjacent blocks, and wherein the restricted area contains the Every. In the example, the restricted area corresponds to one or more coding tree blocks (CTB) next to the current block.

In some embodiments, the upper left sample coordinate of the picture segment is (0, 0), and the upper left sample coordinate of the first non-adjacent block is (NAx, NAy). In the first example, the rules specify (NAx%M=0) and (NAy%N=0), where% represents a modulus function, and M and N are integers. In the second example, the rule designation (NAx, NAy) can be modified to ((NAx/M)×M) and ((NAx/N)×N), where / represents integer division. In the third example, the upper left sample of the largest coding unit (LCU) covering the current block is (Lx, Ly), and (Lx-NAx), abs(Lx-NAx), (Ly-NAy), and abs(Ly- At least one of NAy) is less than a predetermined critical value, where abs() represents an absolute value function.

The method 2800 includes, in step 2830, constructing a Merge candidate list including a first Merge candidate, the first Merge candidate including motion information based on the first non-adjacent block.

In some embodiments, the method 2800 further includes inserting the first Merge candidate into the Merge candidate list, and as described in the context of Example 3, it may be pruned or not pruned with a specific type of Merge candidate. In the first example, the first Merge candidate is not pruned with other Merge candidates constructed from other non-adjacent blocks in the Merge candidate list. In the second example, the first Merge candidate is not pruned with the temporal Merge candidate in the Merge candidate list. In the third example, the first Merge candidate is pruned with other candidates from the first set of neighboring blocks, and the first Merge candidate is not pruned with candidates from the second set of neighboring blocks that are different from the first set of neighboring blocks. prune.

In some embodiments, and as described in the context of Example 4, the first non-adjacent block may or may not be limited to certain types of encoding. For example, the first non-adjacent block is a non-adjacent block encoded by Advanced Motion Vector Prediction (AMVP). For example, the spatial Merge candidate is not used to encode the first non-adjacent block. For example, the Merge mode and the motion refinement process are used to encode the first non-adjacent block. For example, the decoder side motion vector refinement (DMVR) is not used to encode the first non-adjacent block.

In some embodiments, the method 2800 further includes selecting a second non-adjacent block. In the example, the first non-adjacent block and the second non-adjacent block are spatial neighbors of the current block, the first non-adjacent block is coded using the first mode, and the second non-adjacent block is coded using the second mode. And when selecting the second non-adjacent block Select the first non-adjacent block before.

In some embodiments, the motion information of non-neighboring blocks is used as a predictor in Advanced Motion Vector Prediction (AMVP) mode.

The method 2800 includes, in step 2840, processing the current block based on the Merge candidate list. In some implementations, the current block is decoded by trimming the Merge candidate list and using the trimmed Merge candidate list.

6 Example implementation of the disclosed technology

FIG. 29 is a block diagram of the video processing device 2900. The device 2900 can be used to implement one or more of the methods described herein. The device 2900 may be implemented in a smart phone, a tablet computer, a computer, an Internet of Things (IoT) receiver, and the like. The device 2900 may include one or more processors 2902, one or more memories 2904, and video processing hardware 2906. The processor 2902 may be configured to implement one or more methods described herein (including but not limited to the method 2800). One or more memories 2904 can be used to store data and codes used to implement the methods and techniques described herein. Video processing hardware 2906 can be used to implement some of the technologies described in this article in hardware circuits.

In some embodiments, the video decoder device may implement a method using zero units as described herein for video decoding. The various features of the method can be similar to the method 2800 described above.

In some embodiments, the video decoding method may be implemented using a decoding device implemented on a hardware platform, as described with respect to FIG. 29.

The following uses a clause-based description format to describe additional features and embodiments of the above methods/techniques.

1. A video processing method, comprising: receiving a current block of video data; selecting a first non-adjacent block that is not adjacent to the current block based on a rule; constructing a Merge candidate list including a first Merge candidate, the first Merge candidates include motion information based on the first non-neighboring block; and processing the current block based on the Merge candidate list.

2. The method according to clause 1, wherein the current block is processed by pruning the Merge candidate list and using the pruned Merge candidate list.

3. The method according to clause 1, wherein the upper left sample coordinates of the picture fragment is (0, 0), wherein the upper left sample coordinates of the first non-adjacent block are (NAx, NAy), and the rule specifies ( NAx%M=0) and (NAy%N=0), where% is the modulus function, and where M and N are integers.

4. The method according to clause 1, further comprising determining whether the upper left sample of the first non-adjacent block satisfies the rule, and the first Merge candidate includes the first non-adjacent Movement information associated with the block.

5. The method according to clause 3, wherein the rule specifies to modify the upper left sample coordinates (NAx, NAy) to ((NAx/M)×M) and ((NAx/N)×N), where / is Integer division, where M and N are integers.

6. The method according to clause 1, wherein the rule specifies that the first non-adjacent block is selected from a plurality of non-adjacent blocks, and wherein the restricted area includes Every.

7. The method according to clause 1, wherein the size of the restricted area is predefined or signaled.

8. The method of clause 5, wherein the restricted area corresponds to one or more coding tree blocks (CTB) adjacent to the current block.

9. The method according to clause 5, wherein the restricted area includes a rectangular area of W samples by H samples, or a non-adjacent block with coordinates (NAx, NAy) in it satisfies one of the following conditions: NAx>= ((Cx/W)* W), or NAx<=((Cx/W)* W)+W, or NAy>=((Cy/H)* H), or NAy<=((Cy/H) * H)+H.

10. The method according to clause 5, wherein the restricted area includes a rectangular area of W samples by H samples, or a non-adjacent block with coordinates (NAx, NAy) in it satisfies one of the following conditions: NAx>( (Cx/W)* W), or NAx<((Cx/W)* W)+W, or NAy>((Cy/H) * H), or NAy<((Cy/H)* H)+H.

11. The method according to clause 9 or 10, wherein the operation "/" indicates an integer division operation in which the fractional part of the result is discarded.

12. The method according to clause 6, wherein the upper left sample of the largest coding unit (LCU) covering the current block is (Lx, Ly), and wherein (Lx-NAx), abs(Lx-NAx), (Ly -At least one of NAy) and abs (Ly-NAy) is less than one or more critical values.

13. The method of clause 12, wherein the one or more critical values are based on a minimum size of a height of a coding unit (CU) or LCU, or a minimum size of a width of a CU or LCU.

14. The method according to clause 12, wherein the video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), slice header, slice header, coding tree unit (CTU) or CU is used Signaling the size of the restricted area or the one or more critical values.

15. The method according to clause 1, further comprising: inserting the first Merge candidate into a Merge candidate list.

16. The method according to clause 12, wherein the first Merge candidate is not pruned with other Merge candidates constructed from other non-adjacent blocks in the Merge candidate list.

17. The method according to clause 15, wherein the first Merge candidate is not pruned with the temporal Merge candidate in the Merge candidate list, and wherein the temporal Merge candidate includes temporal motion vector prediction (TMVP) or optional temporal motion vector Forecast (ATMVP).

18. The method according to clause 15, wherein the first Merge candidate is pruned with candidates from the first set of neighboring blocks, and wherein the first Merge candidate is not the same as those from the first set of neighboring blocks. The different candidates of the second set of neighboring blocks are pruned.

19. A device comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions when executed by the processor cause the processor to implement the method of any one of clauses 1 to 18 .

20. A computer program product stored on a non-transitory computer readable medium, the computer program product containing program code for executing the method described in any one of clauses 1 to 18.

It can be understood from the foregoing that the specific embodiments of the disclosed technology have been described herein for illustrative purposes, but various modifications can be made without departing from the scope of the present invention. Therefore, the technology of the present disclosure is not limited except for the appended claims.

The subject and functional operations described in this patent document can be implemented in various systems, digital electronic circuits, or computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in them A combination of one or more of them. The realization of the subject described in this manual can be realized as one or more computer program products, that is, one or more computer program instruction modules encoded on a tangible and non-transitory computer readable medium for execution by a data processing device Or control the operation of the data processing device. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a combination of substances that affect a machine-readable propagation signal, or a combination of one or more of them. The term "data processing unit" or "data processing device" covers all devices, equipment, and machines used to process data, including, for example, programmable processors, computers, or multiple processors or computers. In addition to hardware, the device may also contain code that creates an execution environment for the computer program in question, for example, constituting processor firmware, protocol stack, database management system, operating system, or one or more of them The code of the combination.

Computer programs (also called programs, software, software applications, scripts or codes) can be written in any form of programming language, including compiled or interpreted languages, and computer programs can be deployed in any form, including stand-alone programs or suitable for computing environments Modules, components, subprograms or other units used in The computer program does not necessarily correspond to the files in the file system. Programs can be stored in the section of the file that holds other programs or data (for example, one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinations In files (for example, files that store one or more modules, subprograms, or code parts). Computer programs can be deployed to It is executed on one computer or on multiple computers located on one site or distributed on multiple sites and interconnected by a communication network.

The processes and logic flows described in this specification can be executed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The process and logic flow can also be executed by a dedicated logic circuit, and the device can also be implemented as a dedicated logic circuit, such as FPGA (Field Programmable Gate Array) or ASIC (Dedicated Integrated Circuit).

For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, the processor will receive commands and data from read-only memory or random access memory or both. The basic components of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, the computer will also include or be operatively coupled to one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical discs, to receive data from the one or more mass storage devices, or Transmit data to the one or more mass storage devices, or both receive and transmit data. However, the computer does not need to have such equipment. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices. The processor and memory can be supplemented by or incorporated into a dedicated logic circuit.

It is intended to treat the description together with the drawings as merely exemplary, where exemplary means an example. As used herein, unless the context clearly dictates otherwise, the singular forms "a", "an" and "the" are intended to also encompass the plural forms. In addition, the use of "or" is intended to include "and/or" unless the context clearly dictates otherwise.

Although this patent document contains many details, these details should not be construed as limitations on the scope of any invention or claimable, but as a description of the features specific to a particular embodiment of a particular invention. In this patent document, certain features described in the context of separate embodiments may also be Implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable subcombination. Furthermore, although the above features may be described as working in certain combinations and even as claimed initially, in some cases, one or more features from the combination may be removed from the claimed combination, and The claimed combination may refer to a sub-combination or a variant of the sub-combination.

Similarly, although operations are depicted in a specific order in the drawings, this should not be understood as requiring that such operations be performed in the specific order shown or in order, or that all operations depicted are performed to achieve the desired result . In addition, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

Only a few implementations and examples are described, and other implementations, enhancements and modifications can be made based on the content described and illustrated in this patent document.

The foregoing descriptions are only preferred embodiments of the present invention, and all equivalent changes and modifications made in accordance with the scope of the patent application of the present invention should fall within the scope of the present invention.

2800: method

2810 to 2840: steps

Claims (19)

A video processing method includes: receiving a current block of video data; selecting a first non-adjacent block that is not adjacent to the current block according to a rule; constructing a Merge candidate list including a first Merge candidate, the The first Merge candidate includes motion information based on the first non-adjacent block; and the current block is processed according to the Merge candidate list; wherein the upper left sample coordinates of the picture segment covering the current block are (0, 0), and the first The upper left sample coordinates of a non-adjacent block are (NAx, NAy), and the rule specifies (NAx%M=0) and (NAy%N=0), where% is a modulus function, and where M and N are integers . The method according to claim 1, wherein the current block is processed by trimming the Merge candidate list and using the trimmed Merge candidate list. The method according to claim 1, further comprising determining whether the upper left sample of the first non-adjacent block satisfies the rule, and the first Merge candidate includes motion information associated with the first non-adjacent block satisfying the rule . The method according to claim 1, further comprising determining whether the upper left sample of the first non-adjacent block satisfies the rule, and if the rule is not satisfied, the upper left sample coordinates (NAx, NAy) are modified to ((NAx/M )×M) and ((NAx/N)×N), where / is integer division, where M and N are integers. The method according to claim 1, wherein the rule specifies that the first non-adjacent block is selected from a plurality of non-adjacent blocks, and one of the restricted areas includes each of the plurality of non-adjacent blocks. The method according to claim 5, wherein the size of a restricted area is predefined or signaled. The method according to claim 5, wherein the restricted area corresponds to a block adjacent to the current block One or more coding tree blocks (CTB). The method according to claim 5, wherein the restricted area includes a rectangular area of W samples by H samples, or a non-adjacent block with coordinates (NAx, NAy) satisfies one of the following conditions: NAx>=((Cx /W)* W), or NAx<=((Cx/W)* W)+W, or NAy>=((Cy/H)* H), or NAy<=((Cy/H)* H) +H. The method according to claim 5, wherein the restricted area includes a rectangular area of W samples by H samples, or a non-adjacent block with coordinates (NAx, NAy) satisfies one of the following conditions: NAx>((Cx/ W)* W), or NAx<((Cx/W)* W)+W, or NAy>((Cy/H)* H), or NAy<((Cy/H)* H)+H. The method according to claim 8 or 9, wherein the operation "/" indicates an integer division operation, in which the fractional part of the result is discarded. The method according to claim 5, wherein the upper left sample of the largest coding unit (LCU) covering the current block is (Lx, Ly), and (Lx-NAx), abs(Lx-NAx), (Ly-NAy) At least one of) and abs(Ly-NAy) is less than one or more critical values. The method according to claim 11, wherein the one or more critical values are based on the minimum size of the height of the coding unit (CU) or the largest coding unit, or the smallest size of the width of the coding unit or the largest coding unit. The method according to claim 11, wherein the information is used in video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), slice header, slice header, coding tree unit (CTU) or coding unit Let inform the size of the restricted area or the one or more critical values. The method according to claim 1, further comprising: inserting the first Merge candidate into the Merge candidate list. The method according to claim 14, wherein the first Merge candidate is not pruned with other Merge candidates constructed from other non-adjacent blocks in the Merge candidate list. The method according to claim 14, wherein the first Merge candidate is not pruned with the temporal Merge candidate in the Merge candidate list, and wherein the temporal Merge candidate includes temporal motion vector prediction (TMVP) or optional temporal motion vector prediction (ATMVP). The method according to claim 15, wherein the first Merge candidate is pruned from a candidate from a first group of neighboring blocks, and wherein the first Merge candidate is not different from a candidate from a first group of neighboring blocks A candidate for a second set of neighboring blocks is trimmed. A device for video processing, comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions when executed by the processor enable the processor to implement the requirements described in any one of claim items 1 to 18 Methods. A computer program product stored on a non-transitory computer readable medium, the computer program product containing program code for executing the method according to any one of claim items 1 to 17.

Images