TW202337207A - Video coding method and apparatus thereof - Google Patents

Abstract

A video coder receives data for a block of pixels to be encoded or decoded as the current block of a current picture of a video. The video coder identifies multiple candidate coding modes applicable to the current block. The video coder identifies a first group of candidate coding modes that is a subset of the plurality of candidate coding modes. The first group of candidate coding modes may be the highest priority candidate coding modes identified based on cost. The number of candidate coding modes in the first group is less than the number of candidate coding modes in the plurality of candidate coding modes. The video coder selects a candidate coding mode in the first group of candidate coding modes. the video coder encodes or decodes the current block by using the selected candidate coding mode.

Description

Boundary matching of video codecs

This disclosure relates generally to video codecs. In particular, the present disclosure relates to ranking of candidate codec modes based on boundary matching.

Unless otherwise indicated herein, the methods described in this section are not in the art within the scope of the claims listed below and are not admitted to be in the art by inclusion in this section.

High-Efficiency Video Coding (HEVC) is an international video codec standard developed by the Joint Collaborative Team on Video Coding (JCT-VC). HEVC is based on a hybrid block-based motion compensation-like DCT transform coding and decoding architecture. The basic unit of compression, called a coding unit (CU), is a 2Nx2N square block. Each CU can be recursively divided into four smaller CUs until a predetermined minimum size is reached. Each CU contains one or more prediction units (PU).

Versatile Video Coding (VVC) is a codec designed to meet the upcoming needs of video conferencing, OTT (over-the-top) streaming media, mobile phones, etc. VVC is designed to provide a variety of functions to meet all video needs from low resolution and low bit rate to high resolution and high bit rate, high dynamic range (HDR), 360 omnidirectional, etc.

For each inter-predicted CU, motion parameters consisting of a motion vector, a reference picture index, a reference picture list usage index, and additional information are used for inter-prediction sample generation. Motion parameters can be sent explicitly or implicitly. When a CU is coded in skip mode, the CU is associated with a PU and has no significant residual coefficients, no codec motion vector increments, or reference picture indexes. The merging mode is specified, and the motion parameters of the current CU are obtained from neighboring CUs, including spatial and temporal candidates, as well as additional schedules introduced in VVC. Merge mode can be applied to any inter-predicted CU. An alternative to merge mode is explicit transmission of motion parameters, where motion vectors, corresponding reference picture indexes for each reference picture list and reference picture list usage flags, and other required information are sent explicitly per CU.

In addition to the inter codec features in HEVC, VVC also includes several new and improved inter prediction codec tools, listed below: - Extended merge prediction - Merge mode with MVD (MMVD) - Symmetric MVD (SMVD ) Send - Affine motion compensation prediction - Subblock-based temporal motion vector prediction (SbTMVP) - Adaptive motion vector resolution (AMVR) - Motion field storage: 1 / ^16th brightness sample MV storage and 8x8 motion field compression - Bi-prediction with CU level weight (BCW) - Bi-directional optical flow (BDOF) - Decoder side motion vector fineness Decoder side motion vector refinement (DMVR) - Geometric partitioning mode (GPM) - Combined inter and intra prediction (CIIP)

The following overview is illustrative only and is not intended to be limiting in any way. That is, the following summary is provided to introduce the concepts, highlights, benefits, and advantages of the novel and non-obvious technologies described herein. Selected, but not all, embodiments are further described in the detailed description below. Accordingly, the following summary is not intended to identify essential features of the claimed subject matter, nor is it intended to be used to determine the scope of the claimed subject matter.

Some embodiments provide a method of using costs to select candidate codec modes to encode or decode the current block. The video codec receives data for a block of pixels that will be encoded or decoded into the current block of the current picture of the video. The video codec identifies multiple candidate codec modes that are suitable for the current block. The video codec identifies a first candidate codec mode group, which is a subset of a plurality of candidate codec modes. The first candidate codec mode group may be the highest priority candidate codec mode identified based on cost. The number of candidate coding and decoding modes in the first candidate coding and decoding mode group is less than the number of candidate coding and decoding modes in the plurality of candidate coding and decoding modes. The video codec selects a candidate codec mode from the first candidate codec mode group. The video codec uses the selected candidate codec mode to encode or decode the current block.

In some embodiments, the plurality of candidate codec modes include merge candidates for the current block. Merging candidates for the current block may include (i) merging candidates using combined inter- and intra-prediction and/or (ii) merging candidates using affine transform motion compensated prediction. In some embodiments, the plurality of candidate codec modes include candidate codec modes corresponding to different combinations for refining distance and offset of motion information. In some embodiments, the plurality of candidate codec modes include candidate codec modes corresponding to different linear models for deriving predictions of chroma samples of the current block based on luma samples of the current block. son. In some embodiments, the plurality of candidate codec modes include candidate codec modes corresponding to different candidate weights for combining inter predictions in different directions.

In some embodiments, the first candidate codec mode group is the highest priority candidate codec mode identified based on the cost of the plurality of candidate codec modes. The codec may index the candidate codec modes in the identified candidate codec mode group or assign codewords to the candidate codec modes in the identified candidate codec mode group according to the priority of the candidate codec mode. In some embodiments, the cost of the candidate codec mode is calculated by comparing the following samples: (i) reconstructed samples adjacent to the current block and (ii) the current block generated according to the candidate codec mode along the current block boundary. prediction sample.

In some embodiments, each candidate codec mode in the plurality of candidate codec modes is assigned to a candidate codec mode group in the plurality of candidate codec mode groups. For example, in some embodiments, each candidate codec mode in the plurality of candidate codec modes is associated with an original candidate index, wherein each candidate codec mode is based on the original index modulo K or the original index divided by K The result is assigned to one candidate encoding and decoding mode group among the K candidate encoding and decoding mode groups. In some embodiments, candidate codec modes corresponding to spatial merging candidates are assigned to the same candidate codec mode group. In some embodiments, candidate codec modes corresponding to spatial merging candidates are assigned to different candidate codec mode groups. In some embodiments, candidate codec modes having merge candidates with motion differences less than a threshold are assigned to the same candidate codec mode group. In some embodiments, candidate codec modes that are merged candidates with a motion difference greater than a threshold are assigned to the same candidate codec mode group. In some embodiments, the encoder identifies one candidate codec mode group with the lowest representation cost among the plurality of candidate codec mode groups, and sends an index used to select a candidate from the identified candidate codec mode group Codec mode. The representative cost of an identified group may be the average, maximum, or minimum value of the costs (eg, boundary matching costs) of candidate codec modes for the identified group. In some embodiments, the encoder sends an index for selecting the set of candidate codec modes, and identifies from the selected set of candidate codec modes based on a cost (eg, a boundary matching cost) of the set of candidate codec modes. Candidate codec mode.

In the following detailed description, many specific details are set forth by way of examples in order to provide a thorough understanding of the relevant teachings. Any changes, derivations, and/or extensions based on the teachings described herein are within the scope of the present disclosure. In some cases, well-known methods, processes, components and/or circuits related to one or more example implementations disclosed herein may be described at a relatively high level without detail in order to avoid unnecessarily obscuring the present disclosure. aspects of teaching.

As video codecs improve, more codec tools are developed. However, the coding and decoding gains of these different coding and decoding tools are not additive. This is because (1) not all new codec modes can be candidate modes for a block, especially when syntax overhead is taken into account, and (2) as candidate modes for a block increase, longer codewords are needed to indicate from Codec modes for multiple candidate modes.

For example, after HEVC, new merging candidates (such as pairwise average merging candidates, HMVP merging candidates, etc.) are proposed to be added to the merging candidate list. The index of the best merge candidate is encoded/decoded to indicate the merge candidate selected by the current block. However, the number of merging candidates in the merging candidate list is limited to a predetermined number, so not all merging candidates can be added to the merging candidate list. As the number of merge candidates in the merge candidate list increases, the codeword length of the index of the best merge candidate also increases.

Some embodiments of the present disclosure provide a scheme for adaptively reordering candidate patterns. According to this scheme, the video encoder/decoder calculates the cost of each candidate mode, which can be a merge candidate and/or a candidate mode of another tool. The video codec prioritizes candidate modes based on cost. (In some embodiments, candidate patterns with smaller cost get higher priority. In some other embodiments, candidate patterns with smaller cost get lower priority.) The candidate patterns are then re-prioritized according to the priority order. Sort.

In some embodiments, the video codec uses a reduced, reordered set of candidate modes that only includes the top k candidate modes with higher priority (k < the number of all possible candidate modes). As the number of candidate patterns in the candidate pattern set is reduced, the syntax used to indicate the selected candidate pattern is also reduced. The index can be used to refer to the selected candidate patterns after reordering so that smaller index values can refer to candidate patterns with higher priority. (In other words, the index value refers to the index of the candidate pattern in the candidate list, and after the reordering is applied, the index value refers to the reordered index of the candidate pattern.) In some embodiments, for the candidate pattern with a higher priority Candidate mode, shorter codewords are used for encoding/decoding. The candidate mode with the highest priority can be implicitly set as the codec mode of the current block.

In some embodiments, the video codec prioritizes candidate modes in a set of candidate modes based on boundary matching. For each candidate mode, the boundary matching cost for encoding and decoding the current block using the candidate mode is calculated, and the priority of the candidate mode in the reordered candidate mode list is determined based on the boundary matching cost of the candidate mode.

Ⅰ , boundary matching cost

The boundary matching cost of a candidate mode refers to the discontinuity between the current prediction generated from the candidate mode (the prediction samples or predictors of the current block) and the adjacent reconstruction (the reconstructed samples within one or more adjacent blocks) Measurements (including top boundary match and/or left boundary match). Top boundary matching refers to the comparison of the current top prediction sample and the adjacent top reconstructed sample, and the left boundary matching refers to the comparison of the current left prediction sample and the adjacent left reconstructed sample.

Figure 1 shows reconstructed neighbor samples and predicted samples used in boundary matching. As shown in the figure, pred _x,0 is the predicted sample along the top boundary, reco _x,-1 is the reconstructed adjacent sample along the top boundary; where pred _0,y is the predicted sample along the left boundary, reco _{-1, y} is the reconstructed adjacent sample along the left boundary.

In some embodiments, a predetermined subset of the current predictions is used to calculate the boundary matching cost. For example, n rows of the top boundary within the current block and/or m rows of the left boundary within the current block may be used. In some of these embodiments, the n ₂ rows of the top adjacent reconstruction and/or the m ₂ rows of the left adjacent reconstruction may be used for boundary matching.

Example boundary matching cost is calculated according to the following formula, where n = 2, m = 2, n ₂ =2, m ₂ = 2:

In this example, two rows of predicted samples and two rows of reconstructed adjacent samples along the top and left boundaries are used to compute the difference metric (or similarity metric). The weights (a, b, c, d, e, f, g, h, i, j, k, l) can be any positive integer, for example, a = 2, b = 1, c = 1, d = 2 , e = 1, f = 1, g = 2, h = 1, i = 1, j = 2, k = 1, l = 1. Another example of calculating the boundary matching cost is as follows, where n = 2, m = 2, n ₂ = 1, m ₂ = 1:

The weights (a, b, c, g, h, i) can be any positive integer, such as a = 2, b = 1, c = 1, g = 2, h = 1, i = 1. Another example of calculating the boundary matching cost is as follows, where n = 1, m = 1, n ₂ = 2, m ₂ = 2:

The weights (d, e, f, j, k, l) can be any positive integer, for example, d = 2, e = 1, f = 1, j = 2, k = 1, l = 1. Another example of calculating the boundary matching cost is as follows, where n = 1, m = 1, n ₂ = 1, m ₂ = 1:

The weights (a, c, g, i) can be any positive integer, such as a = 1, c = 1, g = 1, i = 1. Another example of calculating the boundary matching cost is as follows, where n = 2, m = 1, n ₂ = 2, m ₂ = 1:

The weights (a, b, c, d, e, f, g, i) can be any positive integer, for example, a = 2, b = 1, c = 1, d = 2, e = 1, f = 1, g = 1, i = 1. Another example of calculating the boundary matching cost is as follows, where n = 1, m = 2, n ₂ = 1, m ₂ = 2:

The weights (a, c, g, h, i, j, k, l) can be any positive integer, for example, a = 1, c = 1, g = 2, h = 1, i = 1, j = 2. k = 1, l = 1. In summary, n or n ₂ can be any positive integer such as 1, 2, 3, 4, etc., and m or m ₂ can be any positive integer such as 1, 2, 3, 4, etc. In some embodiments, n and/or m vary with block width, height, or area. For example, for larger blocks (area > threshold), m becomes larger, and the threshold can be 64, 128, or 256. When area > threshold, m can increase from 1 to 2, or from 1 or 2 to 4. As another example, for taller blocks (height > threshold * width), m becomes larger and/or n becomes smaller, the threshold can be 1, 2 or 4; when height > threshold * width, m may increase from 1 to 2, Or increase from 1 or 2 to 4. For another example, for larger blocks (area > threshold), n becomes larger, and the threshold may be 64, 128 or 256; when area > threshold, n increases from 1 to 2, or from 1 or 2 to 4. For another example, for wider blocks (width > threshold * height), n becomes larger and/or m becomes smaller, threshold = 1, 2 or 4; when width > threshold * height, n increases from 1 to 2; when width > When threshold*height, n increases from 1 or 2 to 4.

As another example, n and/or m may be defined in the video codec standard or depend on the code in CU/CB, PU/PB, TU/TB, CTU/CTB, tile, segment, picture level, sps level and/or pps The codec video syntax sent out in the level or the codec video syntax to be parsed.

In some embodiments, when the current block is at the top boundary within a CTU row, top boundary matching is not used and/or only left boundary matching is used. (Adjacent reconstructed samples at crossing CTU rows are not used.) In some embodiments, when the current block is at the left boundary within the CTU, left boundary matching is not used and/or only top boundary matching is used. In some embodiments, when the current block is taller (height > threshold*width), only left boundary matching is used. In some embodiments, only top boundary matching is used when the current block is wider (width > threshold*height).

In some embodiments, the upper left adjacent reconstructed sample (reco _-1,1 ) may be used for boundary matching. For example, the boundary matching cost plus:

In some embodiments, when calculating the boundary matching cost, residuals can also be added to the current prediction (or predictor of the current block) to reconstruct the samples of the current block, and the reconstructed samples of the current block are used to calculate the boundary. Matching costs. For example, the residual is generated by recovering DC and/or all AC coefficients or any subset of AC coefficients after the transform process. The transformation process may use any predetermined transformation kernel for secondary transformation and/or primary transformation. For example, the transformation kernel used for secondary transformation refers to the Low Frequency Non-Separable Transform (LFNST) transformation kernel. For example, the transformation kernel used for primary transformation refers to DCT2 and/or any transformation kernel used for multiple transform selection (Multiple Transform Selection, MTS for short), such as DST7. For example, the transformation kernel refers to the actual transformation applied in the current block's transformation module.

Ⅱ , as a merge candidate for candidate mode

Some embodiments of the present disclosure provide a solution for reordering and/or reducing merge candidates. In some embodiments, the index of the best merge candidate (index_best_merge) does not refer to the order of the merge candidates in the merge candidate list, but to the priority order based on the boundary matching cost. In some embodiments, only the top k merge candidates (with higher priority) may be candidate patterns.

For example, for the merge candidate list {cand0, cand1, cand2, cand3, cand4, cand5, ...}, the boundary matching cost of the merge candidate is calculated as {cost_cand0, cost_cand1, cost_cand2, ...}, so that cost_cand0 refers to The boundary matching cost of cand0, cost_cand1 refers to the boundary matching cost of cand1, cost_cand2 refers to the boundary matching cost of cand2, and so on. The video codec then reorders {cand0, cand1, cand2,...} based on the boundary matching cost. For example, cost_cand0 > cost_cand1 > cost_cand2 > cost_cand3 > cost_cand4 > cost_cand5, then the reordered merge candidates become {cand5, cand4, cand3, cand2, cand1, cand0}, and index_best_merge 0 can refer to cand5 (the merge candidate with the smallest cost uses the shortest codeword is sent), index_best_merge 1 can refer to cand4, index_best_merge 2 can refer to cand3, and so on.

In some embodiments, the candidate pattern set only includes the top k merge candidates (ordered according to boundary matching cost), rather than being determined by a constant (eg, MaxNumMergeCand in VVC, which can be 6). For example, in some embodiments, k may be 4, and the signaling for each merge candidate may include: index_best_merge 0 refers to cand5 with codeword 0 (the merge candidate with the largest cost is sent using the shortest codeword); index_best_merge 1 refers to cand4 with codeword 10; index_best_merge 2 refers to cand3 with codeword 110; and index_best_merge 3 refers to cand2 with codeword 111. Otherwise, for example, if cost_cand0 < cost_cand1 < cost_cand2 < cost_cand3 < cost_cand4 < cost_cand5, the order of the merge candidates can be the same as the original list without reordering: index_best_merge 0 refers to cand0 with codeword 0; index_best_merge 1 refers to cand0 with codeword 0 cand1 of 10; index_best_merge 2 refers to cand2 with codeword 110; and index_best_merge 3 refers to cand3 with codeword 111, and so on.

In some embodiments, the merge candidate with the largest cost is sent using the shortest codeword. For another example, if cost_cand0＜cost_cand1＜cost_cand2＜cost_cand3＜cost_cand4＜cost_cand5, then the reordered merge candidates become {cand5, cand4, cand3, cand2, cand1, cand0}, so that index_best_merge 0 refers to cand5 with codeword 0 (maximum cost using the shortest codeword); index_best_merge 1 refers to cand4 with codeword 10; and index_best_merge 2 refers to cand3 with codeword 110.

In some sub-embodiments, whether to use the reduced and reordered merge candidate list (reordered based on boundary matching cost and limited to k candidates) or the original merge candidate list (without reordering) is decided based on predetermined rules (e.g. implicitly Dependent on block width, block height or block area, or explicitly on CU/CB, PU/PB, TU/TB, CTU/CTB, tile, fragment level, picture level, SPS level and/or PPS level) . When a reduced and reordered merge candidate list is used, the entropy encoding and decoding context used for transmission may be different than when the original merge candidate list was used.

In some embodiments, for the index_best_merge variable, smaller values are encoded with shorter codeword lengths. The index_best_merge variable can be encoded and decoded with truncated unary codewords.

In some embodiments, reordering is applied only to a subset of the merge candidate list. For example, the subset can refer to the original top n candidates cand0, cand1, and cand2, so that index_best_merge 0/1/2 refers to the priority order based on boundary matching, and index_best_merge 3/4/5 refers to the original cand 3/4/5. For another example, the subset refers to the original last n candidates cand3, cand4, and cand5. Then index_best_merge 3/4/5 refers to the priority order based on boundary matching, and index_best_merge 0/1/2 refers to the original cand 0/1/2. As another example, the subset may refer to spatial merging candidates.

In some embodiments, the best merge candidate is inferred to be the merge candidate with the smallest boundary matching cost among all merge candidates. In some embodiments, the best merge candidate is inferred to be the merge candidate with the largest boundary matching cost among all merge candidates. In some of these embodiments, index_best_merge is not sent/parsed by the encoder/decoder, and may be inferred to be 0.

In some embodiments, merge candidates are divided into several groups. The boundary matching cost for each group is calculated. The promising group is implicitly identified as the group with the highest priority. If more than one merge candidate is included in the identified promising group, a reduced merge index is sent/parsed to indicate the merge candidates from the promising group. Since the number of merge candidates in each group is less than the number of merge candidates in the original merge candidate list, the reduced merge index will occupy fewer codewords than the original merge index.

In some embodiments, if k groups are used and the number of merge candidates in the original merge candidate list is N, then the number of merge candidates in each group is "N/k" and the grouping rules depend on the merge index and/or or merge type. For example, merge candidates with the same Merge Index % k (remainder of the modulo operator) value are in the same group. As another example, merge candidates with the same "merge index/k" (quotient of the division operator) value are in the same group. As another example, spatial merging candidates are in the same group. As another example, k varies with block width, block height, and/or block area. For another example, merge candidates with small motion differences are in the same group. Motion differences include mv differences and/or reference picture differences. An example of calculating the mv difference (denoted as mv_diff) between candidate 0 and candidate 1 is: mv_diff = |(mvx_cand0 –mvx_cand1)| + |(mvy_cand0 –mvy_cand1)|

If the reference pictures are the same and/or the mv difference is less than a predetermined threshold, the motion difference is smaller.

In some embodiments, when a group contains more than one merge candidate, the cost of the group (the representative cost of the group) is the average cost of the costs of all merge candidates in the group. In some sub-embodiments, when a group contains more than one merge candidate, the representative cost of the group is the average, maximum or minimum cost from the costs of all merge candidates in the group.

In some embodiments, the merge candidates are divided into several subsets, and the selection of subsets is explicitly sent to the decoder. Furthermore, selection of merge candidates in the subset of merge candidates is performed using boundary matching costs. Subset partitioning rules can depend on the merge index and/or merge type. For example, merge candidates with the same "Merge Index % k" value are in the same subset. As another example, merge candidates with the same "merge index/k" value are in the same subset (k may vary with block width, block height, and/or block area.) As another example, spatial merge candidates are in different subsets. For another example, merge candidates with larger motion differences are in the same subset. Motion differences include mv differences and/or reference picture differences. An example of calculating the mv difference (denoted as mv_diff) between candidate 0 and candidate 1 is: mv_diff = |(mvx_cand0 –mvx_cand1)| + |(mvy_cand0 –mvy_cand1)|

If the reference pictures are different and/or the mv difference is larger than a predetermined threshold, the motion difference is large.

In some embodiments, the CU's merge candidates include one or more of the following candidates: (1) spatial merge candidates or spatial MVPs from spatially neighboring CUs, (2) temporal MVPs from co-located CUs, (3) from the FIFO table of history-based MVP, (4) pairwise average MVP, and/or (5) zero MV.

In some embodiments, the merging candidates in this section refer to merging candidates for combined inter and intra prediction (CIIP). The prediction samples within the current block are generated as CIIP processing. In some embodiments, a merge candidate in this section refers to a merge candidate of a sub-block merge candidate, such as an affine merge candidate. Prediction samples within the current block are generated by affine processing, e.g., block-based affine transform motion compensated prediction.

a. Spatial merge candidates

At most four merging candidates are selected among the candidates located at positions around the CU, as shown in Figure 2, which shows the locations of spatial merging candidates. The derivation sequence is B ₀ , A ₀ , B ₁ , A ₁ , B ₂ . Position B ₂ is only considered if one or more CUs at position B ₀ , A ₀ , B ₁ , A ₁ are not available (for example, because it belongs to another segment or tile) or are intra-coded. After the candidate at position A ₁ is added, a redundancy check is performed on the remaining candidates to ensure that candidates with the same motion information are excluded from the list, thereby improving encoding and decoding efficiency. In order to reduce the computational complexity, not all possible candidate pairs are considered in the mentioned redundancy check. Candidates are added to the list only if the corresponding candidates used for redundancy checking do not have the same motion information. Only the following candidate pairs are considered: (A ₁ , B ₁ ), (A ₁ , A ₀ ), (A ₁ , B ₂ ), (B ₁ , B ₀ ), (B ₁ , B ₂ ).

b. Time merge candidates

Only one time merge candidate is added to the merge candidate list. Specifically, in the derivation of this temporal merging candidate, the scaled motion vector is derived based on the co-located CU, which belongs to the co-located reference picture. The reference picture list and reference index used to derive co-located CUs are sent explicitly in the segment header. The scaled motion vector of the temporal merging candidate is obtained as shown by the dashed line in Figure 3, which shows the scaling of the motion vector of the temporal merging candidate. The scaled motion vector is scaled from the motion vector of the co-located CU using picture order count (POC) distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture and the current picture, and td It is defined as the POC difference between the reference picture of the co-located picture and the co-located picture. The reference picture index of the temporal merge candidate is set equal to zero. The position of the temporal candidate is selected between candidates C ₀ and C ₁ , as shown in Figure 4 , which shows the candidate position of the temporal merging candidate of the current block. If the CU at position C ₀ is not available, is intra-coded, or is outside the current CTU row, then position C ₁ is used as a temporal merging candidate. Otherwise, position C ₀ is used to derive temporal merge candidates.

c. History-Based Merger Candidates

History-based motion vector prediction (HMVP) merge candidates are added to the merge list after spatial MVP and TMVP. In this method, the motion information of the previous codec block is stored in the table and used as the MVP of the current CU. During the encoding/decoding process, a table with multiple HMVP candidates is maintained. The table is reset (emptied) when a new CTU row is encountered. Whenever a non-subblock inter-codec CU exists, the associated motion information is added to the last entry of the table as a new HMVP candidate.

The HMVP table size S is set to 6, which means that up to 5 history-based MVP (HMVP) candidates can be added to the table. When inserting a new motion candidate into the table, the constrained first-in-first-out (FIFO) rule is applied, where a redundancy check is first applied to check whether the same motion candidate exists in the data table. HMVP. If found, the same HMVP is removed from the table and all subsequent HMVP candidates are moved forward and the same HMVP is inserted into the last entry of the table.

HMVP candidates can be used in the merge candidate list building process. The latest several HMVP candidates in the table are checked sequentially and inserted into the candidate list after the TMVP candidates. Redundancy checks are applied from HMVP candidates to spatial or temporal merge candidates.

In some embodiments, in order to reduce the number of redundant check operations, the last two entries in the table perform redundant checks on the A ₁ and B ₁ space candidates respectively. Once the total number of available merge candidates reaches the maximum allowed merge candidate minus 1, HMVP's merge candidate list construction process is terminated.

d. Pairwise average merge candidates

Pairwise average candidates are generated by averaging a predetermined pair of candidates in the existing merge candidate list using the first two merge candidates. The first merge candidate is defined as pOCand, and the second merge candidate may be defined as p1Cand. The average motion vector is calculated based on the availability of the motion vectors of p0Cand and p1Cand to each reference list respectively. If two motion vectors are available in a list, they will be averaged even if they point to different reference pictures, and their reference picture is set to the reference picture belonging to one of p0Cand and p1Cand or for p0Cand and p1Cand one of the reference pictures (such as the reference picture set to p0Cand or the reference picture set to p1Cand)"; if only one motion vector is available, the merge candidate corresponding to that motion vector is directly used; if no motion vector is available, then keep this list invalid. In addition, if the half-pel inerpolation filter index of p0Cand and p1Cand is different, it is set to 0. In some embodiments, when the merge list is not full, the After the average merge candidates are added, finally zero MVPs are inserted until the maximum number of merge candidates is encountered.

e. Combined inter and intra prediction ( Combined inter and intra prediction , abbreviation CIIP )

When the CU is encoded in merged mode, the extra flag may be sent if the CU contains at least 64 luma samples (i.e., CU width times CU height is equal to or greater than 64), and if both CU width and CU height are less than 128 luma samples. To indicate whether combined inter/intra prediction (CIIP) mode is applied to the current CU. CIIP prediction combines inter prediction signals with intra prediction signals. The inter prediction signal P _inter in CIIP mode is derived using the same inter prediction process as applied to the conventional merge mode; and the intra prediction signal P _intra is derived according to the conventional intra prediction process using one or more intra prediction modes: Planar mode or one or more intra prediction modes derived from a predetermined mechanism. For example, in the following, the predetermined mechanism is based on the neighboring reference areas (templates) of the current block. The intra prediction mode of a CU is implicitly derived from the neighboring templates at the encoder and decoder, rather than being sent to the decoder as the exact intra prediction mode bits. Generate prediction samples of the template for each candidate pattern's template reference sample. The cost is calculated as the SATD between the predicted sample and the reconstructed sample of the template. The intra prediction mode with minimum cost and/or some intra prediction mode with smaller cost is selected and used for intra prediction of the CU. Candidate modes can be all MPMs and/or any subset of MPMs, such as the 67 intra prediction modes in VVC or extended to 131 intra prediction modes. The intra- and inter-prediction signals are combined using a weighted average, where the weight values are calculated based on the codec mode of the top and left neighboring blocks. CIIP predicts P The composition of _CIIP is as follows: ( wt is the weight value)

f. Affine merge candidates

Objects in the video may have different types of motion, including translational motion, zoom-in/zoom-out motion, rotational motion, perspective motion, and other irregular motions. In some embodiments, block-based affine transform motion compensated prediction is used to account for these different types of motion. VVC provides block-based affine transform motion compensated prediction. Specifically, the affine motion field of the current block can use the motion information of two control points (for example, at the upper right corner and the upper left corner of the block) (4 parameters) or the motion information of three control points (for example, at the upper right corner of the block). corner, upper left corner and lower left corner) (6 parameters). For a 4-parameter affine motion model, the motion vector at sample position (x, y) in the block is derived as:

For a 6-parameter affine motion model, the motion vector at sample position (x, y) in the block is derived as:

Among them (mv _0x ,mv _0y ) is the motion vector of the upper left corner control point, (mv _1x ,mv _1y ) is the motion vector of the upper right corner control point, (mv _2x ,mv _2y ) is the motion vector control point of the lower left corner.

Affine merge mode, or AF_MERGE mode, can be applied to CUs with both width and height greater than or equal to 8. In this mode, the motion vector at the control point (CPMV) of the current CU is generated based on the motion information of spatially adjacent CUs. There can be up to five CPMVP candidates, and so is sent to indicate which candidate is to be used for the current CU. The following three types of CPMV candidates are used to form the affine merge candidate list: (1) inherited affine merge candidates, which are inferred from the CPMVs of neighboring CUs; (2) constructed affine merge candidate CPMVPs, which use the CPMVs of neighboring CUs. Translational MV export; (3) Zero MV.

In VVC, there are at most two inherited affine candidates from the affine motion models of neighboring blocks, one from the left neighboring CU (left predictor) and one from the top neighboring CU (top predictor). For the left predictor, the scan order is A0->A1, and for the top predictor, the scan order is B0->B1->B2. Only the first inheritance candidate on each side is selected. No pruning check is performed between two inherited candidates. When a neighboring affine CU is identified, its control point motion vector is used to derive the CPMVP candidate in the affine merge list of the current CU. When the adjacent affine CU adopts the 6-parameter affine mode for encoding and decoding, the three CPMVs of the current CU are calculated based on the motion vectors of the upper left corner, upper right corner and lower left corner of the adjacent affine CU. When an adjacent affine CU uses a 4-parameter affine model for encoding and decoding, two CPMVs of the current CU are calculated based on the motion vectors of the upper left corner and upper right corner of the adjacent affine CU.

The constructed affine candidates are constructed by combining the adjacent translational motion information of each control point. The motion information of the control point comes from the spatial adjacent blocks (A0, A1, A2, B0, B1, B2, B3) and temporal adjacent blocks of the current block. CPMV _k (k=1, 2, 3, 4) represents the MV of the kth control point. For CPMV1, the B2->B3->A2 blocks are checked and the MV of the first available block is used. For CPMV ₂ , the B1->B0 blocks are checked, and for CPMV3, the A1->A0 blocks are checked. If TMVP is available, use it as CPMV4.

After obtaining the MVs of the four control points, affine merging candidates are constructed based on these motion information. The following combinations of control point MVs are used in the construction: {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, { CPMV1, CPMV3}. The combination of 3 CPMVs builds a 6-parameter affine merge candidate, and the combination of 2 CPMVs builds a 4-parameter affine merge candidate. To avoid motion scaling processing, relevant combinations of control point MVs are discarded if their reference indices are different. After the inherited affine merge candidates and the constructed affine merge candidates have been checked, if the candidate list is still not full, zero MV is inserted at the end of the list.

Ⅲ , MMVD as a candidate model

Merge Mode with Motion Vector Difference (MMVD) is a new codec tool suitable for the Versatile Video Coding (VVC) standard. Different from the conventional merging mode (in which the implicitly derived motion information is directly used to generate prediction samples for the current CU), in MMVD, the derived motion information is further refined by motion vector difference MVD. MMVD also extends the candidate list of merge modes by adding additional MMVD candidates based on predetermined offsets (also known as MMVD offsets).

The MMVD flag may be sent after the skip flag and merge flag are sent to specify whether MMVD mode is used for the CU. If MMVD mode is used, the selected merge candidates will be refined by MVD information. MVD information also includes merge candidate flags, a distance index that specifies the magnitude of motion, and an index that indicates the direction of motion. The merge candidate flag is sent to specify which of the first two merge candidates is to be used as the starting MV.

The distance index is used to specify motion amplitude information by indicating a predetermined offset from the starting MV. The offset can be added to the horizontal or vertical component of the starting MV. An example mapping from distance index to predetermined offset is specified in the following Table III-1: Table III-1 . Distance Index distance index 0 1 2 3 4 5 6 7 Offset ( in luma samples ) 1/4 1/2 1 2 4 8 16 32  

The direction index represents the direction of the MVD relative to the starting point. The direction index can represent one of the four directions shown in Table III-2. Table III-2. MV offset symbols specified by direction index distance index 00 01 10 11 x- axis + – N/A N/A y- axis N/A N/A + –  

It should be noted that the meaning of the MVD mark may differ depending on the information of the starting MV. When the starting MV is an un-prediction MV or a bi-prediction MV, both lists point to the same side of the current picture (i.e., the picture order count or POC of the two reference pictures are both greater than the POC of the current picture, or both are less than POC of the current picture), the symbols in Table III-2 specify the symbols of the MV offset added to the starting MV. When the starting MV is a bidirectional prediction MV, the two MVs point to different sides of the current picture (that is, the POC of one reference is greater than the POC of the current picture, and the POC of the other reference is smaller than the POC of the current picture). Each of the MVs in Table III-2 symbols specifies the sign of the MV offset added to the list 0 MV component of the starting MV, the sign of the list 1 MV has the opposite value. In some embodiments, the predetermined offset of the MMVD candidate (MmvdOffset) is derived or represented as a distance value (MmvdDistance) and a direction sign (MmvdSign or MmvdDirection).

Figure 5 conceptually illustrates MMVD candidates and their corresponding offsets. The figure shows the merge candidate 510 as the starting MV and several MMVD candidates in the vertical and horizontal directions. Each MMVD candidate is derived by applying an offset to the starting MV 510. For example, MMVD candidate 522 is derived by adding an offset of 2 to the horizontal component of merge candidate 510 , while MMVD candidate 524 is derived by adding an offset of -1 to the vertical component of merge candidate 510 . MMVD candidates that have an offset in the horizontal direction, such as MMVD candidate 522, are called horizontal MMVD candidates. MMVD candidates with an offset in the vertical direction, such as MMVD candidate 524, are called vertical MMVD candidates.

In some embodiments, a candidate pattern reduction/reordering scheme is used to reorder MMVD candidates to improve specific syntax elements. The following is the syntax table of MMVD in the VVC standard. if( mmvd_merge_flag[ x0 ][ y0 ] = = 1 ) { if(MaxNumMergeCand＞1) mmvd_cand_flag [x0][y0] ae(v) mmvd_distance_idx [x0][y0] ae(v) mmvd_direction_idx [x0][y0] ae(v)

The syntax element mmvd_cand_flag [x0][y0] specifies whether the first (0) or the second (1) candidate in the merge candidate list with the syntax element mmvd_distance_idx[ x0 ][ y0 ] and mmvd_direction_idx derived motion vector difference. The array index x0, y0 specifies the position (x0, y0) of the upper left luma sample in the codec block considered relative to the upper left luma sample of the picture. When mmvd_cand_flag[x0][y0] does not exist, it is inferred to be equal to 0.

The syntax element mmvd_distance_idx [x0][y0] specifies the index used to derive the variable MmvdDistance[x0][y0], as described below. The array index x0, y0 specifies the position (x0, y0) of the top-left luma sample in the codec block considered relative to the top-left luma sample of the picture. mmvd_distance_idx[x0][y0] MmvdDistance[x0][y0] ph_mmvd_fullpel_only_flag = = 0 ph_mmvd_fullpel_only_flag = = 1 0 1 4 1 2 8 2 4 16 3 8 32 4 16 64 5 32 128 6 64 256 7 128 512

The syntax element mmvd_direction_idx[x0][y0] specifies the index used to derive the variable MmvdSign[x0][y0], as described below. The array index x0, y0 specifies the position (x0, y0) of the top-left luma sample in the codec block considered relative to the top-left luma sample of the picture. mmvd_direction_idx[x0][y0] MmvdSign[x0][y0][0] MmvdSign[ x0 ][ y0 ][ 1 ] 0 +1 0 1 −1 0 2 0 +1 3 0 −1

In some embodiments, the syntax element mmvd_cand_flag may be improved by using a reduced/reordered candidate pattern list. Specifically, the boundary matching cost is calculated for each MMVD mode, including MMVD mode 0, in which the MMVD with the first candidate is in the merge candidate list; and MMVD mode 1, in which the MMVD with the second candidate is in the merge candidate list. In some embodiments, after reordering according to cost, if the cost of MMVD mode 0 > the cost of MMVD mode 1 , mmvd_cand_flag equal to 0 indicates MMVD mode 1 and mmvd_cand_flag equals 1 indicates MMVD mode 0. Optionally, mmvd_cand_flag is implicit, and the first or second candidate in the merge candidate list (with the smallest cost) is used for MMVD.

In some embodiments, after reordering according to boundary matching cost, if the cost of MMVD mode 0 < the cost of MMVD mode 1, mmvd_cand_flag equal to 0 indicates MMVD mode 1 and mmvd_cand_flag equals 1 indicates MMVD mode 0. Alternatively, mmvd_cand_flag is implicit and the first or second candidate in the merge candidate list (with the largest cost) is used for MMVD.

In some embodiments, the sending of syntax elements mmvd_cand_flag, mmvd_distance_idx and mmvd_direction_idx may be improved by using a reduced/reordered list of candidate modes. A joint indication (denoted as MMVD_joint_idx) is sent/parsed/allocated to specify the selected combination of mmvd_cand_flag, mmvd_distance_idx and mmvd_direction_idx. The value range of MMVD_joint_idx is from 0 to the calculated value of "MMVD candidate number" (such as 2) * "MMVD distance number" (such as 8) * "MMVD direction number" (such as 4) minus one (such as 63).

In some embodiments, a boundary matching cost is calculated for each MMVD combination. After reordering different MMVD combinations based on cost, MMVD_joint_idx can be used to select an MMVD combination based on the reordering. For example, in some embodiments, if the cost of MMVD combination 0 > the cost of MMVD combination 1 >..., then MMVD_joint_idx equal to 0 indicates MMVD combination 63, and MMVD_joint_idx equal to 63 indicates MMVD combination 0. In some embodiments, MMVD_joint_idx is implicit and the MMVD combination with the minimum cost is used for MMVD. In some embodiments, the number of MMVD combinations is reduced and MMVD combinations with smaller costs are retained in the candidate mode set. The codewords used to send/parse MMVD_joint_idx are thus reduced.

In some embodiments, after reordering different MMVD combinations based on cost, MMVD_joint_idx may be used to select an MMVD combination based on the reordering. In some embodiments, if the cost of MMVD combination 0 < the cost of MMVD combination 1 <..., then MMVD_joint_idx equal to 0 refers to MMVD combination 63, and MMVD_joint_idx equal to 63 refers to MMVD combination 0. In some embodiments, MMVD_joint_idx is implicit and the most costly MMVD combination is used for MMVD. In some embodiments, the number of MMVD combinations is reduced, and MMVD combinations with larger costs are retained in the candidate mode set. The codewords used to send/parse MMVD_joint_idx are thus reduced. Similar methods can be used to improve the sending of mmvd_distance_idx and/or mmvd_direction_idx.

Ⅳ.BCW Weights as candidate patterns

Bi-prediction with CU-level Weight (BCW for short) is a coding and decoding tool used to enhance bi-directional prediction. BCW allows applying different weights to L0 predictions and L1 predictions, and then combining them to generate bidirectional predictions for CUs. For CUs to be encoded and decoded by BCW, a weighting parameter w is sent for L0 and L1 prediction so that the bidirectional prediction result Pbi-pred is calculated based on w according to the following formula:

P0 represents the pixel value predicted by L0 MV (or L0 prediction). P1 represents the pixel value predicted by L1 MV (or L1 prediction). P _bi-pred is the weighted average of P0 and P1 according to w. For low latency pictures, even pictures using reference frames with small picture order count (POC), possible values of w include {-2, 3, 4, 5, 10}, these are also called BCW Candidate weight. For non-low latency images, possible values for w (BCW candidate weight) include {3, 4, 5}. In some embodiments, in order to find the best w for encoding and decoding the current CU, instead of searching all possible values of w for all candidate bi-predictive MV positions, the LC-RDO stage can adopt an interleaved search mode to search for BCW weights The optimal value of parameter w. More weights may be supported, as shown below. For example, for merge mode, the weights extend from {-2, 3, 4, 5, 10} to {-4, -3, -2, -1, 1, 2, 3, 4, 5, 6, 7, 9 , 10, 11, 12} or any subset of above. When negative bidirectional prediction weights are not supported, the merge mode weights are extended from {-2, 3, 4, 5, 10} to {1, 2, 3, 4, 5, 6, 7}. In addition, the negative bidirectional prediction weights of the non-merging mode are replaced with positive weights, i.e., the weights {-2, 10} are replaced with {1, 7}.

In some embodiments, the proposed candidate pattern reduction/reordering scheme is used to reorder BCW candidates to improve the sending of specific syntax elements, eg, bcw_idx. The following is the syntax table of BCW in the VVC standard. if( sps_bcw_enabled_flag && inter_pred_idc[ x0 ][ y0 ] = = PRED_BI && luma_weight_l0_flag[ ref_idx_l0 [ x0 ][ y0 ] ] = = 0 && luma_weight_l1_flag[ ref_idx_l1 [ x0 ][ y0 ] ] = = 0 && chroma_weight_l0_ flag[ ref_idx_l0 [ x0 ][ y0 ] ] = = 0 && chroma_weight_l1_flag[ ref_idx_l1 [ x0 ][ y0 ] ] = = 0 && cbWidth * cbHeight ＞= 256 ) bcw_idx [x0][y0] ae(v)

The syntax element bcw_idx [x0][y0] specifies the weight index of bidirectional prediction with CU weights. The array index x0, y0 specifies the position (x0, y0) of the upper left luma sample of the codec block considered relative to the upper left luma sample of the picture. For each bi-predicted CU, the weight w is determined in one of the following two ways: 1) for non-merged CUs, the weight index is sent after the motion vector difference; 2) for merged CUs, the weight index is sent from the adjacent CU according to the merge candidate index inferred from the block. grammatical structure syntax elements binarization handle input parameters bcw_idx[ ][ ] TR cMax = NoBackwardPredFlag ? 4: 2, cRiceParam = 0

To reduce and reorder BCW candidates, the boundary matching cost of each BCW candidate weight is calculated, and different BCW candidate weights are reordered according to the cost. In one embodiment, the cost of predetermining candidates may be reduced. For example, a predetermined candidate refers to a candidate inferred from one or more neighboring blocks. In another embodiment, the candidate weights include (only) two adjacent bidirectional prediction weights (i.e. ±1) and the inherited (inferred) bidirectional prediction weight. In some embodiments, bcw_idx equal to 0 represents the BCW candidate weight with the minimum cost, and bcw_idx equal to 4 represents the BCW candidate weight with the maximum cost. In some embodiments, bcw_idx is implicit and the BCW candidate weight with minimum cost is used. In some embodiments, bcw_idx equal to 0 represents the BCW candidate weight with maximum cost, and bcw_idx equal to 4 represents the BCW candidate weight with minimum cost. In some embodiments, bcw_idx is implicit and the BCW candidate weight with the largest cost is used.

In some embodiments, only the top k BCW weights (with higher priority due to cost) can be used as candidate weights in the candidate pattern set (the original number of BCW weights is 5). In some embodiments, the number of BCW weights in the candidate pattern set is k (e.g., 3), and each BCW weight is sent as follows: bcw_idx 0 refers to the BCW candidate with the highest priority using codeword 0; bcw_idx 1 is refers to the BCW candidate with the second highest priority using codeword 10; bcw_idx 2 refers to the BCW candidate with the third highest priority using codeword 11.

In some sub-embodiments, whether to use reduced and reordered BCW candidates (reordered based on boundary matching cost and limited to k candidates) or original BCW candidates (without reordering) is based on predetermined rules (e.g. implicitly depends on the block Width, block height, or block area, or one or more explicitly dependent on CU/CB, PU/PB, TU/TB, CTU/CTB, tile, fragment level, picture level, SPS level, and/or PPS level flag.) When a reduced and reordered merge candidate list is used, the entropy encoding and decoding context used for transmission may be different than when the original merge candidate list was used.

In some embodiments, BCW weights are divided into groups. Next, the boundary matching cost for each group is calculated. Then, a promising group is implicitly selected as the group with the highest priority. If more than one BCW weight is included in the hopeful group, a reduced BCW weight index is sent/parsed to indicate the BCW weight from the hopeful group. Since the number of BCW weights in each group is smaller than the number of BCW weights in the original BCW weight set, the reduced BCW weight index may occupy fewer codewords than the original BCW weight index.

In some embodiments, if k groups are used and the number of BCW weights in the original BCW weight set is N, then the number of BCW weights in each group is "N/k" and the grouping rules depend on the BCW weight index and/ Or BCW weight value. For example, BCW weights with the same "BCW weight index %k" value belong to the same group. For another example, BCW weights with the same "BCW weight index/k" value belong to the same group. As another example, BCW weights with similar values are in the same group (e.g., BCW weights with negative values are in one group and/or BCW weights with positive values are in another group). In some embodiments, k (the number of candidates in the group) varies with block width, block height, and/or block area.

In some embodiments, when a group contains more than one BCW weight, the cost of the group is the average cost from the costs of all BCW weights in the group. In some embodiments, when a group contains more than one BCW weight, the cost of the group is the average, maximum, or minimum cost from the costs of all BCW weights in the group. In some embodiments, the BCW weights are divided into several subsets, and the selection of subsets is explicitly sent to the decoder. And subset selection is performed by using boundary matching costs.

In some embodiments, the BCW weights are divided into multiple subsets. In some embodiments, BCW weights with similar values are divided into different subsets, such that BCW weights of the same subset have larger differences from each other. For example, in some embodiments, weights with negative values may be divided into different groups. For another example, in some embodiments, weights whose weight differences are greater than a predetermined threshold are classified into the same group. The predetermined threshold may be fixed or explicitly determined. This may help improve hit rate on boundary matches.

Ⅴ , linear model as candidate model

Cross Component Linear Model (CCLM) or Linear Model (LM) mode is a cross component prediction mode in which the chrominance component of a block is predicted from the co-located reconstructed luminance samples by a linear model . The parameters of the linear model (e.g., scale and offset) are derived from the reconstructed luma and chroma samples adjacent to the block. For example, in VVC, the CCLM mode exploits inter-channel dependence to predict chroma samples from reconstructed luma samples. The prediction is made using a linear model of the form:

represents the predicted chroma samples in a CU (or the predicted chroma samples of the current CU), and Represents a downsampled reconstructed luminance sample of the same CU (or the corresponding reconstructed luminance sample of the current CU).

CCLM model parameters (scaling parameter) and (offset parameter) is derived based on up to four adjacent chroma samples and their corresponding downsampled luma samples. In LM_A mode (also denoted LM-T mode or INTRA_T_CCLM), only the upper (also called top) neighbor template is used to calculate the linear model coefficients. In LM_L mode (also denoted LM-L mode or INTRA_L_CCLM), only the left template is used to calculate the linear model coefficients. In LM-LA mode (also denoted LM-LT mode or INTRA_LT_CCLM), both the left and upper templates are used to calculate linear model coefficients. The proposed method can be applied to any cross-component mode and is not limited to the LM mode. For example, the cross-component mode uses the information of the first color component (such as Y) to predict the second/third color component (such as Cb/Cr).

In some embodiments, the proposed candidate mode reduction/reordering scheme is used to reorder LM candidates to improve signaling of specific syntaxes, such as cclm_mode_idx. The following is the syntax table of LM in the VVC standard. if(cclm_mode_flag) cclm_mode_idx ae(v)

The syntax element cclm_mode_idx specifies which of the LM candidate modes (for example, INTRA_LT_CCLM, INTRA_L_CCLM and INTRA_T_CCLM) is applied. For example, boundary matching costs are calculated for different LM candidate patterns. After re-ranking different LM candidate modes based on boundary matching costs, cclm_mode_idx can be used to select candidate LM candidate modes based on the re-ranking. In some embodiments, cclm_mode_idx equal to 0 refers to the LM candidate mode with the minimum cost, and cclm_mode_idx equal to 2 refers to the LM candidate mode with the maximum cost. In some embodiments, cclm_mode_idx is implicit and the LM candidate mode with the smallest cost is used.

In some embodiments, only the top k LM candidate modes (with higher priority due to cost) may be candidate LM modes (the original number of LM candidate modes is 3). In some embodiments, k=2 and LM candidate modes are sent as follows: cclm_mode_idx 0 refers to the candidate LM mode with the smallest cost using codeword 0; cclm_mode_idx 1 refers to the candidate LM mode with the second smallest cost using codeword 1.

In some embodiments, after reordering according to cost, when k=2, cclm_mode_idx equal to 0 indicates the LM candidate mode with the largest cost, and cclm_mode_idx equals 1 indicates the LM candidate mode with the smallest cost. In some embodiments, cclm_mode_idx is implicit and the LM candidate mode with the largest cost is used. In some of these embodiments, cclm_mode_idx 0 refers to the candidate LM mode with the largest cost using codeword 0; cclm_mode_idx 1 refers to the candidate LM mode with the second largest cost using codeword 1.

In addition to applying the reduced/reordered candidate mode list to the traditional CCLM mode where the chrominance component (Cr or Cb) is predicted from the luma component, the reduced/reordered candidate mode list can also be used in other variants of CCLM. used in. The CCLM variant here means that when the block indication refers to using one of the cross-component modes of the current block (e.g. CCLM_LT, MMLM_LT, CCLM_L, CCLM_T, MMLM_L, MMLM_T and/or intra prediction mode, not the traditional DC , planar and angular modes), some optional modes can be selected.

The following is an example of using convolutional cross-component mode (CCCM) as an optional mode. When this optional mode is applied to the current block, cross-component information using the model (including nonlinear terms) is used to generate chroma predictions. Optional modes may follow CCLM's template selection, so the CCCM family includes CCCM_LT CCCM_L and/or CCCM_T.

In some embodiments, the cost of predetermining candidates may be reduced. The predetermined candidates may refer to candidates inferred from one or more neighboring blocks and/or popular or common modes such as CCLM_LT, MMLM_LT, CCCM_LT. In some embodiments, candidate patterns include (only) relevant popular patterns and popular patterns. In one example, the relevant popular modes are CCCM_L and/or CCCM_T, and the popular mode is CCCM_LT. For example, the relevant popular modes are CCLM_L and/or CCLM_T, and the popular mode is CCLM_LT. In another example, the relevant popular modes are MMLM_L and/or MMLM_T, and the popular mode is MMLM_LT. In another example, the popular pattern indicates whether the reference used is from L, T, or LT. If the reference used is from LT, the relevant popular modes are all variants or a subset of variants of LT, such as CCLM_LT, MMLM_LT and CCCM_LT.

In some embodiments, the candidate pattern corresponds to a linear model derived by using adjacent reconstructed samples of Cb and Cr as inputs X and Y (i.e., using reconstructed Cb samples to derive or predict Cr samples ,vice versa). In some embodiments, the candidate pattern corresponds to a linear model derived from multiple co-located luma blocks for predicting or deriving chrominance components. In some embodiments, the candidate modes in the reduced/reordered candidate list may correspond to a multi-model linear model (MMLM) mode, so that different linear models can be selected from multiple linear models. To predict the chroma components of different regions or different pixel groups. Similar to CCLM, MMLM can have MMLM_LT, MMLM_L and/or MMLM_T.

The LM candidate patterns in the reduced/reordered candidate pattern list may also include any LM extension/variant, so that the number of candidate patterns in the list may also be increased. As the number of LM candidate patterns increases, the encoding performance improvement using boundary matching cost-based reduction/reordering of the candidate pattern list becomes more significant.

Ⅵ. MHP as a candidate model

For blocks to which multi-hypothesis prediction (MHP) is applied, one or more prediction hypotheses (prediction signals) are combined with existing prediction hypotheses to form the final (resultant) prediction of the current block. MHP can be applied to many inter-modes such as merging, sub-block merging, inter-AMVP and/or affine. The final prediction can be iteratively accumulated with each additional hypothesis of the prediction signal. An accumulation example looks like this:

The predicted signal is obtained as the last (Right now, has the largest index ). P ₀ is the first (existing) prediction for the current block. For example, if MHP is applied to a merge candidate, p ₀ is indicated by the existing merge index. The additional predictions are denoted h and will be further combined with the previously accumulated predictions by weight α. Therefore, for each additional hypothesis of prediction, a weighting index is sent/parsed to indicate weighting, and/or for each hypothesis of prediction, an inter-frame index is sent/parsed to indicate motion candidates (used to generate prediction sample hypotheses for this ).

The proposed reordering scheme is needed to make the sending of MHP weights and/or the sending of motion candidates more efficient. In some embodiments, a boundary matching cost for each MHP candidate weight is calculated for each additional prediction hypothesis. For example, assume that the prediction hypothesis has two candidate weights: Step 0: For the additional assumptions of prediction, cost_w0 and cost_w1 are calculated as the costs of the first and second candidate weights respectively; Step 1: For this prediction hypothesis, candidate weights are reranked based on cost.

In some sub-embodiments, candidate weights with smaller costs receive higher priority. If cost_w0 > cost_w1, weight index 0 refers to w1, and weight index 1 refers to w0. Otherwise, reordering is not used and weight indices 0 and 1 refer to the original w0 and w1. In some embodiments, candidate weights with larger costs receive higher priority. If cost_w0 < cost_w1, weight index 0 refers to w1, and weight index 1 refers to w0. Otherwise, reordering is not used and weight indices 0 and 1 refer to the original w0 and w1.

In some embodiments, the candidate weight with the smallest cost is used for the current additional prediction hypothesis. In this case the weight index of the current additional forecast hypothesis is inferred. In some embodiments, the weight of each hypothesis or any subset of hypotheses is implicitly set based on cost. One possibility is that the weight is a scaled value of the cost or the reciprocal of a scaled value of the cost. (For example, if cost = 2, then the inverse of cost = 1/2.) In some sub-embodiments, the candidate weight with the largest cost is used for the current additional prediction hypothesis. In this case the weight index of the current additional forecast hypothesis is inferred.

Steps 0 and 1 can be repeated for each additional forecast hypothesis and the meaning of each weight index obtained for each additional forecast hypothesis. In some embodiments, it is assumed that the prediction hypothesis has t candidate weights (t can be any positive integer. Take t = 5 in the following examples.) Step 0: For additional assumptions for prediction, calculate the cost of each candidate weight separately. Step 1: For this prediction hypothesis, candidate weights are reranked based on cost.

For example, candidate weights with smaller costs receive higher priority. As another example, candidate weights with larger costs receive higher priority. Only the top k candidate weights (those with high priority) can be used as candidate weights. The set of candidate patterns includes only the top k candidate weights. In some embodiments, the number of candidate weights in the candidate mode set is k (eg, 2) and each candidate weight is sent as follows: weight_idx 0 refers to the candidate with the highest priority and codeword 0; weight_idx 1 refers to the candidate with The candidate with the second highest priority and codeword 1. Steps 0 and 1 can be repeated for each additional forecast hypothesis and the meaning of each weight index obtained for each additional forecast hypothesis.

In some embodiments, MHP weights are divided into groups. Next, the boundary matching cost for each group is calculated. The promising group is then implicitly determined as the group with the highest priority. If more than one MHP weight is included in the promising group, a reduced MHP weight index is sent/parsed to indicate the MHP weight from the promising group. Since the number of MHP weights in each group is smaller than the number of MHP weights in the original MHP weight set, the reduced MHP weight index should occupy fewer codewords than the original MHP weight index.

In some embodiments, if k groups are used and the number of MHP weights in the original MHP weight set is N, then the number of MHP weights in each group is "N/k" and the grouping rules depend on the MHP weight index and/or MHP weight value. For example, MHP weights with the same "MHP weight index %k" value belong to the same group. For another example, MHP weights with the same "MHP weight index/k" belong to the same group. As another example, MHP weights with similar values are in the same group (e.g., MHP weights with negative values are in one group and/or MHP weights with positive values are in another group). Another example, k varies with block width, block height and/or block area varies. In some embodiments, when a group contains more than one merge candidate, the cost of the group is the average cost from the costs of all merge candidates in the group. In some embodiments, when a group contains more than one merge candidate, the cost of the group is the average, maximum, or minimum cost of the costs of all merge candidates in the group.

In some embodiments, the MHP weights are divided into several subsets, and the subset selection is explicitly sent to the decoder. and selection within subsets is performed using boundary matching costs. In some embodiments, weights with larger different values from each other will be grouped into the same subset. That is, weights with similar values will be divided into different subsets. This may help improve hit rate on boundary matches. For example, weights with negative values are divided into different groups. For another example, those weights whose weight differences are greater than a predetermined threshold are classified into the same group. The predetermined threshold may be fixed or explicitly determined.

In some embodiments, for each hypothesis, the current prediction of the candidate weights (used to calculate the boundary matching cost) is the combined prediction (the weighted average of the predictions from the motion candidate and the previously accumulated predictions). (Motion candidates are indicated by the sent/resolved motion index.) An example is shown in the image below. When applying the proposed method to reorder the sending of candidate weights for h2, h2 has 4 candidate weights (including candidates 0 to 3). The cost of candidate n is calculated as the weighted average of the previous cumulative predictions and h2 predictions (the weights come from candidate n).

When reordering is applied to the sending of h1's candidate weights, there are 4 candidate weights (including candidates 0 to 3): for h1, candidate 0's current prediction = the weighted average of p0's prediction and h1's prediction, with weights from candidate 0; Candidate 1's current prediction = weighted average of p0's prediction and h1's prediction, with weights from candidate 1.

In some embodiments, a boundary matching cost is calculated for each MHP motion candidate. The following sub-embodiments take MHP as the merging method as an example. (MHP can be applied to other inter-modes, such as inter-AMVP and/or affine, and when inter-AMVP or affine is used, "merge" in the following examples is replaced with the naming of that inter-mode.) In In some embodiments, a boundary matching cost is calculated for each motion candidate. For example, the candidate pattern includes candidates 0 to 4. Initially, index 0 (shorter codeword) refers to candidate 0, and index 4 (longer codeword) refers to candidate 4. Using the proposed approach, the meaning of the index follows the order of precedence. If the priority order (based on boundary matching cost) specifies that candidate 4 has the highest priority. Index 0 maps to candidate 4.

In some embodiments, a boundary matching cost is calculated for each motion candidate. For example, the candidate pattern includes candidates 0 to 4. Initially, index 0 (shorter codeword) refers to candidate 0, and index 4 (longer codeword) refers to candidate 4. Using the proposed approach, the meaning of the index follows the order of precedence. If the priority order (based on boundary matching cost) specifies that candidate 4 has the highest priority. This index is not sent/parsed, and the selected motion candidate is inferred to be the motion candidate with the highest priority.

In some embodiments, for each hypothesis, the current prediction of a motion candidate (used to calculate the boundary matching cost) is the motion compensated result produced by that motion candidate.

An example combining the above two sub-embodiments is shown below. For example, if the number of additional hypotheses equals 2, then the top three motion candidates with higher priority are used to form the outcome prediction of the current MHP block. For another example, if the number of additional hypotheses is equal to 2, then the existing hypotheses are retained, and the two motion candidates with higher priority are used to form predictions of the additional hypotheses, and the resulting predictions are formed from the existing hypotheses and the additional hypotheses.

In some embodiments, the current prediction of the motion candidate (used to calculate the boundary matching cost) is the combined prediction (the weighted average of the prediction from the motion candidate and the existing prediction (p0)). (The weight is indicated by the weight index sent/parsed.)

In some embodiments, for each hypothesis, the current prediction of the motion candidate (used to calculate the boundary matching cost) is the combined prediction (the weighted average of the prediction from the motion candidate and the previously accumulated predictions). (The weight is indicated by the weight index sent/parsed.) An example is shown below. When the proposed method is applied to reorder the sending of motion candidates for h2, h2 has 4 motion candidates (including candidates 0 to 3). The cost of candidate n is calculated as the weighted average of p0's prediction, h1's prediction, and candidate n's prediction. In other words, for h2: candidate 0's current prediction = the weighted average of the h1 prediction and the prediction from candidate 0; candidate 1's current prediction = the weighted average of the h1 prediction and the prediction from candidate 1.

When applying the proposed method to reorder the sending of motion candidates for h1, h1 has 4 motion candidates (including candidates 0 to 3): the current prediction of candidate 0 = the weighted average of the p0 prediction and the prediction from candidate 0; candidate 1 The current prediction of = the weighted average of the p0 prediction and the prediction from candidate 1.

In some embodiments, the boundary matching cost is calculated for each MHP combination (motion candidate and weight) of each prediction hypothesis. The following sub-embodiments take MHP as the merging method as an example. (MHP can be applied to other inter-modes, such as inter-AMVP and/or affine, and when inter-AMVP or affine is used, "merge" in the following examples is replaced with the naming of that inter-mode.)

In some embodiments, a combination refers to motion candidates and weights. If there are m motion candidates and each motion candidate has n measures, the number of combinations is m*n. In some sub-embodiments, the combined current prediction (used to calculate the boundary matching cost) is the combined prediction (the weighted average of the predictions from the motion candidate and the existing prediction (p0)). Using this approach, the merging index and the weights representing the additional hypotheses for the prediction are jointly determined. For example, the combination with the highest priority is the selected MHP combination. (No need to send/parse merge indexes and weights to indicate additional assumptions for predictions). As another example, the union index is sent/parsed to determine the MHP combination. This embodiment can fix the number of additional hypotheses.

In some embodiments, a boundary matching cost is calculated for each MHP motion candidate. Take MHP in merge mode as an example. The merge index (indicating each hypothesized motion candidate) can be inferred. For example, depending on the order of the merge candidates in the merge candidate list, assume 0 is merge candidate 0, assume 1 is merge candidate 1, and so on. As another example, according to cost, the merge candidate with smaller cost is used first. For another example, it depends on a predetermined number of merge candidates. If the number of hypotheses is 4, then 4 merging candidates in the merging candidate list are used as motion candidates for each hypothesis. The first 4 merge candidates are used, and any 4 merge candidates from the merge candidate list are used. The weights (used to combine forecast hypotheses) are implicit and depend on the cost. MHP outcome predictions are formed as follows. (weight0)*(hypothesis0) + (weight1)*(hypothesis1) + (weight2)*(hypothesis2)+…

In some embodiments, a fixed number of prediction hypotheses are used. That is, mixing a fixed number of hypotheses and implicitly using matching costs as weights. In some embodiments, a motion candidate (or hypothesis, so to speak) with a higher priority is weighted more than a motion candidate with a lower priority. In some embodiments, the current prediction of a motion candidate (used to calculate the boundary matching cost) is the motion compensated result generated by the motion candidate. In some embodiments, the top n motion candidates in the merged candidate list are used to generate prediction hypotheses. With this proposed approach, no merge index is sent to the MHP. In some embodiments, all motion candidates in the merged candidate list are used to generate prediction hypotheses. (The weight can determine whether a motion candidate is fully used. If its weight is zero, this motion candidate is not actually used.) With this proposed approach, no merge index is sent to the MHP. In some embodiments, motion candidates with higher priority are weighted more than motion candidates with lower priority. For example, the weights follow the cost ratio of different motion candidates. If there are two motion candidates and cost_cand0 = 2* cost_cand1, weight_cand0 = 2 * weight_cand1 or weight_cand0 = 1/2* weight_cand1.

As another example, the cost of each motion candidate is first normalized to the interval [MIN_VALUE, MAX_VALUE]. MAX_VALUE is a predetermined number, such as the number of forecast hypotheses. MIN_VALUE is predetermined, such as 0. For example, (MAX_VALUE - normalized cost) can be the weight of the motion candidate or the normalized cost can be the weight of the motion candidate.

As another example, a weight is a scaled value of a cost or a scaled value of the inverse of a cost. (For example, if cost = 2, then reciprocal of cost = ½.) The scale value represents the scaling factor * the original value. If scale factor = 1, scale.

In some embodiments, weights and merge indexes are implicit for the proposed method. Of course the predictions of this method can be generated with reference to any of the other methods proposed in the present invention. In some embodiments, the proposed scheme is applied to a subset of all additional prediction hypotheses. (i.e., repeat steps 0 and 1 above for a subset of all additional prediction hypotheses.) For example, only the candidate weights of the first additional prediction hypothesis (combined with the existing prediction hypothesis) are reranked using the proposed scheme. In some embodiments, this subset is specified in the video codec standard. In an embodiment, the subset depends on the current block width, height or area. For example, for blocks whose block area is larger (or smaller) than the threshold, the subset includes more prediction hypotheses.

In some embodiments, the reranked results from the subset can be reused for the remaining additional prediction hypotheses. For example, based on the reranking results of the first prediction hypothesis, weight indices 0 and 1 refer to w1 and w0 respectively. For the following forecast assumptions, weight indices 0 and 1 also refer to w1 and w0, respectively.

In some embodiments, the proposed scheme is applied to a subset of all candidate weights for additional prediction hypotheses. That is, repeat steps 0 and 1 above for a subset of all candidate weights for additional prediction hypotheses. As an example, take the number of candidate weights (for additional prediction hypotheses) equal to 4. For additional prediction hypotheses, only the top (or last) two candidate weights are reranked using the proposed scheme. In some embodiments, this subset is specified in the video codec standard. In some embodiments, the subset depends on the current block width, height, or area. For example, for blocks whose block area is larger (or smaller) than the threshold, the subset includes more candidate weights.

In some embodiments, the prediction hypothesis may be a prediction signal from a uni-prediction or bi-prediction motion compensation result. Proposed reordering schemes for different tools (not limited to those in the examples below) can be unified. For example, the proposed re-ranking schemes for MHP, LM, BCW, MMVD and/or merge candidates are unified with the same rules for calculating boundary matching costs.

The method proposed in this invention can be enabled and/or disabled according to implicit rules (e.g. block width, height or area) or according to explicit rules (e.g. based on blocks, tiles, fragments, pictures, SPS or PPS levels) . For example, when the block area is smaller than a threshold, the proposed reordering is applied. Any combination of the methods proposed in this invention can be applied.

Any of the previously proposed methods can be implemented in the encoder and/or decoder. For example, any of the proposed methods can be implemented in the intra/inter coding module of the encoder, the motion compensation module of the decoder, and the merge candidate derivation module. Alternatively, any of the proposed methods may be implemented as circuitry coupled to an intra/inter coding module of the encoder and/or a motion compensation module, merge candidate derivation module of the decoder.

Ⅶ , sample video encoder

Figure 6 illustrates an example video encoder 600 that may implement candidate codec mode selection based on boundary matching cost. As shown, video encoder 600 receives an input video signal from video source 605 and encodes the signal into a bit stream 695. Video encoder 600 has several components or modules for encoding signals from video source 605, including at least some components selected from the following: transform module 610, quantization module 611, inverse quantization module 614, inverse transform Module 615, intra estimation module 620, intra prediction module 625, motion compensation module 630, motion estimation module 635, loop filter 645, reconstructed picture buffer 650, MV buffer 665, MV prediction Module 675 and entropy encoder 690. Motion compensation module 630 and motion estimation module 635 are part of inter prediction module 640.

In some embodiments, modules 610-690 are modules of software instructions executed by one or more processing units (eg, processors) of a computing device or electronic device. In some embodiments, the modules 610-690 are hardware circuit modules implemented by one or more integrated circuits (ICs) of the electronic device. Although modules 610-690 are shown as individual modules, some modules may be combined into a single module.

Video source 605 provides an uncompressed raw video signal that represents the pixel data of each video frame. The subtractor 608 calculates the difference between the original video pixel data of the video source 605 and the predicted pixel data 613 from the motion compensation module 630 or the intra prediction module 625. Transform module 610 converts the difference values (or residual pixel data or residual signal 608) into transform coefficients (eg, by performing a discrete cosine transform, or DCT). The quantization module 611 quantizes the transform coefficients into quantized data (or quantized coefficients) 612, which is encoded into a bit stream 695 by the entropy encoder 690.

The inverse quantization module 614 performs inverse quantization on the quantized data (or quantized coefficients) 612 to obtain transform coefficients, and the inverse transform module 615 performs inverse transform on the transform coefficients to generate a reconstructed residual 619 . The reconstructed residual 619 is added to the predicted pixel data 613 to produce reconstructed pixel data 617 . In some embodiments, reconstructed pixel data 617 is temporarily stored in a line buffer (not shown) for intra-picture prediction and spatial MV prediction. The reconstructed pixels are filtered by loop filter 645 and stored in reconstructed picture buffer 650. In some embodiments, the reconstructed picture buffer 650 is a memory external to the video encoder 600 . In some embodiments, the reconstructed picture buffer 650 is an internal memory of the video encoder 600 .

The intra estimation module 620 performs intra prediction based on the reconstructed pixel data 617 to generate intra prediction data. The intra prediction data is provided to an entropy encoder 690 to be encoded into a bit stream 695. The intra prediction data is also used by the intra prediction module 625 to generate predicted pixel data 613 .

The motion estimation module 635 performs inter prediction by generating MVs to reference pixel data of previously decoded frames stored in the reconstructed picture buffer 650 . These MVs are provided to the motion compensation module 630 to generate predicted pixel data.

Instead of encoding the complete actual MV in the bitstream, the video encoder 600 uses MV prediction to generate the predicted MV, and the difference between the MV used for motion compensation and the predicted MV is encoded as residual motion data and stored In bitstream 695.

The MV prediction module 675 generates a predicted MV based on the reference MV generated for encoding the previous video frame, ie, the motion compensation MV used to perform motion compensation. The MV prediction module 675 obtains the reference MV from the previous video frame from the MV buffer 665 . The video encoder 600 stores the MV generated for the current video frame in the MV buffer 665 as a reference MV for generating a predicted MV.

The MV prediction module 675 uses the reference MV to create predicted MVs. The predicted MV can be calculated by spatial MV prediction or temporal MV prediction. Entropy encoder 690 encodes the difference between the predicted MV and the motion compensated MV (MC MV) of the current frame (residual motion data) into bit stream 695.

The entropy encoder 690 encodes various parameters and data into the bit stream 695 by using an entropy coding technique such as context-adaptive binary arithmetic coding (CABAC) or Huffman coding. Entropy encoder 690 encodes various header elements, flags along with quantized transform coefficients 612 and residual motion data as syntax elements into bit stream 695. The bit stream 695 is stored in a storage device or transmitted to the decoder via a communication medium (such as a network).

The in-loop filter 645 performs a filtering or smoothing operation on the reconstructed pixel data 617 to reduce coding and decoding artifacts, especially at the boundaries of pixel blocks. In some embodiments, the filtering operation performed includes sample adaptive offset (SAO). In some embodiments, the filtering operation includes an adaptive loop filter (ALF).

Figure 7 shows a portion of a video encoder 600 that implements candidate codec mode selection based on boundary matching costs. Specifically, this figure illustrates the components of the inter prediction module 640 of the video encoder 600, which calculates the boundary matching cost for all or any subset of different candidate codec modes, and selects candidates based on the group assignment and calculated cost. Codec mode.

Inter prediction module 640 includes various boundary prediction modules 710 . These boundary prediction modules 710 generate prediction samples 715 of the current block along the boundaries of the current block for various candidate codec modes. The boundary matching cost calculator 730 compares the boundary prediction subsamples 715 of each candidate codec mode with the neighboring samples 725 of the current block (obtained from the reconstructed picture buffer 650) to calculate each candidate codec mode or Boundary matching cost for any subset of candidate codec modes. Part I above describes the calculation of boundary matching costs based on adjacent samples and boundary prediction samples.

The group allocation and selection module 740 allocates different candidate codec modes into different groups and selects one of the groups. Candidate selection module 750 then selects a candidate codec mode from the selected group. Inter prediction module 640 then performs motion compensation using the selected candidate codec mode. In some embodiments, the group allocation module 740 allocates a number of candidate codec modes with the lowest boundary matching costs to form a lowest cost group, and the candidate selection module 750 selects candidate codec modes from the lowest cost group. In some embodiments, the group allocation module 740 allocates a number of candidate codec modes to form groups with predetermined rules (which may or may not depend on cost), and the candidate selection module 750 selects candidate codecs from the group based on cost. model. For example, the selected candidate encoding and decoding mode is the candidate mode with the highest priority in the group.

In some embodiments, the identification of the selected group is sent to entropy encoder 690 for sending in bitstream 695. In some embodiments, the identification of the selected group is determined implicitly and is not sent in the bitstream. In some embodiments, group selection is determined based on calculated boundary matching costs for different groups, for example, group selection module 740 may select the group with the lowest representation cost.

In some embodiments, identifications of selected candidate codec modes within the selected group are provided to entropy encoder 690 for transmission in bitstream 695 . In some embodiments, the candidate codec modes within the group are reordered based on the calculated boundary matching costs such that the lowest cost (or highest cost) candidate will be sent using the shortest codeword. In some embodiments, the identity of the selected candidate codec mode will be determined implicitly and not sent in the bitstream, for example, by selecting the candidate codec mode with the lowest boundary matching cost within the group. The selection of a set of candidate codec modes and the selection of a candidate codec mode from the set are described in Sections II-VI above.

Boundary prediction modules 710 include modules for various different candidate codec modes. These candidate codec modes may generate boundary prediction samples using luma and chroma samples stored in reconstruction buffer 650 , motion information from MV buffer 665 , and/or incoming luma and chroma samples from video source 605 715 (e.g., by deriving merge candidates, deriving MMVD candidates, using motion information to obtain reference samples, generating linear models, etc.) In some embodiments, the various candidate codec modes include those corresponding to various merge candidates of the current block. Decoding mode. Various types of merging candidates (e.g., spatial, temporal, affine, CIIP, etc.) that are candidate codec modes are described in Section II above. Different MMVD distances and directions as candidate codec modes are described in Section III above. Different BCW weights as candidate codec modes are described in Section IV above. Various types of linear models (e.g., LM-L, LM-T, LM-LT) that are candidate encoding and decoding modes are described in Section V above.

Figure 8 conceptually illustrates a process 800 for selecting candidate codec modes to encode a current block using boundary matching costs. In some embodiments, one or more processing units (eg, processors) of a computing device implementing encoder 600 perform process 800 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing encoder 600 performs process 800.

The encoder receives (at block 810) the data for the block of pixels that will be encoded as the current block in the current picture. The encoder identifies (at block 820) multiple candidate codec modes suitable for the current block. In some embodiments, the plurality of candidate codec modes include merge candidates for the current block. Merging candidates for the current block may include (i) merging candidates using CIIP and/or (ii) merging candidates using affine transform motion compensated prediction. In some embodiments, the plurality of candidate codec modes include candidate codec modes corresponding to different MMVD combinations of distance and offset used to refine the motion information. In some embodiments, the plurality of candidate codec modes includes candidate codec modes that correspond to different linear models for deriving predictors of chroma samples of the current block based on luma samples of the current block. In some embodiments, the plurality of candidate codec modes include candidate codec modes corresponding to different candidate BCW weights for combining inter prediction in different directions.

The encoder identifies (at block 830 ) a first set of candidate codec modes that is a subset of the plurality of candidate codec modes such that there are fewer codec modes in the first set of candidate codec modes than in the plurality of candidate codec modes. The number of candidate codec modes.

In some embodiments, the first set of candidate codec modes is based on the highest priority candidate codec mode identified by cost of the plurality of candidate codec modes. The encoder may index the candidate codec modes according to the priority order of the candidate codec modes or assign codewords to the candidate codec modes in the identified candidate codec mode group. In some embodiments, the cost of the candidate codec mode is calculated by comparing (i) reconstructed samples adjacent to the current block with (ii) predicted samples of the current block along the current block boundary generated according to the candidate codec mode. boundary matching cost. Section I above describes the calculation of boundary matching costs based on adjacent samples and boundary prediction samples.

In some embodiments, each of the plurality of candidate codec modes is assigned to one of the plurality of candidate codec mode groups. For example, in some embodiments, each candidate codec mode in the plurality of candidate codec modes is associated with an original candidate index, wherein each candidate codec mode is based on the result of the original index modulo K or the original index divided by K The result is assigned to one of the K sets of candidate encoding and decoding modes. In some embodiments, candidate codec modes corresponding to spatial merging candidates are assigned to the same candidate codec mode group. In some embodiments, candidate codec modes corresponding to spatial merging candidates are assigned to different candidate codec mode groups. In some embodiments, candidate codec modes having merge candidates with motion differences less than a threshold are assigned to the same group. In some embodiments, candidate codec modes having merge candidates with motion differences greater than a threshold are assigned to the same group. In some embodiments, the encoder identifies a candidate codec mode group with the lowest representation cost among the plurality of candidate codec mode groups, and sends an index used to select a candidate from the identified candidate codec mode group Codec mode. The representative cost of the identified group may be the average, maximum, or minimum value of the costs (eg, boundary matching costs) of candidate codec modes for the identified group. In some embodiments, the encoder sends an index for selecting a candidate codec mode set and a cost based on the selected candidate codec mode set (eg, a boundary matching cost) from the selected candidate codec mode set. Identify candidate codec modes. The selection of a set of candidate codec modes and the selection of a candidate codec mode from the set are described in Sections II-VI above.

The encoder selects (at block 840) a candidate codec mode in the first set of candidate codec modes. The candidate codec mode selected in the first candidate codec mode group may be selected based on cost.

The encoder encodes the current block using the selected candidate codec mode (at block 850). Specifically, the encoder constructs a predictor of the current block according to the selected candidate encoding and decoding mode, and encodes the current block using the predictor.

Ⅷ . Sample video decoder

In some embodiments, the encoder may send (or generate) one or more syntax elements in the bitstream such that the decoder may parse the one or more syntax elements from the bitstream.

Figure 9 illustrates an example video decoder 900 that may implement boundary matching cost-based candidate decoding mode selection. As shown in the figure, the video decoder 900 is an image decoding or video decoding circuit that receives a bit stream 995 and decodes the contents of the bit stream into pixel data of a video frame for display. Video decoder 900 has several components or modules for decoding bit stream 995, including some components selected from the following: inverse quantization module 911, inverse transform module 910, intra prediction module 925, motion compensation module 930, in-loop filter 945, decoded picture buffer 950, MV buffer 965, MV prediction module 975 and parser 990. Motion compensation module 930 is part of inter prediction module 940 .

In some embodiments, modules 910-990 are modules of software instructions executed by one or more processing units (eg, processors) of a computing device. In some embodiments, modules 910-990 are hardware circuit modules implemented by one or more ICs of an electronic device. Although modules 910-990 are shown as individual modules, some modules may be combined into a single module.

A parser 990 (or entropy decoder) receives the bitstream 995 and performs initial parsing according to the syntax defined by the video codec or image codec standard. Parsed syntax elements include various header elements, flags, and quantization data (or quantization coefficients) 912 . The parser 990 uses, for example, context-adaptive binary arithmetic coding (CABAC) or Huffman coding.

The inverse quantization module 911 performs inverse quantization on the quantized data (or quantized coefficients) 912 to obtain the transform coefficients, and the inverse transform module 910 performs inverse transformation on the transform coefficients 916 to obtain the reconstructed residual signal 919 . The reconstructed residual signal 919 is added to the predicted pixel data 913 from the intra prediction module 925 or the motion compensation module 930 to together produce decoded pixel data 917. The decoded pixel data is filtered by the in-loop filter 945 and stored in the decoded picture buffer 950. In some embodiments, the decoded picture buffer 950 is a memory external to the video decoder 900 . In some embodiments, the decoded picture buffer 950 is a memory internal to the video decoder 900 .

The intra prediction module 925 receives intra prediction data from the bit stream 995 and generates predicted pixel data 913 from the decoded pixel data 917 stored in the decoded picture buffer 950 accordingly. In some embodiments, decoded pixel data 917 is also stored in a line buffer (not shown) for intra prediction and spatial MV prediction.

In some embodiments, the contents of picture buffer 950 are decoded for display. The display device 955 obtains the contents of the decoded picture buffer 950 for direct display, or obtains the contents of the decoded picture buffer to a display buffer. In some embodiments, the display device receives pixel values from decoded picture buffer 950 via pixel transfer.

The motion compensation module 930 generates predicted pixel data 913 from the decoded pixel data 917 stored in the decoded picture buffer 950 according to the motion compensated MV (MC MV). These motion compensated MVs are decoded by adding the residual motion data received from bit stream 995 to the predicted MV received from MV prediction module 975 .

The MV prediction module 975 generates prediction MVs based on reference MVs generated for decoding previous video frames, eg, motion compensation MVs for performing motion compensation. The MV prediction module 975 obtains the reference MV of the previous video frame from the MV buffer 965 . The video decoder 900 stores the motion compensated MV generated for decoding the current video frame in the MV buffer 965 as a reference MV for generating the predicted MV.

In-loop filter 945 performs a filtering or smoothing operation on decoded pixel data 917 to reduce encoding and decoding artifacts, particularly at pixel block boundaries. In some embodiments, the filtering operation performed includes sample adaptive offset (SAO). In some embodiments, the filtering operation includes an adaptive loop filter (ALF).

Figure 10 shows a portion of a video decoder 900 that implements candidate codec mode selection based on boundary matching costs. Specifically, this figure shows the components of the inter prediction module 940 of the video decoder 900, which calculates the boundary matching cost for all or any subset of different candidate codec modes, and selects based on the group allocation and calculated cost. Candidate codec mode.

Inter prediction module 940 includes various boundary prediction modules 1010 . These boundary prediction modules 1010 generate prediction samples 1015 of the current block along the boundaries of the current block for various candidate codec modes. The boundary matching cost calculator 1030 compares the boundary predictor samples 1015 of each candidate codec mode with the neighboring samples 1025 of the current block (obtained from the reconstructed picture buffer 950) to calculate each candidate codec mode or Boundary matching cost for any subset of candidate codec modes. Section I above describes the calculation of boundary matching costs based on adjacent samples and boundary prediction samples.

The group allocation and selection module 1040 allocates different candidate codec modes into different groups and selects one of the groups. Then, the candidate selection module 1050 selects a candidate codec mode from the selected group. The inter prediction module 940 then performs motion compensation using the selected candidate codec mode. In some embodiments, the group allocation module 1040 allocates a number of candidate codec modes with the lowest boundary matching costs to form a lowest cost group, and the candidate selection module 1050 selects candidate codec modes from the lowest cost group. In some embodiments, the group allocation module 1040 allocates a number of candidate codec modes to form groups with predetermined rules (which may or may not depend on cost), and the candidate selection module 1050 selects candidates from the group based on cost. Codec mode. For example, the selected candidate encoding and decoding mode is the candidate mode with the highest priority in the group.

In some embodiments, entropy decoder 990 parses bitstream 995 to obtain an identification of the selected group and provides the identification of the selected group to group allocation and selection module 1040 . In some embodiments, the identity of the selected group is determined implicitly and is not sent in the bitstream. In some embodiments, group selection is determined based on calculated boundary matching costs for different groups, for example, the group selection module 1040 may select the group with the lowest representation cost. In some embodiments, entropy decoder 990 parses bitstream 995 to obtain an identification of the selected candidate codec mode within the selected group and provides the identification of the selected candidate codec mode to candidate selection module 1050 .

In some embodiments, candidate codec modes within a group are reordered according to calculated boundary matching costs such that the lowest cost (or highest cost) candidate will be sent using the shortest codeword. In some embodiments, the identity of the selected candidate codec mode will be determined implicitly and not sent in the bitstream, for example, by selecting the candidate codec mode with the lowest boundary matching cost within the group. The selection of a set of candidate codec modes and the selection of a candidate codec mode from the set are described in Sections II-VI above.

Boundary prediction modules 1010 include modules for various different candidate codec modes. These candidate codec modes may use the luma and chroma samples stored in the reconstruction buffer 950 and the motion information from the MV buffer 965 to generate boundary prediction samples 1015 (e.g., by deriving merge candidates, deriving MMVD candidates, using Motion information acquisition reference samples, generation of linear models, etc.) In some embodiments, the candidate codec modes include those candidate codec modes corresponding to various merging candidates of the current block. Various types of merging candidates (e.g., spatial, temporal, affine, CIIP, etc.) that are candidate codec modes are described in Section II above. Different MMVD distances and directions as candidate codec modes are described in Section III above. The different BCW weights used as candidate codec modes are described in Section IV above. Various types of linear models (e.g., LM-L, LM-T, LM-LT) that are candidate encoding and decoding modes are described in Section V above.

Figure 11 conceptually illustrates a process 1100 of selecting candidate codec modes to decode a current block using boundary matching costs. In some embodiments, one or more processing units (eg, processors) of a computing device implementing decoder 900 perform process 1100 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing decoder 900 performs process 1100 .

The decoder receives (at block 1110) the data for the block of pixels that will be decoded into the current block in the current picture. The decoder identifies (at block 1120) a plurality of candidate codec modes applicable to the current block. In some embodiments, the plurality of candidate codec modes include merge candidates for the current block. Merging candidates for the current block may include (i) merging candidates using combined inter and intra prediction (CIIP) and/or (ii) merging candidates using affine transform motion compensated prediction. In some embodiments, the plurality of candidate codec modes include candidate codec modes corresponding to different MMVD combinations of distance and offset used to refine the motion information. In some embodiments, the plurality of candidate encoding and decoding modes include candidate encoding and decoding modes corresponding to different linear models (LMs) for deriving predictors of chroma samples of the current block based on luma samples of the current block. In some embodiments, the plurality of candidate codec modes include candidate codec modes corresponding to different candidate BCW weights for combining inter prediction in different directions.

The decoder identifies (at block 1130) a first set of candidate codec modes that is a subset of the plurality of candidate codec modes. The number of candidate coding and decoding modes in the first candidate coding and decoding mode group is less than the number of candidate coding and decoding modes in the plurality of candidate coding and decoding modes. (That is, the first candidate codec mode group is a subset of the plurality of candidate codec modes.)

In some embodiments, the first set of candidate codec modes is based on the highest priority candidate codec mode identified by cost of the plurality of candidate codec modes. The decoder may index the candidate codec modes in the identified candidate codec mode group or assign codewords to the candidate codec modes according to the priority order of the candidate codec modes. In some embodiments, the cost of the candidate codec mode is calculated by comparing (i) reconstructed samples adjacent to the current block with (ii) predicted samples of the current block along the current block boundary generated according to the candidate codec mode. boundary matching cost. Section I above describes the calculation of boundary matching costs based on adjacent samples and boundary prediction samples.

In some embodiments, each of the plurality of candidate codec modes is assigned to one of the plurality of candidate codec mode groups. For example, in some embodiments, each candidate codec mode in the plurality of candidate codec modes is associated with an original candidate index, wherein each candidate codec mode is based on the result of the original index modulo K or the original index divided by K The result is assigned to one of the K sets of candidate encoding and decoding modes. In some embodiments, candidate codec modes corresponding to spatial merging candidates are assigned to the same candidate codec mode group. In some embodiments, candidate codec modes corresponding to spatial merging candidates are assigned to different candidate codec mode groups. In some embodiments, candidate codec modes having merge candidates with motion differences less than a threshold are assigned to the same group. In some embodiments, candidate codec modes having merge candidates with motion differences greater than a threshold are assigned to the same group. In some embodiments, the decoder identifies a candidate codec mode group with the lowest representation cost among the plurality of candidate codec mode groups, and sends an index used to select a candidate from the identified candidate codec mode group Codec mode. The representative cost of the identified group may be the average, maximum, or minimum value of the costs (eg, boundary matching costs) of candidate codec modes for the identified group. In some embodiments, the decoder sends an index for selecting a candidate codec mode group, and selects from the selected candidate codec mode group based on a cost (eg, a boundary matching cost) of the selected candidate codec mode group. Identify candidate codec modes. The selection of a set of candidate codec modes and the selection of a candidate codec mode from the set are described in Sections II-VI above.

The decoder selects (at block 1140) a candidate codec mode in the first set of candidate codec modes. The candidate codec mode selected in the first candidate codec mode group may be selected based on cost.

The decoder decodes (at block 1150) the current block using the selected candidate codec mode to reconstruct the current block. Specifically, the decoder constructs a predictor of the current block according to the selected candidate encoding and decoding mode, and uses the predictor to reconstruct the current block. The decoder may then provide the reconstructed current block for display as part of the reconstructed current picture.

Ⅷ . Example electronic system

Many of the above features and applications are implemented as software processes specified as a set of instructions recorded on a computer-readable storage medium (also referred to as a computer-readable medium). When these instructions are executed by one or more computing or processing units (e.g., one or more processors, processor cores, or other processing units), they cause the processing unit to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, compact disc read-only memory (CD-ROM), flash memory drives, random-access memory (RAM) chips, Hard drive, erasable programmble read-only memory (EPROM), electrically erasable proagrammble read-only memory (EEPROM), etc. . Computer-readable media does not include carrier waves and electronic signals transmitted over wireless or wired connections.

In this specification, the term "software" is intended to include firmware that resides in read-only memory or applications stored in magnetic memory that can be read into memory for processing by a processor. Furthermore, in some embodiments, multiple software inventions may be implemented as sub-portions of a larger program while retaining distinct software inventions. In some embodiments, multiple software inventions may also be implemented as separate programs. Finally, any combination of individual programs that together implement the software inventions described herein is within the scope of this disclosure. In some embodiments, a software program, when installed to run on one or more electronic systems, defines one or more specific machine implementations that process and perform the operations of the software program.

Figure 12 conceptually illustrates an electronic system 1200 implementing some embodiments of the present disclosure. Electronic system 1200 may be a computer (eg, desktop computer, personal computer, tablet computer, etc.), telephone, PDA, or any other type of electronic device. Such electronic systems include various types of computer-readable media and interfaces for various other types of computer-readable media. Electronic system 1200 includes bus 1205, processing unit 1210, graphics-processing unit (GPU) 1215, system memory 1220, network 1225, read-only memory 1230, permanent storage device 1235, input device 1240, and output device 1245.

Bus 1205 collectively represents all of the numerous internal devices, peripherals, and chipset buses that are communicatively connected to electronic system 1200 . For example, bus 1205 communicatively connects processing unit 1210 to GPU 1215, read-only memory 1230, system memory 1220, and persistent storage 1235.

The processing unit 1210 obtains instructions to be executed and data to be processed from these various memory units in order to perform the processes of the present disclosure. In different embodiments, the processing unit may be a single processor or a multi-core processor. Some instructions are passed to and executed by the GPU 1215. GPU 1215 may offload various computations or supplement the image processing provided by processing unit 1210.

Read-only memory (ROM) 1230 stores static data and instructions used by the processing unit 1210 and other modules of the electronic system. On the other hand, the permanent storage device 1235 is a read-write storage device. This device is a non-volatile storage unit that stores instructions and data even when the electronic system 1200 is turned off. Some embodiments of the present disclosure use large-capacity memory devices (such as magnetic disks or optical disks and their corresponding disk drives) as the permanent storage device 1235 .

Other embodiments use off-mount storage devices (such as floppy disks, flash memory devices, etc., and their corresponding disk drives) as permanent storage devices. Like persistent storage device 1235, system memory 1220 is a read-write memory device. However, unlike the permanent storage device 1235, the system memory 1220 is a volatile read-write memory, such as a random access memory. System memory 1220 stores some instructions and data used by the processor during operation. In some embodiments, processes in accordance with the present disclosure are stored in system memory 1220, persistent storage device 1235, and/or read-only memory 1230. For example, according to some embodiments of the present disclosure, various memory units include instructions for processing multimedia clips. From these various memory units, the processing unit 1210 obtains instructions to be executed and data to be processed in order to perform the processing of some embodiments.

Bus 1205 also connects to input device 1240 and output device 1245 . Input device 1240 enables the user to communicate information and select commands to the electronic system. Input devices 1240 include alphanumeric keyboards and pointing devices (also known as "cursor control devices"), cameras (eg, webcams), microphones or similar devices for receiving voice commands, and the like. Output device 1245 displays images or output material generated by the electronic system. Output devices 1245 include printers and display devices, such as cathode ray tubes (CRT) or liquid crystal displays (LCD), as well as speakers or similar audio output devices. Some embodiments include devices used as input and output devices, such as touch screens.

Finally, as shown in Figure 12, bus 1205 also couples electronic system 1200 to network 1225 via a network interface card (not shown). In this manner, the computer may be part of a computer network, such as a local area network ("LAN"), a wide area network ("WAN"), or an intranet, or a network of multiple networks, such as the Internet. Electronic system 1200 Any or all components may be used in conjunction with the present disclosure.

Some embodiments include electronic components, such as microprocessors, storage devices, and memories that store computer program instructions on a machine-readable or computer-readable medium (alternatively referred to as a computer-readable storage medium, machine-readable medium, or machine-readable medium). readable storage media). Some examples of such computer-readable media include RAM, ROM, read-only compact discs (CD-ROM), recordable compact discs (CD-R), rewritable discs compact discs (CD-RW for short), read-only digital versatile discs (e.g., DVD-ROM, double-layer DVD-ROM), various recordable/rewritable DVDs (e.g., DVD -RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini SD card, micro SD card, etc.), magnetic and/or solid-state hard drive, read-only and recordable Blu-Ray ® optical disc, ultra-density optical disc, any other optical or magnetic media, and floppy disks. The computer-readable medium may store a computer program that is executable by at least one processing unit and includes a set of instructions for performing various operations. Examples of computer programs or computer code include machine code such as that produced by a compiler, as well as documents that include high-level code executed by a computer, electronic component, or microprocessor using an interpreter.

While the above discussion primarily relates to microprocessors or multi-core processors executing software, many of the above features and applications are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable Design a field programmable gate array (FPGA for short). In some embodiments, such integrated circuits execute instructions stored on the circuit itself. Additionally, some embodiments execute software stored in a programmable logic device (PLD), ROM, or RAM device.

As used in this specification and any claim in this application, the terms "computer", "server", "processor" and "memory" refer to electronic or other technical equipment. These terms do not include persons or groups of people. For the purposes of this specification, the term display or display refers to display on an electronic device. As used in this specification and any claims claimed in this application, the terms "computer-readable medium," "computer-readable medium" and "machine-readable medium" are exclusively limited to tangible physical media that stores information in a computer-readable form. object. These terms do not include any wireless signals, wired download signals and any other short-lived signals.

Although the present disclosure has been described with reference to numerous specific details, those skilled in the art will recognize that the disclosure may be embodied in other specific forms without departing from the spirit of the disclosure. Additionally, several figures, including Figures 8 and 11, conceptually illustrate the process. The specific operations of these processes may not be performed in the exact sequence shown and described. Specific operations may not be performed in a continuous series of operations, and different specific operations may be performed in different embodiments. Furthermore, this process can be implemented using multiple sub-processes or as part of a larger macro process. Accordingly, one of ordinary skill in the art will understand that the present disclosure is not limited by the foregoing illustrative details, but rather by the scope of the appended claims. Additional information

The subject matter described herein sometimes represents different components that are contained within or connected to other different components. It will be understood that the structures described are examples only and may in fact be implemented by many other structures to achieve the same functionality, and conceptually any arrangement of components achieving the same functionality is in fact "related" , in order to achieve the required functions. Therefore, any two components, regardless of structure or intermediate components, that are combined to achieve a specific function are considered to be "interrelated" to achieve the required function. Likewise, any two associated components are considered to be "operably connected" or "operably coupled" to each other to achieve the specified functionality. Any two components that can be associated with each other are also said to be "operably coupled" with each other to achieve the specified functionality. Any two components that can be associated with each other are also said to be "operably coupled" with each other to achieve the specified functionality. Specific examples of operably connected components include, but are not limited to, physically pairable and/or physically interacting components, and/or wirelessly interactable and/or wirelessly interacting components, and/or logically interacting and/or logically interacting components. Interactive components.

Furthermore, with regard to the use of substantially any plural and/or singular term, one of ordinary skill in the art may convert the plural to the singular and/or from the singular to the plural depending on the context and/or application. For the sake of clarity, this disclosure expressly sets out different singular/plural arrangements.

In addition, those of ordinary skill in the art will understand that generally, terms used in the present invention, especially within the scope of the application, such as the subject matter of the scope of the application, are generally used as "open" terms, for example, "including" should be interpreted as "Including but not limited to", "have" should be understood as "at least have", "include" should be interpreted as "including but not limited to", etc. One of ordinary skill in the art will further understand that if a specific amount of claimed content is intended to be introduced, this will be explicitly stated within the claimed scope and, in the absence of such content, it will not be shown. For example, to aid understanding, the following patent claims may contain the phrases "at least one" and "one or a plurality" to introduce the content of the patent claims. However, the use of these phrases should not be construed as implying that the indefinite article "a" or "an" is used to introduce the scope of the claim and thereby limit the scope of any particular claim. Even when the same claim includes the introductory phrase "one or plural" or "at least one", the indefinite article, such as "a" or "an", shall be construed to mean at least one or more, for The same holds true for the use of an explicit description to introduce the scope of a patent claim. Furthermore, even if an introductory reference to a particular number is expressly cited, one of ordinary skill in the art would recognize that such reference should be construed to mean the number cited, e.g., "two citations" without other modifications, means at least Two citations, or two or more citations. In addition, when an expression similar to "at least one of A, B and C" is used, it is usually expressed so that a person of ordinary skill in the art can understand the expression, for example, "the system includes A, B and C" "At least one of" will include, but is not limited to, a system with A alone, a system with B alone, a system with C alone, a system with A and B, a system with A and C, a system with B and C, and/ Or a system with A, B and C etc. It will be further understood by those of ordinary skill in the art that any separated words and/or phrases represented by two or more alternative terms, whether in the specification, patent claims or drawings, should be understood as , including the possibility of one, one, or both of these terms. For example, "A or B" should be understood as the possibility of "A", or "B", or "A and B".

It will be understood from the foregoing that various embodiments of the present invention have been described for purposes of illustration, and that various modifications may be made without departing from the scope and spirit of the invention. Accordingly, the various embodiments disclosed herein are not to be considered limiting, and the true scope and applicability are indicated by the claims.

510: Merge candidate 522:MMVD candidate 524:MMVD candidate 600:Video encoder 605:Video source 608:Subtractor 610:Transformation module 611:Quantization module 612:Quantized transform coefficient 613:Pixel data 614:Inverse quantization module 615:Inverse transformation module 616: Transformation coefficient 617:Reconstructed pixel data 619:Reconstruction residuals 620: Intra-frame estimation module 625: Intra prediction module 630: Motion compensation module 635: Motion estimation module 640: Inter-frame prediction module 645: Loop filter 650: Reconstruct image buffer 665:MV buffer 675:MV prediction module 690:Entropy encoder 695:Bit stream 710: Boundary prediction module 715: Boundary prediction sample 725: Adjacent samples 730: Boundary matching cost calculator 740:Group selection module 750: Candidate selection module 800: Processing 810: Steps 820: Steps 830: Steps 840: Steps 850: Steps 900:Video decoder 910:Inverse transformation module 911:Inverse quantization module 912:Quantitative data 913: Predict pixel data 916: Transformation coefficient 917: Decode pixel data 919:Reconstruct the residual signal 925: Intra prediction module 930: Motion compensation module 940: Inter prediction module 945: Loop filter 950: Decode picture buffer 955:Display device 965:MV buffer 975:MV prediction module 990:Parser 895:Bit stream 1010: Boundary prediction module 1015: Boundary prediction sample 1020:Adjacent acquisition 1025: Adjacent samples 1030: Boundary matching cost calculator 1040:Group selection module 1050: Candidate selection module 1100: Processing 1110: Steps 1120: Steps 1130: Steps 1140: Steps 1150: Steps 1200: Electronic systems 1205:Bus 1210: Processing unit 1215:GPU 1220:System memory 1225:Internet 1230: Read-only memory 1235:Permanent storage of equipment 1240:Input device

The accompanying drawings are included to provide a further understanding of the disclosure, and are incorporated in and constitute a part of this disclosure. The drawings illustrate embodiments of the disclosure and, together with the description, serve to explain the principles of the disclosure. Notably, the drawings are not necessarily to scale, as some components may be shown to be disproportionate to the size of an actual implementation in order to clearly illustrate the concepts of the present disclosure. Figure 1 shows reconstructed neighbor samples and predicted samples of the current block for boundary matching. Figure 2 shows the locations of spatial merging candidates. Figure 3 shows motion vector scaling of temporal merging candidates. Figure 4 shows the candidate positions of the temporal merging candidates of the current block. Figure 5 conceptually illustrates Merge Mode with Motion Vector Difference (MMVD) candidates and their corresponding offsets. Figure 6 illustrates an example video encoder that can implement candidate codec mode selection based on boundary matching cost. Figure 7 shows a portion of a video encoder that implements candidate codec mode selection based on boundary matching costs. Figure 8 conceptually illustrates the process of using boundary matching costs to select candidate codec modes to encode the current block. Figure 9 illustrates an example video decoder that can implement candidate codec mode selection based on boundary matching cost. Figure 10 shows a portion of a video decoder that implements boundary matching cost based candidate codec mode selection. Figure 11 conceptually illustrates the process of using boundary matching costs to select candidate codec modes to decode the current block. Figure 12 conceptually illustrates an electronic system implementing some embodiments of the present disclosure.

1100: Processing

1110: Steps

1120: Steps

1130: Steps

1140: Steps

1150: Steps

Claims (21)

A video encoding and decoding method, including: Receive data for a block of pixels that will be encoded or decoded into a current block of a current picture of a video; Identify multiple candidate codec modes applicable to the current block; Identify a first candidate coding and decoding mode group, the first candidate coding and decoding mode group is a subset of the candidate coding and decoding modes, wherein the number of candidate coding and decoding modes in the first candidate coding and decoding mode group is less than the number of candidate coding and decoding modes The number of candidate coding and decoding modes of the candidate coding and decoding modes; Select a candidate codec mode from the first candidate codec mode group; and The current block is encoded or decoded using the selected candidate encoding and decoding mode. The video coding and decoding method of claim 1, wherein the candidate coding and decoding mode selected in the first candidate coding and decoding mode is selected based on cost. The video encoding and decoding method of claim 1, wherein the first candidate encoding and decoding mode group is the highest priority candidate encoding and decoding mode identified based on cost. The video encoding and decoding method as described in claim 3 further includes indexing a plurality of candidate encoding and decoding modes in the identified candidate encoding and decoding mode group according to the priority of the candidate encoding and decoding modes. The video encoding and decoding method as described in claim 3 further includes allocating a plurality of codewords to the plurality of candidate encoding and decoding modes in the identified candidate encoding and decoding mode group according to the priority order of the candidate encoding and decoding modes. The video coding and decoding method of claim 3, wherein the cost of a candidate coding mode is calculated by comparing (i) a plurality of reconstructed samples adjacent to the current block and (ii) the current block along the current block. A plurality of prediction samples at block boundaries are calculated, wherein the prediction samples are generated according to the candidate encoding and decoding mode. The video encoding and decoding method of claim 1, wherein each of the candidate encoding and decoding modes is assigned to a candidate encoding and decoding mode group among a plurality of candidate encoding and decoding mode groups. The video encoding and decoding method of claim 7, wherein each of the candidate encoding and decoding modes is associated with an original candidate index, wherein each candidate encoding and decoding mode is based on an original index modulo K. The result or the result of dividing an original index by K is assigned to a candidate codec mode group among the K candidate codec mode groups. The video encoding and decoding method as described in claim 7 of the application, wherein the candidate encoding and decoding modes corresponding to the plurality of spatial merging candidates are assigned to the same candidate encoding and decoding mode group. The video encoding and decoding method of claim 7, wherein the candidate encoding and decoding modes corresponding to the plurality of spatial merging candidates are assigned to different candidate encoding and decoding mode groups. The video encoding and decoding method of claim 7, wherein the candidate encoding and decoding modes of multiple merging candidates whose motion differences are less than a threshold are assigned to the same candidate encoding and decoding mode group. The video encoding and decoding method of claim 7, wherein the candidate encoding and decoding modes of multiple merge candidates whose motion difference is greater than the threshold are assigned to the same candidate encoding and decoding mode group. The video encoding and decoding method as described in request item 7, which also includes: identifying a candidate codec mode group with the lowest representation cost among the candidate codec mode groups; and Send or receive an index, which is used to select a candidate codec mode from the identified candidate codec mode group. The video encoding and decoding method of claim 13, wherein the representative cost of the identified candidate encoding and decoding mode group is the average, maximum or minimum cost of the candidate encoding and decoding modes of the identified candidate encoding and decoding mode group. value. The video encoding and decoding method as described in request item 7, which also includes: Send or receive an index used to select a candidate codec mode group; and A candidate codec mode is identified from the selected candidate codec mode group based on the cost of the selected candidate codec mode. The video encoding and decoding method of claim 1, wherein the candidate encoding and decoding modes include multiple merging candidates of the current block. The video encoding and decoding method as described in claim 16, wherein the merging candidates of the current block include: (i) Use multiple merge candidates combining inter- and intra-prediction; or (ii) Multiple merging candidates predicted using affine transform motion compensation. In the video encoding and decoding method of claim 1, the candidate encoding and decoding modes include a plurality of candidate encoding and decoding modes corresponding to a plurality of different combinations, and the different combinations are used to refine the distance and offset of the motion information. As for the video encoding and decoding method described in claim 1, the candidate encoding and decoding modes include a plurality of candidate encoding and decoding modes corresponding to different candidate weights, and the different candidate weights are used to combine inter predictions in different directions. For the video encoding and decoding method described in claim 1, the candidate encoding and decoding modes include a plurality of candidate encoding and decoding modes corresponding to a plurality of different models, and the different models are used to derive the current block based on a plurality of luminance samples of the current block. Multiple predictors for multiple chroma samples of the current block. An electronic device including: A video codec circuit configured to perform multiple operations, including: receiving data for a block of pixels to be encoded or decoded into a current block of a current picture of a video; Identify multiple candidate codec modes applicable to the current block; Identify a first candidate coding and decoding mode group, the first candidate coding and decoding mode group is a subset of the candidate coding and decoding modes, wherein the number of candidate coding and decoding modes in the first candidate coding and decoding mode group is less than the number of candidate coding and decoding modes The number of candidate coding and decoding modes of the candidate coding and decoding modes; Select a candidate codec mode from the first candidate codec mode group; and The current block is encoded or decoded using the selected candidate encoding and decoding mode.