TW202002638A - Boundary enhancement for sub-block - Google Patents

Abstract

Devices, systems and methods for boundary enhancement for sub-block based prediction are described. Partitioning a block can result in discontinuities between adjacent sub-blocks that may introduce some undesirable high-frequency energy in the residual signal, which can deteriorate the performance of subsequent transform coding. Implementations of the disclosed technology can reduce the effect of the discontinuities. In a representative aspect, a method for video coding includes partitioning a video block into multiple sub-blocks, forming a first prediction candidate based on a sub-block based prediction of a plurality of samples in the video block, forming a second prediction candidate based on an inter prediction of the plurality of samples in a sub-block boundary region, forming a final prediction candidate as a function of the first prediction candidate and the second prediction candidate, and processing the video block using the final prediction candidate.

Description

Sub-block boundary enhancement

Generally speaking, this application document is directed to image and video coding technologies. [Cross reference to related applications] In accordance with the provisions of the applicable Patent Law and/or the Paris Convention, this application promptly requests the priority and benefits of the international patent application number PCT/CN2018/093633 filed on June 29, 2018. According to US law, the entire disclosure of International Patent Application No. PCT/CN2018/093633 is incorporated herein by reference as part of the disclosure of this application.

Digital video uses the most bandwidth on the Internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, it is expected that the demand for digital video bandwidth will continue to grow.

Motion compensation is a technique in video processing that gives previous and/or future frames and predicts frames in the video by considering the motion of the camera and/or objects in the video. Motion compensation can be used to encode and decode video data to achieve video compression.

A device, system, and method related to boundary enhancement based on sub-block prediction of image and video encoding are described.

In a typical aspect, the technology of the present disclosure can be used to provide a method of video encoding. The method includes: dividing a video block into multiple sub-blocks; forming a first prediction candidate based on sub-block-based prediction of multiple samples in the video block; forming a second prediction candidate based on inter-prediction of multiple samples in the sub-block boundary region Forming the final prediction candidate as a function of the first prediction candidate and the second prediction candidate; and processing the video block using the final prediction candidate.

In another typical aspect, the above method can be implemented in the form of processor-executable code and stored in a computer-readable program medium.

In yet another typical aspect, an apparatus is disclosed that is configured or operable to perform the above method. The device may include a processor programmed to implement the method.

In yet another typical aspect, a video decoding device can implement the method described herein.

The above and other aspects and features of the disclosed technology are described in more detail in the drawings, specification, and patent scope.

Due to the increasing demand for high-resolution video, video encoding methods and techniques are ubiquitous in modern technology. Video transcoders usually include electronic circuits or software that compresses or decompresses digital video, and is constantly being improved to provide higher coding efficiency. The video transcoder converts uncompressed video to a compressed format, or vice versa. Video quality, amount of data used to represent video (determined by bit rate), complexity of encoding and decoding algorithms, sensitivity to data loss and errors, ease of editing, random access, and end-to-end delay (delay) There is a complicated relationship. Compression formats generally conform to standard video compression specifications, such as High Efficiency Video Coding (HEVC) standards (also known as H.265 or MPEG-H Part 2), multi-function video coding standards to be finalized, or other current and/or future Video coding standard.

Subblock-based prediction was first introduced into the video coding standard by the High Efficiency Video Coding (HEVC) standard. Using sub-block-based prediction, a block (such as a coding unit (CU) or prediction unit (PU)) is divided into several non-overlapping sub-blocks. Different sub-blocks can be assigned different motion information, such as a reference index or motion vector (MV), and perform motion compensation (MC) separately for each sub-block. FIG. 1 shows an example of subblock-based prediction.

Embodiments of the disclosed technology can be applied to existing video coding standards (eg, HEVC, H.265) and future standards to improve execution time performance. In this article, chapter titles are used to improve the readability of the description, and the discussion or embodiments (and/or implementation) are not limited to the respective chapters in any way.

In addition, although some embodiments are described with reference to multi-function video encoding or other specific video transcoders, the disclosed technology is also applicable to other video encoding technologies. In addition, although some embodiments describe the video encoding step in detail, it should be understood that the corresponding decoding step (revocation of encoding) will be implemented by the decoder. In addition, the term "video processing" includes video encoding or compression, video decoding or decompression, and video transcoding, where video pixels are expressed from one compression format to another compression format or at different compression bit rates.

1. Example of Joint Exploration Model (JEM)

In some embodiments, a reference software called Joint Exploration Model (JEM) is used to explore future video coding technologies. In JEM, subblock-based prediction is used in various coding tools, such as affine prediction, optional time-domain motion vector prediction (ATMVP), space-time motion vector prediction (STMVP), bidirectional optical flow (BIO), and frame Play rate up conversion (FRUC), local self-adjusting motion vector resolution (LAMVR), overlapping block motion compensation (OBMC), local lighting compensation (LIC), and decoder-side motion vector refinement (DMVR).

1.1 Example of affine prediction

In HEVC, only translational motion models are applied to motion compensated prediction (MCP). However, cameras and objects may have multiple movements, such as zoom in/out, rotation, perspective movements, and/or other irregular movements. On the other hand, JEM applies simplified affine transformation motion compensation prediction. FIG. 2 shows an example of the affine sports field of the block 200 described by two control point motion vectors V ₀ and V ₁ . The motion vector field (MVF) of block 200 can be described by the following equation:

等式（1）

Equation (1)

是左上角控制點的運動向量，並且

是右上角控制點的運動向量。為了簡化運動補償預測，可以應用基於子塊的仿射變換預測。子塊大小MxN推導如下：as shown in picture 2,

Is the motion vector of the control point in the upper left corner, and

Is the motion vector of the control point in the upper right corner. In order to simplify motion compensation prediction, affine transformation prediction based on sub-blocks may be applied. The sub-block size MxN is derived as follows:

等式（2）

Equation (2)

)是左下控制點的運動向量，其根據等式（1）計算。如果需要，M和N可以被向下調節使其分別作為w和h的除數。Here, MvPre is the motion vector fractional accuracy (for example, 1/16 in JEM).

) Is the motion vector of the lower left control point, which is calculated according to equation (1). If necessary, M and N can be adjusted downward to make them the divisors of w and h, respectively.

FIG. 3 shows an example of an affine MVF for each sub-block of block 300. In order to derive the motion vector of each M×N sub-block, the motion vector of the center sample of each sub-block can be calculated according to equation (1) and rounded to the motion vector fractional accuracy (for example, 1/16 in JEM). Then, a motion compensation interpolation filter can be applied to generate the prediction of each sub-block using the derived motion vector. After the MCP, the high-precision motion vectors of each sub-block are rounded and saved to the same accuracy as the normal motion vectors.

的候選列表。In JEM, there are two affine motion modes: AF_INTER mode and AF_MERGE mode. For CUs with a width and height greater than 8, the AF_INTER mode can be applied. In the bitstream, CU level affine flags are signaled to indicate whether to use AF_INTER mode. In AF_INTER mode, use adjacent blocks to construct pairs with motion vectors

Candidate list.

FIG. 4 shows an example of motion vector prediction (MVP) of the block 400 in the AF_INTER mode. As shown in FIG. 4, v _{0 is} selected from the motion vectors of sub-blocks A, B, or C. The motion vectors of adjacent blocks can be scaled according to the reference list. The motion vector can also be scaled according to the relationship between the picture order count (POC) referenced by adjacent blocks, the POC referenced by the current CU, and the POC current CU. The method of selecting v ₁ from adjacent sub-blocks D and E is similar. If the number of candidate lists is less than 2, the list is filled by copying the pair of motion vectors composed of each AMVP candidate. When the candidate list is greater than 2, the candidates can be first sorted according to adjacent motion vectors (for example, based on the similarity of two motion vectors in a pair of candidates). In some implementations, the first two candidates are retained. In some embodiments, a rate distortion (RD) cost check is used to determine which motion vector pair candidate is selected as the control point motion vector prediction (CPMVP) of the current CU. The index indicating the position of CPMVP in the candidate list may be signaled in the bitstream. After the CPMVP of the current affine CU is determined, affine motion estimation is applied and the control point motion vector (CPMV) is found. Then, the difference between CPMV and CPMVP is signaled in the bit stream.

When the CU is applied in AF_MERGE mode, it obtains the first block coded in affine mode from valid neighboring reconstruction blocks. FIG. 5A shows an example of the selection order of the candidate blocks of the current CU 500. As shown in FIG. 5A, the selection order may be from the left (501), top (502), top right (503), bottom left (504) to top left (505) of the current CU 500. FIG. 5B shows another example of the candidate block of the current CU 500 in the AF_MERGE mode. If the adjacent lower left block 501 is encoded in an affine mode, as shown in FIG. 5B, the motion vectors v ₂ , v _3, and v _{4 of the} upper left corner, upper right corner, and lower left corner of the CU containing the sub-block 501 are derived. The motion vector v ₀ in the upper left corner of the current CU 500 is calculated based on v ₂ , v ₃ and v ₄ . The motion vector v ₁ at the upper right of the current CU can be calculated accordingly.

After calculating the CPMV v ₀ and v ₁ of the current CU according to the affine motion model in equation (1), the MVF of the current CU can be generated. In order to identify whether the current CU uses the AF_MERGE mode encoding, when at least one adjacent block is encoded in the affine mode, the affine flag can be signaled in the bit stream.

In JEM, the non-Merge affine mode can be used only when the width and height of the current block are greater than 8, and the affine mode can be used only when the area of the current block (that is, width x height) is not less than 64.

1.2 Examples of optional time-domain motion vector prediction (ATMVP)

In the ATMVP method, the temporal motion vector prediction (TMVP) method is modified by extracting multiple sets of motion information (including motion vectors and reference indices) from blocks smaller than the current CU.

FIG. 6 shows an example of the ATMVP motion prediction process of the CU 600. The ATMVP method predicts the motion vector of sub-CU 601 in CU 600 in two steps. The first step is to identify the corresponding block 651 in the reference picture 650 with the time vector. The reference picture 650 is also called a motion source picture. The second step is to divide the current CU 600 into sub-CU 601 and obtain the motion vector and reference index of each sub-CU from the block corresponding to each sub-CU.

In the first step, the reference picture 650 and the corresponding block are determined by the motion information of the spatial neighboring block of the current CU 600. To avoid repeated scanning processing of adjacent blocks, the first Merge candidate in the current Merge candidate list of the CU 600 is used. The first available motion vector and its associated reference index are set as the index of the time vector and the motion source picture. In this way, compared to TMVP, the corresponding block can be more accurately identified, where the corresponding block (sometimes called a juxtaposed block) is always located in the lower right corner or center position relative to the current CU.

In the second step, by adding the time vector to the coordinates of the current CU, the corresponding block of the sub-CU 651 is identified by the time vector in the motion source picture 650. For each sub-CU, the motion information of its corresponding block (for example, the smallest motion grid covering the center sample) is used to derive the motion information of the sub-CU. After recognizing the motion information of the corresponding N×N block, it is converted into the motion vector and reference index of the current sub-CU in the same way as the TMVP of HEVC, in which motion scaling and other programs are applied. For example, the decoder checks whether the low-latency condition is satisfied (for example, the POC of all reference pictures of the current picture is less than the POC of the current picture), and may use the motion vector MV _x (for example, the motion vector corresponding to the reference picture list X) to The motion vector MV _{y of} each sub-CU is predicted (for example, X is equal to 0 or 1 and Y is equal to 1−X).

1.3 Example of space-time motion vector prediction (STMVP)

In the STMVP method, the motion vector of the sub-CU is derived recursively according to the raster scan order. FIG. 7 shows an example of one CU with four sub-blocks and adjacent blocks. Consider an 8×8 CU 700, which includes four 4×4 sub-CUs A (701), B (702), C (703), and D (704). The adjacent 4×4 blocks in the current frame are marked as a (711), b (712), c (713), and d (714).

The motion derivation of sub-CU A starts by identifying its two spatial neighbors. The first neighbor is the N×N block (block c 713) above the sub-CU A 701. If this block c (713) is unavailable or intra-coded, then check the other NxN blocks above sub-CU A (701) (from left to right, starting at block c 713). The second neighbor is a block to the left of sub-CU A 701 (block b 712). If block b (712) is not available or is internally coded, then check the other blocks on the left side of sub-CU A 701 (from top to bottom, starting at block b 712). The motion information obtained from neighboring blocks for each list is scaled to the first reference frame of the given list. Next, the temporal motion vector prediction (TMVP) of sub-block A 701 is derived according to the same procedure as that specified in HEVC and TMVP. The motion information of the collocated block at block D 704 is extracted and scaled accordingly. Finally, after retrieving and scaling the motion information, all available motion vectors are averaged separately for each reference list. The average motion vector is specified as the motion vector of the current sub-CU.

1.4 Example of bidirectional optical flow (BIO)

The bidirectional optical flow (BIO) method is based on block motion compensation to refine the sample direction motion for bidirectional prediction. In some implementations, sample-level motion refinement does not use signaling.

為塊運動補償後到參考k（k=0，1）的亮度值，並且

,

分別為

梯度的水平分量和垂直分量。假設光流是有效的，則運動向量場

由下式給出：Assume

The brightness value to the reference k (k=0, 1) after block motion compensation, and

,

Are

The horizontal and vertical components of the gradient. Assuming that the optical flow is valid, the motion vector field

Is given by:

等式（3）

Equation (3)

和其導數

,

。該多項式在t=0時的值是BIO預測：Combining this optical flow equation with the Hermitian interpolation of each sample motion trajectory yields a unique third-order polynomial that simultaneously matches the function value at the end

And its derivative

,

. The value of this polynomial at t=0 is the BIO prediction:

等式（4）

Equation (4)

和

表示到參考幀的距離。基於Ref0和Ref1的POC計算距離

和

：

=POC(current) − POC(Ref₀ ),

= POC(Ref₁ ) − POC(current)。如果兩個預測都來自同一個時間方向（都來自過去或都來自未來），則符號是不同的（例如，

）。在這種情況下，如果預測不是來自同一時間點（例如，

），則應用BIO。兩個參考區域都具有非零運動（例如，

），並且塊運動向量與時間距離成比例（例如，

）。FIG. 8 shows an example optical flow trajectory in the bidirectional optical flow (BIO) method. Here,

with

Indicates the distance to the reference frame. Calculate distance based on Ref0 and Ref1 POC

with

:

=POC(current) − POC(Ref ₀ ),

= POC(Ref ₁ ) − POC(current). If both predictions come from the same time direction (both from the past or both from the future), the signs are different (for example,

). In this case, if the predictions are not from the same point in time (for example,

), then apply BIO. Both reference areas have non-zero motion (for example,

), and the block motion vector is proportional to the time distance (for example,

).

The motion vector field is determined by minimizing the difference Δ between points A and B. 9A to 9B show examples in which the motion trajectory intersects the reference frame plane. For ∆, the model uses only the first linear term of the local Taylor expansion:

等式(5)

Equation (5)

。假設在局部周圍區域的運動是一致的，那麼Δ可以在以當前預測點（i，j）為中心的（2 M+1）x（2 M+1）方形視窗Ω內最小化，其中M等於2：All values in the above equation depend on the sample position, expressed as

. Assuming that the motion in the local surrounding area is consistent, then Δ can be minimized in the (2 M+1) x (2 M+1) square window Ω centered on the current prediction point (i, j), where M is equal to 2:

等式 (6)

Equation (6)

For this optimization problem, JEM uses a simplified method, first minimizing in the vertical direction, and then minimizing in the horizontal direction. The results are as follows:

等式(7)

Equation (7)

等式(8)

Equation (8)

among them,

等式(9)

Equation (9)

In order to avoid division by zero or very small values, the normalization parameters r and m can be introduced in equations (7) and (8), where

等式 (10)

Equation (10)

等式 (11)

Equation (11)

Here, d is the bit degree of the video sample.

，圖9A示出了塊900外部的訪問位置示例。如圖9A所示，在等式（9）中，以預測區塊邊界上當前預測點為中心的（2M+1）x（2M+1）的方形視窗Ω需要訪問區塊外的位置。在JEM中，塊外的值

設置為等於塊內最近的可用值。例如，這可以實現為填充區域901，如圖9B所示。In order to make the memory access of the BIO the same as the conventional bidirectional predictive motion compensation, calculate all the prediction and gradient values of the current block position

9A shows an example of the access location outside the block 900. As shown in FIG. 9A, in equation (9), a square window Ω of (2M+1) x (2M+1) centered on the current prediction point on the boundary of the prediction block needs to access a position outside the block. In JEM, the value outside the block

Set equal to the latest available value in the block. For example, this can be realized as a filled area 901, as shown in FIG. 9B.

Using BIO, the sports field of each sample can be refined. In order to reduce the computational complexity, BIO based on block design is adopted in JEM. Motion refinement can be calculated based on 4x4 blocks. In block-based BIO, the _sn values in equation (9) of all samples in the 4x4 block can be aggregated, and then the aggregated value of _sn can be used for the deduced BIO motion vector offset of the 4x4 block. More specifically, the following equation can be used for block-based BIO derivation:

等式（12）

Equation (12)

Here, b _k represents the sample group of the k-th 4x4 block belonging to the prediction block. Equation (7) and Equation (8) is replaced by s _n ((sn, bk) >> 4 ) to derive motion vectors associated with the offset.

，否則其被設置為

。In some cases, due to noise or irregular movement, BIO's MV group (regiment) may not be reliable. Therefore, in BIO, the size of the MV group is fixed to a threshold. The threshold is determined based on whether the reference pictures of the current picture all come from one direction. For example, if all the reference pictures of the current picture come from one direction, the value of the threshold is set to

, Otherwise it is set to

.

，首先使用BIOfilterS對信號進行垂直內插，該BIOfilterS對應於具有去縮放標度位移d-8的分數位置fracY。然後在水平方向上應用梯度濾波器BIOfilterG，該BIOfilterG對應於具有去縮放標度位移18-d的分數位置fracX。對於垂直梯度

，使用BIOfilterG垂直地應用梯度濾波器，該BIOfilterG對應於具有去縮放標度位移d-8的分數位置fracY。然後，然後在水平方向上使用BIOfilterS執行信號替換，該BIOfilterS對應於具有去縮放標度位移18-d的分數位置fracX。用於梯度計算BIOfilterG和信號替換BIOfilterS的插值濾波器的長度可以更短（例如，6-tap），以保持合理的複雜度。表2示出了可用在BIO中塊運動向量的不同分數位置的梯度計算的示例濾波器。表3示出了可用在BIO中預測信號生成的插值示例濾波器。An operation consistent with the HEVC motion compensation process (for example, 2D separable finite impulse response (FIR)) can be used to simultaneously calculate the BIO gradient through motion compensation interpolation. In some embodiments, the input of the 2D separable FIR is the same reference frame as the motion compensation process, and the fractional position (fracX, fracY) according to the fractional part of the block motion vector. For horizontal gradient

First, the signal is first vertically interpolated using BIOfilterS, which corresponds to the fractional position fracY with a descaling scale displacement d-8. A gradient filter BIOfilterG is then applied in the horizontal direction, which corresponds to the fractional position fracX with a descaling scale shift of 18-d. For vertical gradient

, The gradient filter is applied vertically using BIOfilterG, which corresponds to the fractional position fracY with a descaling scale displacement d-8. Then, the signal replacement is then performed in the horizontal direction using BIOfilterS, which corresponds to the fractional position fracX with a descaling scale displacement of 18-d. The length of the interpolation filter used for gradient calculation BIOfilterG and signal replacement BIOfilterS can be shorter (for example, 6-tap) to maintain a reasonable complexity. Table 2 shows example filters that can be used for gradient calculation of different fractional positions of block motion vectors in BIO. Table 3 shows interpolation example filters that can be used in BIO prediction signal generation.

Table 2 Example filters for gradient calculation in BIO

Table 3 Example interpolation filters used for prediction signal generation in BIO

In JEM, when two predictions are from different reference pictures, BIO can be applied to all bidirectional prediction blocks. When Local Illumination Compensation (LIC) is enabled for the CU, BIO can be disabled.

In some embodiments, OBMC is applied to the block after normal MC processing. To reduce computational complexity, BIO may not be applied during OBMC processing. This means that during OBMC processing, when using its own MV, BIO is applied to the MC processing of the block, and when using the MV of the adjacent block, BIO is not applied to the MC processing of the block.

1.5 Example of frame rate up conversion (FRUC)

For the CU, when its Merge flag is true, it can signal the FRUC flag. When the FRUC flag is false, the Merge index can be signaled and the conventional Merge mode can be used. When the FRUC flag is true, another FRUC mode flag can be signaled to indicate which method (eg, bilateral matching or template matching) will be used to derive the motion information for the block.

On the encoder side, it is decided whether to use the FRUC Merge mode for the CU based on the RD cost selection made for the normal Merge candidate. For example, multiple matching patterns of CUs are checked by using RD cost selection (for example, bilateral matching and template matching). The mode leading to the lowest cost is further compared with other CU modes. If the FRUC matching mode is the most effective mode, then for the CU, the FRUC flag is set to true, and the relevant matching mode is used.

In general, the motion derivation process in the FRUC Merge mode has two steps: first perform CU-level motion search, and then perform sub-CU-level motion refinement. At the CU level, based on bilateral matching or template matching, the initial motion vector of the entire CU is derived. First, a MV candidate list is generated, and the candidate that results in the lowest matching cost is selected as the starting point for further CU-level refinement. Then local search based on bilateral matching or template matching is performed near the starting point. The MV result of the minimum matching cost is taken as the MV value of the entire CU. Subsequently, starting from the derived CU motion vector, the motion information is further refined at the sub-CU level.

For example, for the W×H CU motion information derivation, the following derivation process is performed. In the first stage, the MV of the entire W×H CU is derived. In the second stage, the CU is further divided into M×M sub-CUs. The value of M is calculated according to (16), D is the pre-defined division depth, and is set to 3 by default in JEM. Then derive the MV value of each sub-CU.

等式（13）

Equation (13)

FIG. 10 shows an example of bilateral matching used in the Frame Play Rate Up Conversion (FRUC) method. By finding the closest match between two blocks along the motion trajectory of the current CU (1000) in two different reference pictures (1010, 1011), bilateral motion matching is used to obtain the motion information of the current CU. Under the assumption of continuous motion trajectory, the time distance between the motion vectors MV0 (1001) and MV1 (1002) pointing to the two reference blocks and the current picture and the two reference pictures (for example, TD0 (1003) and TD1 (1004)) Proportional to. In some embodiments, when the current picture 1000 is temporarily located between two reference pictures (1010, 1011) and the time distance from the current picture to the two reference pictures is the same, bilateral matching becomes a mirror-based bidirectional MV.

FIG. 11 shows an example of template matching used in the FRUC method for up-conversion of the frame playback rate. Template matching can be used to find the closest match between the template in the current picture (for example, the top and/or left adjacent blocks of the current CU) and the block in the reference picture 1110 (for example, the same size as the template) Current CU 1100 motion information. In addition to the FRUC Merge mode described above, template matching can also be applied to the AMVP mode. In JEM and HEVC, AMVP has two candidates. Through the template matching method, new candidates can be derived. If the newly derived candidate matched by the template is different from the first existing AMVP candidate, it is inserted at the beginning of the AMVP candidate list, and then the list size is set to 2 (for example, by deleting the second existing AMVP candidate) . When applied to the AMVP mode, only CU level search is applied.

The MV candidates set at the CU level may include the following: (1) the original AMVP candidate, if the current CU is in AMVP mode, (2) all Merge candidates, (3) several MVs in the interpolated MV field (described later), and the top And the left adjacent motion vector.

When bilateral matching is used, each valid MV of the Merge candidate can be used as an input to generate MV pairs that are assumed to be bilateral matches. For example, a valid MV of the Merge candidate at the reference list A is (MVa, ref _{a )} . Then find the reference picture ref _{b of} its paired bidirectional MV in another reference list B, so that ref _a and ref _b are located on different sides of the current picture in time. If the reference ref _b in the reference list B is not available, the reference ref _{b is} determined to be _a different reference from the reference ref _a , and its temporal distance to the current picture is the minimum distance in list B. After the reference ref _{b is} determined, the MVb is derived by scaling the MVa based on the time distance between the current picture and the reference ref _a and the reference ref _b .

In some implementations, four MVs from the interpolated MV field can also be added to the CU-level candidate list. More specifically, the interpolated MV at the current CU position (0, 0), (W/2, 0), (0, H/2), and (W/2, H/2) is added. When applying FRUC in AMVP mode, the original AMVP candidate is also added to the CU-level MV candidate set. In some implementations, at the CU level, 15 MVs of AMVP CU and 13 MVs of Merge CU can be added to the candidate list.

The MV candidates set at the sub-CU level include the MVs determined from the CU level search, (2) the top, left, top left, and top right adjacent MVs, (3) the scaled version of the MV juxtaposed in the reference picture, (4) one or Multiple ATMVP candidates (for example, up to four) and (5) one or more STMVP candidates (for example, up to four). The zoomed MV from the reference picture is derived as follows. The reference pictures in both lists are traversed. The MV at the collocated position of the sub-CU in the reference picture is scaled to the reference of the starting CU-level MV. ATMVP and STMVP candidates can be the first four. At the sub-CU level, one or more MVs (for example, up to 17) are added to the candidate list.

Generation of interpolated MV field. Before encoding the frame, an interpolated motion field for the entire picture is generated based on the unidirectional ME. The sports field can then be used as a MV level or sub-CU level MV candidate.

In some embodiments, the motion field of each reference picture in the two reference lists is traversed on a 4x4 block level. FIG. 12 shows an example of unilateral motion estimation (ME) 1200 in the FRUC method. For each 4×4 block, if the motion associated with the block passes through the 4×4 block in the current picture, and the block is not assigned any interpolated motion, the motion of the reference block is scaled to The current picture (in the same way as the MV scaling of TMVP in HEVC), and the scaling motion is assigned to the block in the current frame. If an MV without scaling is assigned to a 4×4 block, the motion of the block is marked as unavailable in the interpolation motion field.

Interpolation and matching costs. When the motion vector points to the fractional sampling position, motion compensation interpolation is required. In order to reduce complexity, bilinear interpolation and template matching are used instead of the conventional 8-order HEVC interpolation.

The calculation of matching costs is a bit different at different steps. When selecting a candidate from the CU-level candidate set, the matching cost may be the absolute sum difference (SAD) of bilateral matching or template matching. After the initial MV is determined, the matching cost C of the bilateral search at the sub-CU level is calculated as follows:

等式（14）

Equation (14)

Here, w is a weight coefficient. In some embodiments, w may be set to 4 empirically. MV and MVs indicate the current MV and the starting MV, respectively. SAD can still be used as the matching cost of pattern matching searching at the sub-CU level.

In FRUC mode, the MV is derived by using only the luminance (brightness) samples. The exported motion will be used for luma (luminance) and chroma (chroma) for MC inter prediction. After the MV is determined, an 8-order (8-taps) interpolation filter is used for luma and a 4-order (4-taps) interpolation filter is used for chroma.

MV refinement is a pattern-based MV search, with the cost of bilateral matching or template matching as the standard. In JEM, two search modes are supported—unlimited center offset diamond search (UCBDS) and self-adjusting cross search, and MV refinement is performed at the CU level and the sub-CU level, respectively. For the MV refinement at the CU level and the sub-CU level, both search the MV directly with the accuracy of the 1/4 brightness sample MV, followed by the MV refinement of the one-eighth brightness sample. The search range of MV refinement of CU and sub-CU steps is set to 8 luminance samples.

In the bilateral matching Merge mode, bidirectional prediction is applied because the CU motion information is based on the closest match between two blocks along the current CU motion trajectory in two different reference pictures. In the template matching Merge mode, the encoder can choose from one-way prediction in list 0, one-way prediction in list 1, or two-way prediction for CU. This selection can be based on the template matching costs as follows: If costBi>=factor*min (cost0, cost1) Then use two-way prediction; Otherwise, if cost0>=cost1 Then use the one-way prediction in list 0; otherwise, Use the one-way forecast in Listing 1;

Here, cost0 is the SAD matched by the template of list 0, cost1 is the SAD matched by the template of list 2, and costBi is the SAD matched by the bidirectional template. For example, when the value of factor is equal to 1.25, it means that the selection process is shifted towards bidirectional prediction. Inter prediction direction selection can be applied to CU-level template matching processing.

1.6 Example of MV derived for MC in chroma component

In one example, the HEVC standard defines how to derive the MV used by the MC in the chroma component (called mvC) from the MV used by the MC in the luma component (called mv). In general, mvC is calculated by multiplying mv by a coefficient, depending on the color format, such as 4:2:0 or 4:2:2.

2. Examples of existing methods based on sub-block implementation

In some existing implementations, subblock-based prediction is used because it is usually more accurate than whole-block prediction because it can use its own MV to divide the block into more parts.

However, this division may cause discontinuity between two adjacent sub-blocks along its boundary. This discontinuity may introduce some unwanted high-frequency energy into the residual signal, which may reduce the performance of subsequent transform coding.

3. Example method of subblock-based prediction in video coding

The examples described below for various implementations illustrate the use of boundary enhancement for sub-block-based prediction to improve video coding efficiency and enhance existing and future video coding standards. In the following examples, which should not be interpreted as limiting, the width and height of the current block for the component are denoted as W and H, respectively, and the width and height of the sub-blocks allocated to the component are denoted as w and h, respectively.

Example 1. Predict samples along subblock boundaries by subblock-based prediction (called first prediction); also predict them by another inter prediction (called second prediction). The first prediction and the second prediction are used together to derive the final prediction of samples along the boundary. The final prediction of samples that do not follow the boundary may not change, for example, equal to the first prediction.

(A) In one example, the boundary includes only the inner boundary, for example, the boundary between sub-blocks, as shown in the example in FIG. 13A. The shaded area covers samples along the border.

(B) In one example, the boundary includes an inner boundary and an outer boundary, for example, a boundary between a sub-block and other blocks that have been encoded or decoded, as shown in the example in FIG. 13B. The shaded area covers samples along the border.

Example 2. In one example, the range of samples along the boundary to be enhanced may be predefined or adaptive. For example, there may be N columns of samples along the vertical boundary and M rows of samples along the horizontal boundary to be enhanced. 14A and 14B show examples of prediction samples with M=N=2.

(A) In one example, M and/or N depend on the width and height of the sub-block. For example, if the shape of the sub-block is 4×4, then M=N=2; if the shape of the sub-block is 8×8, then M=N=4.

(B) In one example, M and/or N depend on the color component. For example, for the luma component, M=N=2; for the chroma component, M=N=1.

(C) In one example, M and/or N depend on the position of the boundary. For example, if the boundary is between the sub-block and the adjacent block for encoding/decoding, then M=N=2; if the boundary is between two sub-blocks, then M=N=1.

(D) In one example, M and/or N may depend on the position of the sub-block. Or, it may depend on how many neighboring blocks are encoded/decoded and/or how many neighboring block prediction blocks are available.

(E) In one example, M and/or N is signaled by the encoder to the decoder. For example, M and N can be signaled in a video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), slice header, coding tree unit (CTU), or coding unit (CU).

Example 3. In one example, the first prediction can be obtained by a conventional method based on sub-block prediction, and the second prediction is generated using the same model that generated the first prediction but using different locations. For example, both the first prediction and the second prediction are generated by equation (1), but have different (x, y).

(A) In an example, as shown in FIG. 15A, the first prediction is generated using the affine prediction method in JEM, where the MV of each sub-block is set by the equation (1) in the center of the sub-block (X, y). That is to say, assuming that the upper left corner of the sub-block is at (i×w, j×h), then the MV of the sub-block ((i, j)th sub-block) is calculated by equation (1), where ( x,y)= (i×w+w/2, j×h+h/2). Then the second prediction can be generated as:

(I) At the vertical boundary as shown in FIG. 15B. The second prediction is generated by the MC using auxiliary sub-blocks covering vertical boundaries. The size of the auxiliary sub-block (denoted as w’×h’) may be the same as the size of the atomic block (for example, w’=w and h’=h), or the two sizes may be different. In the example of FIG. 15B (where w'=w and h'=h), the upper left corner of the auxiliary sub-block (at the i-th row and j-th column of the auxiliary sub-block) is (i×w+w/2, j×h), and the MV of the auxiliary sub-block is calculated by equation (1), where (x, y)=(i×w+w, j×h+h/2).

(Ii) At the horizontal boundary shown in Figure 15C. The second prediction is generated by the MC using auxiliary sub-blocks covering horizontal boundaries. The size of the auxiliary sub-block (denoted as w’×h’) may be the same as the size of the original sub-block (ie, w’=w and h’=h), or the two sizes may be different. In the example of FIG. 15C (where w'=w and h'=h), the upper left corner of the auxiliary sub-block is (i×w, j×h+h/2), and the MV of the auxiliary sub-block is determined by the equation (1) It is calculated that (x, y) = (i×w+w/2, j×h+h).

(Iii) At the intersection of the vertical boundary and the horizontal boundary as shown in FIG. 15D. The second prediction is generated by the MC using auxiliary sub-blocks covering vertical and horizontal boundaries. The size of the auxiliary sub-block (denoted as w’×h’) may be the same as the size of the original sub-block (ie, w’=w and h’=h), or the two sizes may be different. In the example of FIG. 15D where w'=w and h'=h), the upper left corner of the auxiliary sub-block is (i×w+w/2, j×h+h/2), and the MV of the auxiliary sub-block Calculated by formula (1), where (x, y) = (i×w+w, j×h+h).

(Iv) For the outer boundary, the motion information used to generate the second prediction can be derived in the same way as the inner boundary (which can be derived on a smaller block size), as shown in FIGS. 15A to 15D. Alternatively, the second motion information derived for the nearest inner horizontal/vertical boundary can be used for the horizontal/vertical outer boundary.

(B) In one example, how to generate the second prediction depends on the width and/or height of the sub-block, and/or block shape, and/or block size, and/or coding mode information.

(C) In one example, how to generate the second prediction depends on the color component.

(D) In one example, how to generate the second prediction depends on the position of the boundary.

(E) In one example, the first prediction and the second prediction are from the same reference picture. Or, the second prediction and the first prediction are from different reference pictures.

Example 4. In one example, the final prediction of samples along the boundary is calculated as a function of the first prediction signal and the second prediction signal.

(A) In one example, the function is defined as a linear or nonlinear weighted sum.

(I) In an example, Pf=(w1×P1+w2×P2 + off)/(w1+w2), where Pf is the final prediction; P1 and P2 are the first and second predictions respectively; w1 and w2 are Two weighted values; off is the rounding offset, for example, off=(w1+w2)/2.

(Ii) In an example, Pf=(w1×P1+w2×P2 + off)>>B, where Pf is the final prediction; P1 and P2 are the first and second predictions respectively; w1 and w2 are the two weights Value, and w1+w2 = 2B; off is the rounding offset, for example, off=(w1+w2)/2. Some examples of weighted values (w1, w2): (1,1), (1,3), (3,1), (1,7), (7,1), (3,5), (5, 3) Wait.

(B) In one example, the function is defined as an exponential function.

(C) The weighted value or other function parameters may depend on one or some of the following conditions:

(I) W and H;

(Ii) w and h;

(Iii) w’ and h’;

(Iv) The location of samples along the border;

(V) The location of the border;

(Vi) Color components;

(Vii) Encoding information, including MV, QP, inter prediction mode, inter prediction direction, Merge mode or AMVP mode, etc.;

(Viii) distance to the border; and/or

(Ix) Boundary continuity, such as gradient value

Example 5. In one example, in addition to the second prediction, there may be a third, fourth, or kth prediction generated for boundary enhancement.

(A) In one example, two or more boundary enhancement operations may be performed in cascade. The final prediction output of the previous enhancement operation is input as the first prediction into the following enhancement operation.

(I) In one example, the vertical boundary is first enhanced, as shown in FIG. 15B, and finally Pf0 is predicted as an output. The horizontal boundary is then enhanced, as shown in FIG. 15C, where Pf0 is used as the first prediction, and Pf1 is finally predicted as the output. Pf1 is regarded as the true final prediction to derive the residual.

(Ii) In one example, the horizontal boundary is first enhanced, as shown in FIG. 15C, and finally Pf0 is predicted as an output. Then, use Pf0 as the first prediction and final prediction Pf1 as the output enhancement vertical boundary, as shown in FIG. 15B. Pf1 is regarded as the true final prediction to derive the residual.

(Iii) Alternatively, in addition, the vertical (or horizontal) boundary enhancement order of different sub-blocks may be defined as raster scan order, water wave shape, or others.

(B) In one example, two or more boundary enhancement operations can be performed in an independent manner. The first prediction is recorded as P(0), and the second to Kth predictions are recorded as P(1)~P(K-1). P(0)~P(K-1) are used together to derive the final prediction of samples along the boundary.

，其中w(r)是加權值，off是取整偏移量，例如，

。(I) In an example, Pf=

, Where w(r) is the weighted value and off is the rounding offset, for example,

.

，其中w（r）是加權值，

，並且off是取整偏移量，例如，

。(Ii) In an example,

, Where w(r) is the weighted value,

, And off is the rounding offset, for example,

.

Example 6. In one example, the encoder signals to the decoder whether to apply the boundary enhancement method and how to generate the second prediction. For example, the video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), slice header, coding tree unit (CTU), coding tree block (CTB), coding unit (CU) or The prediction unit (PU) signals the selection in an area covering multiple CTU/CTB/CU/PU.

Example 7. In one example, embodiments of the disclosed technology may be implemented with interleaving prediction methods. For example, in a specific division mode, boundary enhancement is performed for each sub-block based prediction. In addition, the boundaries of different division modes may be different.

Example 8. The proposed method can be applied to certain color components.

(A) In one example, only the brightness block can enable the proposed method.

(B) The proposed method can be applied to certain block sizes/shapes and/or certain sub-block sizes.

(C) The proposed method can be applied to some coding tools, such as ATMVP and/or affine.

The above example can be incorporated in the context of the following method, for example method 1600, which can be implemented on a video decoder and/or video transcoder.

16 shows a flowchart of an example method of video encoding. Method 1600 includes, in step 1610, dividing the video block into multiple sub-blocks.

Method 1600 includes, in step 1620, subblock-based prediction based on a plurality of samples in a video block forming a first candidate prediction.

The method 1600 includes in step 1630, forming a second prediction candidate based on inter prediction of a plurality of samples in a sub-block boundary region. In some embodiments, the first and second prediction candidates use the same prediction model, where the first prediction candidate is based on a first subset of multiple samples and the second prediction candidate is based on multiple samples that are different from the first subset The second subset.

In some embodiments, the sub-block boundary area is the inner boundary of the video data block, and wherein the sub-block boundary area includes prediction samples from neighboring sub-blocks of multiple sub-blocks, as described in the context of FIG. 13A. In other embodiments, the sub-block boundary area is the outer boundary of the video data block, and wherein the boundary samples include reconstructed samples from adjacent video data blocks, as described in the context of FIG. 13B.

In some embodiments, as described in the context of Example 2, the plurality of samples in the sub-block boundary area includes N column samples along the vertical boundary and M row samples along the horizontal boundary. For example, M or N may be based on the dimension of sub-blocks of multiple sub-blocks. For example, M or N may be based on the component type (eg, luma component or chroma component) of the sub-blocks of the plurality of sub-blocks. For example, M or N may be based on the position of the sub-block boundary area relative to the video block. For example, M or N may be based on the positions of sub-blocks of multiple sub-blocks. In some embodiments, M or N is selected in the video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), slice header, coding tree unit (CTU) or coding unit (CU) Signaling.

In some embodiments, as described in the context of Example 3, the second prediction candidate may be based on the dimensions of the sub-blocks of the plurality of sub-blocks or the dimensions of the video block, or the chroma components of the sub-blocks of the plurality of sub-blocks.

In some embodiments, sub-block based prediction is the same as inter prediction. In other embodiments, the first and second prediction candidates are based on the same reference picture. In yet other embodiments, the first prediction candidate is based on a first reference picture, and wherein the second prediction candidate is based on a second reference picture that is different from the first reference picture.

The method 1600 includes forming the final prediction candidate as a function of the first prediction candidate and the second prediction candidate of the plurality of samples in the sub-block boundary region in step 1640. In some embodiments, as described in the context of Example 4, the final prediction candidate (Pf) may be a weighted sum of the first prediction candidate (P1) and the second prediction candidate (P2). In one example, Pf = (w1×P1+w2×P2+offset)/(w1+w2), in another example, Pf = (w1×P1+w2×P2+offset)>>B, where w1 And w2 are weighted values, where offset = (w1+w2)/2 is the rounding offset, and where w1+w2=2B. In some embodiments, the final prediction candidate is the same as the first prediction candidate of multiple samples that are not in the sub-block boundary region.

In some embodiments, the weight value is based on the dimensions of the sub-blocks of the plurality of sub-blocks, the dimensions of the video block, the chroma components of the sub-blocks, one or more attributes of the video block, or the location of the sub-block boundary region. For example, one or more attributes include motion vector, quantization parameter (QP), inter prediction mode, inter prediction direction, Merge mode, or advanced motion vector prediction (AMVP) mode.

Method 1600 includes in step 1650, processing the video block using the final prediction candidate.

As described in the context of Example 5, method 1600 may further include further forming a final prediction candidate based on one or more additional prediction candidates, and each prediction candidate of one or more additional prediction candidates is based on prediction of multiple samples in the subblock boundary region . In some embodiments, the types of component prediction candidates may be in a video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), slice header, coding tree unit (CTU), or coding unit (CU ) In the signaling.

4. Example implementation of open technology

17 is a schematic diagram illustrating an example of a structure of a computer system or other control device 1700 that can be used to implement various parts of the disclosed technology, including (but not limited to) method 1600. In FIG. 17, the computer system 1700 includes one or more processors 1705 and memory 1710 connected through a network 1725. The network 1725 may represent any one or more individual physical buses, point-to-point connections, or both connected by appropriate bridges, interface cards, or controllers. Therefore, the network 1725 may include, for example, a system bus, a peripheral component connection (PCI) bus, a super transmission or an industry standard architecture (ISA) bus, a small computer system interface (SCSI) bus, and a universal serial bus (USB) , IIC (I2C) busbar or Institute of Electrical and Electronics Engineers (IEEE) standard 674 busbar (sometimes called "FireWire").

The processor 1705 may include a central processing unit (CPU) to control the overall operation of the host, for example. In some embodiments, the processor 1705 achieves this by executing software or firmware stored in the memory 1710. The processor 1705 may be or may include one or more programmable general-purpose or special-purpose microprocessors, digital signal processors (DSPs), programmable controllers, dedicated integrated circuits (ASICs), programmable logic devices ( PLD), etc., or a combination of these devices.

The memory 1710 may be or include a main memory of a computer system. Memory 1710 represents any suitable form of random access memory (RAM), read only memory (ROM), flash memory, etc., or a combination of these devices. In use, the memory 1710 may contain a set of machine instructions, among others, and when the processor 1705 executes the instructions, causes the processor 1705 to perform operations to implement the embodiments of the disclosed technology.

Also connected to the processor 1705 via the network 1725 is an (optional) network interface card 1715. The network interface card 1715 provides the computer system 1700 with the ability to communicate with remote devices such as storage clients and/or other storage servers, and may be, for example, an Ethernet interface card or a fiber channel interface card.

18 shows a block diagram of an example embodiment of a mobile device 1800 that may be used to implement various parts of the disclosed technology, including (but not limited to) method 1600. The mobile device 1800 may be a notebook computer, smart phone, tablet computer, video camera, or other device capable of processing video. The mobile device 1800 includes a processor or controller 1801 to process data, and a memory 1802 in communication with the processor 1801 to store and/or buffer data. For example, the processor 1801 may include a central processing unit (CPU) or a microcontroller unit (MCU). In some implementations, the processor 1801 may include a field programmable gate array (FPGA). In some implementations, the mobile device 1800 includes or communicates with a graphics processing unit (GPU), video processing unit (VPU), and/or wireless communication unit to implement various visual and/or communication data processing functions of a smartphone device. For example, the memory 1802 may include and store processor executable code, and when the processor 1801 executes the code, the mobile device 1800 is configured to perform various operations, such as receiving information, commands and/or data, processing information and data, and Send or provide processed information/data to another data device, such as an actuator or external display. To support various functions of the mobile device 1800, the memory 1802 can store information and data, such as instructions, software, values, images, and other data processed or referenced by the processor 1801. For example, various types of random access memory (RAM) devices, read only memory (ROM) devices, flash memory devices, and other suitable storage media may be used to implement the storage function of the memory 1802. In some implementations, the mobile device 1800 includes an input/output (I/O) interface 1803 to interface the processor 1801 and/or memory 1802 with other modules, units, or devices. For example, the I/O interface 1803 can interface with the processor 1801 and memory 1802 to utilize various wireless interfaces compatible with typical data communication standards, such as one or more computers and user equipment in the cloud. between. In some implementations, the mobile device 1800 can interface with other devices using a wired connection through the I/O interface 1803. The mobile device 1800 can also be connected to other external interfaces (such as data memory) and/or a visual or audio display 1804 for retrieval and transmission that can be processed by the processor, stored in memory or displayed on the display 1804 or output unit of an external device Information and information. For example, the display 1804 may display a video frame including a block (CU, PU, or TU) to which intra block copying is applied based on whether the block is encoded using a motion compensation algorithm according to the disclosed technology. .

In some embodiments, a video decoder device that can implement a method of subblock-based prediction as described herein may be used for video decoding. Various features of this method may be similar to method 1600 described above.

In some embodiments, the video decoding method may be implemented using a decoding device implemented on the hardware platform described in FIGS. 17 and 18.

Various embodiments and techniques disclosed in the documents herein can be described in the following list of examples.

1. A video processing method (for example, the method 1900 described in FIG. 19), including: dividing (step 1902) a video block into a plurality of sub-blocks; forming a final prediction candidate (step 1904) into a first prediction candidate and a second A function of a prediction candidate, the first prediction candidate corresponds to subblock-based prediction of a plurality of samples in a subblock boundary area, and the second prediction candidate corresponds to the plurality of subblocks in the subblock boundary area Inter prediction of samples; and processing (step 1906) the video block using the final prediction candidate. For example, the processing may include generating an encoded representation of the video block at the video transcoder, wherein the video block is encoded using predictive encoding of the video block. For example, the processing may include performing motion compensation at the video decoder final prediction candidate to generate a video block.

2. The method according to example 1, wherein the sub-block boundary area includes an inner boundary of the video block.

3. The method according to example 1, wherein the sub-block boundary area is an outer boundary of the video.

4. The method of example 1, wherein the plurality of samples in the sub-block boundary area include N-column samples along a vertical boundary and M-row samples along a horizontal boundary.

5. The method according to example 4, wherein M or N is based on the dimensions of the plurality of sub-blocks.

6. The method according to example 4, wherein M or N is based on the type of color components of the sub-blocks of the plurality of sub-blocks.

7. The method of example 4, wherein M or N is based on the position of the sub-block boundary area relative to the video block.

8. The method according to example 4, wherein M or N is based on the positions of sub-blocks of the plurality of sub-blocks.

9. The method according to example 4, wherein M or N is in a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, and a coding tree unit (CTU) Or be signaled in the coding unit (CU).

10. The method of example 1, wherein the first prediction candidate and the second prediction candidate use the same prediction model, wherein the first prediction candidate is based on a first subset of the plurality of samples, and The second prediction candidate is based on a second subset of the plurality of samples that is different from the first subset.

11. The method according to example 1 or 10, wherein the second prediction candidate is based on a dimension of a sub-block of the plurality of sub-blocks or a dimension of the video block.

12. The method according to example 1 or 10, wherein the second prediction candidate is based on color components of sub-blocks of the plurality of sub-blocks.

13. The method of example 1, wherein the sub-block based prediction is the same as the inter prediction.

14. The method of example 1, wherein the first prediction candidate and the second prediction candidate are based on the same reference picture.

15. The method of example 1, wherein the first prediction candidate is based on a first reference picture, and wherein the second prediction candidate is based on a second reference picture different from the first reference picture.

16. The method according to example 1, wherein the final prediction candidate (Pf) is a weighted sum of the first prediction candidate (P1) and the second prediction candidate (P2).

17. The method according to example 16, wherein Pf=(w1×P1+w2×P2+offset)/(w1+w2), where w1 and w2 are weight values, and where offset=(w1+w2)/2 Is the rounding offset.

18. The method according to example 17, wherein the weight value is based on one of the dimensions of the sub-blocks of the plurality of sub-blocks, the dimensions of the video block, the chroma component of the sub-blocks, one of the video blocks, or Multiple attributes, or the location of the sub-block boundary area.

19. The method of example 18, wherein the one or more attributes include a motion vector, quantization parameter (QP), inter prediction mode, inter prediction direction, Merge mode, or advanced motion vector prediction (AMVP) mode .

20. The method according to Example 16, where Pf=(w1×P1+w2×P2+offset)>>B, where w1 and w2 are weight values, and where offset = (w1+w2)/2 is rounded Shift, and where w1+w2=2B.

21. The method according to example 1, further comprising:

Based on the prediction of the plurality of samples in the sub-block boundary area, one or more additional prediction candidates are formed, wherein the final prediction candidate is also based on the one or more additional prediction candidates.

22. According to the method described in Example 1, in the video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), slice header, coding tree unit (CTU) or coding unit ( CU) signaling the type of the second prediction candidate.

23. The method of example 1, wherein the chroma component of the sub-block includes the plurality of samples in the sub-block boundary area.

24. The method according to example 1, wherein the prediction modes of the first prediction candidate and the second prediction candidate are affine predictions.

25. The method of example 1, wherein the prediction modes of the first prediction candidate and the second prediction candidate are selectable time-domain motion vector prediction (ATMVP).

26. The method according to example 1, wherein the prediction modes of the first prediction candidate and the second prediction candidate are space-time motion vector prediction (STMVP).

27. The method according to example 1, wherein the prediction modes of the first prediction candidate and the second prediction candidate are bidirectional optical flow (BIO).

28. The method according to example 1, wherein the prediction mode of the first prediction candidate and the second prediction candidate is frame play rate up conversion (FRUC).

29. The method according to example 1, wherein the prediction modes of the first prediction candidate and the second prediction candidate are local self-adjusting motion vector resolution (LAMVR).

30. The method according to example 1, wherein the prediction modes of the first prediction candidate and the second prediction candidate are time-domain motion vector prediction (TMVP).

31. The method according to example 1, wherein the prediction modes of the first prediction candidate and the second prediction candidate are overlapping block motion compensation (OBMC).

32. The method according to example 1, wherein the prediction modes of the first prediction candidate and the second prediction candidate are decoder-side motion vector refinement (DMVR).

33. A video encoding device including a processor configured to implement the method of any one of Examples 1 to 32.

34. A video decoding device including a processor configured to implement the method described in any one of Examples 1 to 32.

35. A computer program product stored on a non-volatile computer-readable medium, the computer program product comprising program code for implementing the method of any one of Examples 1 to 32.

From the above, it should be understood that, for the convenience of explanation, specific embodiments of the technology disclosed in the present invention have been described herein, but various modifications can be made without departing from the scope of the present invention. Therefore, in addition to the above, the technology disclosed in the present invention is not limited to the limitation of the scope of patent application.

The implementation and functional operations of the subject matter described in this document can be implemented in various systems, digital electronic circuits, or computer software, firmware or hardware, including the structures disclosed in this specification and their structural equivalents, or One or more combinations. The implementation of the subject matter described in this specification can be implemented as one or more computer program products, ie one or more modules of computer program instructions encoded on a tangible and non-volatile computer readable medium for data processing The device performs or controls the operation of the data processing device. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a storage device, a material composition that affects a machine-readable transmission signal, or a combination of one or more of them. The term "data processing unit" or "data processing device" includes all devices, equipment and machines used to process data, including, for example, programmable processors, computers or multiple processors or computer groups. In addition to the hardware, the device may also include code to create an execution environment for the computer program, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

Computer programs (also known as programs, software, software applications, scripts, or codes) can be written in any form of programming language (including compiled or interpreted languages) and can be deployed in any form, including as standalone programs or as modules , Components, subprograms or other units suitable for use in computing environments. The computer program does not necessarily correspond to the document in the document case system. Programs can be stored in parts of documents that hold other programs or data (for example, one or more scripts stored in markup language documents), in a single file dedicated to the program, or in multiple coordination documents (for example, stored One or more modules, subprograms, or part of the code). Computer programs can be deployed on one or more computers for execution. These computers are located on one website or distributed on multiple websites and are interconnected through a communication network.

The processes and logic flows described in this specification can be performed by one or more programmable processors that execute one or more computer programs and perform functions by operating on input data and generating output. Processing and logic flow can also be performed by special purpose logic circuits, and the device can also be implemented as special purpose logic circuits, such as FPGA (field programmable gate array) or ASIC (dedicated integrated circuit).

For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, and any one or more of any type of digital computer. Typically, the processor will receive commands and data from read-only memory or random access memory or both. The basic components of a computer are a processor that executes instructions and one or more storage devices that store instructions and data. Typically, the computer will also include one or more large storage area devices for storing data, such as magnetic disks, magneto-optical disks, or optical discs, or be operatively coupled to one or more large storage area devices to receive data from or Data is transferred to one or more large storage area equipment, or both. However, computers do not necessarily have such equipment. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices. The processor and memory can be supplemented by dedicated logic circuits or incorporated into dedicated logic circuits.

This description and the drawings are intended to be considered exemplary only, and the examples are indicative examples. As used herein, unless the context clearly indicates otherwise, the singular forms "a", "an", and "this" should also include the plural forms. In addition, the use of "or" is intended to include "and/or" unless the context clearly dictates otherwise.

Although this patent document contains many details, it should not be construed as a limitation on the scope of any invention or patent application, but as a description of features of specific embodiments of specific inventions. Certain features described in this patent document in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various functions that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. In addition, although the above features can be described as functioning in certain combinations, even if initially required, in some cases, one or more features in the combination of patent applications can be deleted from the combination, and the patent scope The combination of can point to sub-combinations or variants of sub-combinations.

Likewise, although the operations are described in a specific order in the drawings, it should not be understood that in order to obtain the desired result, such operations must be performed in the specific order or order shown, or all illustrated operations. Furthermore, the separation of various system elements in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments. Only some implementations and examples are described, and other implementations, enhancements, and variations can be made based on what is described and illustrated in this patent document.

1600, 1900‧‧‧ 1610～1650、1902～1906‧‧‧Step 1700‧‧‧ Computer system 1705‧‧‧ processor 1710‧‧‧Memory 1715‧‧‧Network interface card 1725‧‧‧ Internet 1800‧‧‧Mobile device 1801‧‧‧ processor 1802‧‧‧Memory 1803‧‧‧I/O interface 1804‧‧‧Monitor

FIG. 1 shows an example of subblock-based prediction. Figure 2 shows an example of a simplified affine motion model. Figure 3 shows an example of an affine motion vector field (MVF) for each sub-block. FIG. 4 shows an example of motion vector prediction (MVP) of the AF_INTER affine motion mode. 5A and 5B show example candidates of AF_MERGE affine motion patterns. FIG. 6 shows an example of motion prediction using an optional time-domain motion vector prediction (ATMVP) algorithm of a coding unit (CU). 7 shows an example of a coding unit (CU) with sub-blocks and neighboring blocks used by the space-time motion vector prediction (STMVP) algorithm. FIG. 8 shows an example of the optical flow trajectory used by the bidirectional optical flow (BIO) algorithm. 9A and 9B show an example snapshot of a bidirectional optical flow (BIO) algorithm using blockless expansion. FIG. 10 shows an example of bilateral matching in the Frame Play Rate Up Conversion (FRUC) algorithm. FIG. 11 shows an example of template matching in the FRUC algorithm. Fig. 12 shows an example of unilateral motion estimation in the FRUC algorithm. 13A and 13B show examples of sub-block boundaries that may be filtered. 14A and 14B show examples of prediction samples to be filtered. 15A, 15B, 15C, and 15D show examples of boundary enhancement for affine prediction according to the disclosed technique. 16 shows a flowchart of an example method of video encoding according to the disclosed technology. 17 is a block diagram illustrating an example of the structure of a computer system or other control device that can be used to implement various parts of the disclosed technology. 18 shows a block diagram of an example embodiment of a mobile device that can be used to implement various parts of the disclosed technology. 19 is a flowchart of an example method for video processing.

1900‧‧‧method

1902~1906‧‧‧Step

Claims (35)

A video processing method, including: Split the video block into multiple sub-blocks; The final prediction candidate is formed as a function of a first prediction candidate and a second prediction candidate, the first prediction candidate corresponds to subblock-based prediction of a plurality of samples in a subblock boundary region, and the second prediction candidate corresponds to Inter prediction of the plurality of samples in the sub-block boundary region; and The video block is processed using the final prediction candidate. The method according to item 1 of the patent application scope, wherein the sub-block boundary area includes an inner boundary of the video block. The method according to item 1 of the patent application scope, wherein the sub-block boundary area is an outer boundary of the video. The method according to item 1 of the patent application scope, wherein the plurality of samples in the sub-block boundary area include N rows of samples along the vertical boundary and M columns of samples along the horizontal boundary. The method according to item 4 of the patent application scope, wherein M or N is based on the dimensions of the plurality of sub-blocks. The method as described in item 4 of the patent application range, wherein M or N is based on the type of color components of the sub-blocks of the plurality of sub-blocks. The method according to item 4 of the patent application scope, wherein M or N is based on the position of the sub-block boundary area relative to the video block. The method as described in item 4 of the patent application scope, wherein M or N is based on the positions of sub-blocks of the plurality of sub-blocks. The method as described in item 4 of the patent application, where M or N is in the video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), slice header, and coding tree unit (CTU) ) Or coding unit (CU) is signaled. The method according to item 1 of the patent application scope, wherein the first prediction candidate and the second prediction candidate use the same prediction model, wherein the first prediction candidate is based on a first subset of the plurality of samples , And wherein the second prediction candidate is based on a second subset of the plurality of samples that is different from the first subset. The method according to item 1 or item 10 of the patent application scope, wherein the second prediction candidate is based on a dimension of a sub-block of the plurality of sub-blocks or a dimension of the video block. The method according to item 1 or item 10 of the patent application range, wherein the second prediction candidate is based on color components of sub-blocks of the plurality of sub-blocks. The method according to item 1 of the patent application scope, wherein the subblock-based prediction is the same as the inter prediction. The method according to item 1 of the patent application scope, wherein the first prediction candidate and the second prediction candidate are based on the same reference picture. The method according to item 1 of the patent application scope, wherein the first prediction candidate is based on a first reference picture, and wherein the second prediction candidate is based on a second reference picture different from the first reference picture. The method according to item 1 of the patent application scope, wherein the final prediction candidate (Pf) is a weighted sum of the first prediction candidate (P1) and the second prediction candidate (P2). The method as described in item 16 of the patent scope, where Pf = (w1×P1+w2×P2+offset)/(w1+w2), where w1 and w2 are weight values, and where offset = (w1+w2) /2 is the rounding offset. The method according to item 17 of the patent application scope, wherein the weight value is based on the dimensions of the sub-blocks of the plurality of sub-blocks, the dimensions of the video block, the chroma component of the sub-blocks, the One or more attributes, or the location of the sub-block boundary area. The method of claim 18, wherein the one or more attributes include motion vector, quantization parameter (QP), inter prediction mode, inter prediction direction, Merge mode, or advanced motion vector prediction (AMVP )mode. The method as described in item 16 of the patent application scope, where Pf = (w1×P1+w2×P2+offset)>>B, where w1 and w2 are weight values, where offset = (w1+w2)/2 is taken The entire offset, and where w1+w2=2B. The method as described in item 1 of the patent application scope also includes: Based on the prediction of the plurality of samples in the sub-block boundary area, one or more additional prediction candidates are formed, wherein the final prediction candidate is also based on the one or more additional prediction candidates. According to the method described in item 1 of the patent application scope, in the video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), slice header, slice tree unit (CTU) or coding The unit (CU) signals the type of the second prediction candidate. The method according to item 1 of the patent application scope, wherein the chroma component of the sub-block includes the plurality of samples in the boundary area of the sub-block. The method according to item 1 of the patent application scope, wherein the prediction modes of the first prediction candidate and the second prediction candidate are affine predictions. The method according to item 1 of the patent application scope, wherein the prediction modes of the first prediction candidate and the second prediction candidate are optional time-domain motion vector prediction (ATMVP). The method according to item 1 of the patent application scope, wherein the prediction mode of the first prediction candidate and the second prediction candidate is space-time motion vector prediction (STMVP). The method according to item 1 of the patent application scope, wherein the prediction modes of the first prediction candidate and the second prediction candidate are bidirectional optical flow (BIO). The method according to item 1 of the patent application scope, wherein the prediction mode of the first prediction candidate and the second prediction candidate is frame rate up conversion (FRUC). The method according to item 1 of the patent application scope, wherein the prediction modes of the first prediction candidate and the second prediction candidate are local adaptive motion vector resolution (LAMVR). The method according to item 1 of the patent application scope, wherein the prediction modes of the first prediction candidate and the second prediction candidate are time-domain motion vector prediction (TMVP). The method as described in item 1 of the patent application scope, wherein the prediction modes of the first prediction candidate and the second prediction candidate are overlapping block motion compensation (OBMC). The method as described in item 1 of the patent application scope, wherein the prediction modes of the first prediction candidate and the second prediction candidate are decoder-side motion vector refinement (DMVR). A video encoding device includes a processor configured to implement the method described in any one of claims 1 to 32. A video decoding device includes a processor configured to implement the method described in any one of claims 1 to 32. A computer program product stored on a non-volatile computer readable medium, the computer program product including program code for implementing the method described in any one of the items 1 to 32 of the patent application.

Images