KR20230108215A - Method for Decoder-side Motion Vector List Modification in Inter Prediction - Google Patents

Abstract

Disclosed is a method for modifying a decoder-side motion vector list in inter prediction. The present embodiment provides a video coding method and device. In the merge mode and advanced motion vector prediction (AMVP) mode of inter prediction, based on a template matching method or a two-way matching method, one or more of adding, pruning, and reordering candidates to the motion vector list is performed on a decoder side. The method includes the steps of decoding a candidate index; generating a candidate list; extracting a motion vector pair of a current block; and generating a prediction block of the current block.

Description

Method for modifying decoder-side motion vector list in inter prediction {Method for Decoder-side Motion Vector List Modification in Inter Prediction}

The present disclosure relates to a method for modifying a motion vector list at a decoder side in inter prediction.

The contents described below merely provide background information related to the present embodiment and do not constitute prior art.

Since video data has a large amount of data compared to audio data or still image data, it requires a lot of hardware resources including memory to store or transmit itself without processing for compression.

Therefore, when video data is stored or transmitted, an encoder is used to compress and store or transmit the video data, and a decoder receives, decompresses, and reproduces the compressed video data. Examples of such video compression technologies include H.264/AVC, High Efficiency Video Coding (HEVC), and Versatile Video Coding (VVC), which has improved coding efficiency by about 30% or more compared to HEVC.

However, since the size, resolution, and frame rate of video are gradually increasing, and the amount of data to be encoded accordingly increases, a new compression technology with higher encoding efficiency and higher picture quality improvement effect than existing compression technologies is required. In particular, in inter prediction, efficient operation of motion vector lists needs to be considered in order to improve video encoding efficiency and quality.

In the present disclosure, in a merge mode and an advanced motion vector prediction (AMVP) mode of inter prediction, candidates are added, pruned, and rearranged to a motion vector list at a decoder side based on a template matching method or a bidirectional matching method. An object of the present invention is to provide a video coding method and apparatus for performing one or more of the above.

According to an embodiment of the present disclosure, in a method of inter-predicting a current block performed by a video decoding apparatus, decoding a candidate index from a bitstream, wherein the candidate index selects one of a plurality of candidates in a candidate list. Indicates, and each candidate represents a pair of motion vectors of bi-directional prediction; generating the candidate list using neighboring information of the current block; Modifying the candidate list based on multiple passes of MBM (Multi-pass Bilateral Matching) and an MBM cost, wherein a first pass among the multiple passes is the current block, and a second pass is a subroutine within the current block. block, and the third pass searches motion vector pairs for sub-blocks having a smaller size than the sub-blocks based on the MBM cost, and the MBM cost is indicated by the searched motion vector pair for each pass. depends on the difference between the two blocks; extracting a motion vector pair of the current block from the modified candidate list using the candidate index; and generating a prediction block of the current block using the extracted motion vector pair.

According to another embodiment of the present disclosure, in a method of inter-predicting a current block performed by an image encoding apparatus, determining a candidate index, wherein the candidate index indicates one of a plurality of candidates in a candidate list, , each candidate represents a motion vector pair of bidirectional prediction; generating the candidate list using neighboring information of the current block; Modifying the candidate list based on multiple passes of MBM (Multi-pass Bilateral Matching) and an MBM cost, wherein a first pass among the multiple passes is the current block, and a second pass is a subroutine within the current block. block, and the third pass searches motion vector pairs for sub-blocks having a smaller size than the sub-blocks based on the MBM cost, and the MBM cost is indicated by the searched motion vector pair for each pass. depends on the difference between the two blocks; extracting a motion vector pair of the current block from the modified candidate list using the candidate index; generating a prediction block of the current block using the extracted motion vector pair; and encoding the candidate index.

According to another embodiment of the present disclosure, a computer-readable recording medium storing a bitstream generated by an image encoding method, the image encoding method comprising: determining a candidate index, wherein the candidate index is in a candidate list Indicates one of a plurality of candidates, and each candidate represents a motion vector pair of bi-directional prediction; generating the candidate list using neighboring information of the current block; Modifying the candidate list based on multiple passes of MBM (Multi-pass Bilateral Matching) and an MBM cost, wherein a first pass among the multiple passes is the current block, and a second pass is a subroutine within the current block. block, and the third pass searches motion vector pairs for sub-blocks having a smaller size than the sub-blocks based on the MBM cost, and the MBM cost is indicated by the searched motion vector pair for each pass. depends on the difference between the two blocks; extracting a motion vector pair of the current block from the modified candidate list using the candidate index; generating a prediction block of the current block using the extracted motion vector pair; and encoding the candidate index.

As described above, according to the present embodiment, in the merge mode and the AMVP mode of inter prediction, one of adding, pruning, and rearranging candidates to the motion vector list at the decoder side based on the template matching method or the bidirectional matching method. By providing a video coding method and apparatus that perform the above, there is an effect of improving video coding efficiency and improving video quality.

1 is an exemplary block diagram of an image encoding apparatus capable of implementing the techniques of this disclosure.
2 is a diagram for explaining a method of dividing a block using a QTBTTT structure.
3A and 3B are diagrams illustrating a plurality of intra prediction modes including wide-angle intra prediction modes.
4 is an exemplary diagram of neighboring blocks of a current block.
5 is an exemplary block diagram of a video decoding apparatus capable of implementing the techniques of this disclosure.
6A and 6B are exemplary diagrams for explaining affine motion prediction according to an embodiment of the present disclosure.
7 is an exemplary diagram for explaining a method of deriving combinatorial affine merge candidates for affine motion prediction.
8 is a flowchart illustrating a process of searching for an affine AMVP candidate in an affine AMVP mode according to an embodiment of the present disclosure.
9 is an exemplary diagram illustrating triangulation types supported in a geometric segmentation mode according to an embodiment of the present disclosure.
10 is an exemplary diagram illustrating weights used in a geometric segmentation mode according to an embodiment of the present disclosure.
11 is an exemplary diagram illustrating a GPM candidate list according to an embodiment of the present disclosure.
12 is an exemplary diagram illustrating template matching in intra prediction according to an embodiment of the present disclosure.
13 is an exemplary diagram illustrating template matching in bi-directional prediction according to an embodiment of the present disclosure.
14 is an exemplary diagram illustrating a bidirectional matching AMVP-MERGE mode according to an embodiment of the present disclosure.
15 is an exemplary diagram illustrating neighboring blocks for deriving a motion information candidate list according to an embodiment of the present disclosure.
16 is an exemplary diagram illustrating neighboring blocks for deriving spatial candidates according to an embodiment of the present disclosure.
17 is a flowchart illustrating a process of searching for an affine AMVP candidate, including rearrangement of inherited affine AMVP candidates, according to an embodiment of the present disclosure.
18 is a flowchart illustrating a method of inter-predicting a current block by an image encoding apparatus according to an embodiment of the present disclosure.
19 is a flowchart illustrating a method of inter-predicting a current block by an image decoding apparatus according to an embodiment of the present disclosure.
20 is a flowchart illustrating a method of inter-predicting a current block by an image encoding apparatus according to another embodiment of the present disclosure.
21 is a flowchart illustrating a method of inter-predicting a current block by an image decoding apparatus according to another embodiment of the present disclosure.

DETAILED DESCRIPTION Some embodiments of the present disclosure are described in detail below with reference to exemplary drawings. In adding reference numerals to components of each drawing, it should be noted that the same components have the same numerals as much as possible even if they are displayed on different drawings. In addition, in describing the present embodiments, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present embodiments, the detailed description will be omitted.

1 is an exemplary block diagram of an image encoding apparatus capable of implementing the techniques of this disclosure. Hereinafter, with reference to the illustration of FIG. 1, an image encoding device and sub-components of the device will be described.

The image encoding apparatus includes a picture division unit 110, a prediction unit 120, a subtractor 130, a transform unit 140, a quantization unit 145, a rearrangement unit 150, an entropy encoding unit 155, and an inverse quantization unit. 160, an inverse transform unit 165, an adder 170, a loop filter unit 180, and a memory 190.

Each component of the image encoding device may be implemented as hardware or software, or as a combination of hardware and software. Also, the function of each component may be implemented as software, and the microprocessor may be implemented to execute the software function corresponding to each component.

One image (video) is composed of one or more sequences including a plurality of pictures. Each picture is divided into a plurality of areas and encoding is performed for each area. For example, one picture is divided into one or more tiles and/or slices. Here, one or more tiles may be defined as a tile group. Each tile or/slice is divided into one or more Coding Tree Units (CTUs). And each CTU is divided into one or more CUs (Coding Units) by a tree structure. Information applied to each CU is coded as a CU syntax, and information commonly applied to CUs included in one CTU is coded as a CTU syntax. In addition, information commonly applied to all blocks in one slice is coded as syntax of a slice header, and information applied to all blocks constituting one or more pictures is a picture parameter set (PPS) or picture coded in the header. Furthermore, information commonly referred to by a plurality of pictures is coded into a Sequence Parameter Set (SPS). Also, information commonly referred to by one or more SPSs is coded into a video parameter set (VPS). Also, information commonly applied to one tile or tile group may be encoded as syntax of a tile or tile group header. Syntax included in the SPS, PPS, slice header, tile or tile group header may be referred to as high level syntax.

The picture divider 110 determines the size of a coding tree unit (CTU). Information on the size of the CTU (CTU size) is encoded as SPS or PPS syntax and transmitted to the video decoding apparatus.

The picture division unit 110 divides each picture constituting an image into a plurality of Coding Tree Units (CTUs) having a predetermined size, and then iteratively divides the CTUs using a tree structure. Divide (recursively). A leaf node in the tree structure becomes a coding unit (CU), which is a basic unit of encoding.

As a tree structure, a quad tree (QT) in which a parent node (or parent node) is divided into four subnodes (or child nodes) of the same size, or a binary tree (BinaryTree) in which a parent node is divided into two subnodes , BT), or a TernaryTree (TT) in which a parent node is split into three subnodes at a ratio of 1:2:1, or a structure in which two or more of these QT structures, BT structures, and TT structures are mixed. there is. For example, a QuadTree plus BinaryTree (QTBT) structure may be used, or a QuadTree plus BinaryTree TernaryTree (QTBTTT) structure may be used. Here, BTTT may be combined to be referred to as MTT (Multiple-Type Tree).

2 is a diagram for explaining a method of dividing a block using a QTBTTT structure.

As shown in FIG. 2, the CTU may first be divided into QT structures. Quadtree splitting can be repeated until the size of the splitting block reaches the minimum block size (MinQTSize) of leaf nodes allowed by QT. A first flag (QT_split_flag) indicating whether each node of the QT structure is split into four nodes of a lower layer is encoded by the entropy encoder 155 and signaled to the video decoding device. If the leaf node of QT is not larger than the maximum block size (MaxBTSize) of the root node allowed in BT, it may be further divided into either a BT structure or a TT structure. A plurality of division directions may exist in the BT structure and/or the TT structure. For example, there may be two directions in which blocks of the corresponding node are divided horizontally and vertically. As shown in FIG. 2, when MTT splitting starts, a second flag (mtt_split_flag) indicating whether nodes are split, and if split, a flag indicating additional split direction (vertical or horizontal) and/or split type (Binary or Ternary) is encoded by the entropy encoding unit 155 and signaled to the video decoding apparatus.

Alternatively, prior to coding the first flag (QT_split_flag) indicating whether each node is split into four nodes of a lower layer, a CU split flag (split_cu_flag) indicating whether the node is split is coded. It could be. When the value of the CU split flag (split_cu_flag) indicates that it is not split, the block of the corresponding node becomes a leaf node in the split tree structure and becomes a coding unit (CU), which is a basic unit of encoding. When the value of the CU split flag (split_cu_flag) indicates splitting, the video encoding apparatus starts encoding from the first flag in the above-described manner.

As another example of a tree structure, when QTBT is used, the block of the corresponding node is divided into two blocks of the same size horizontally (i.e., symmetric horizontal splitting) and the type that splits vertically (i.e., symmetric vertical splitting). Branches may exist. A split flag (split_flag) indicating whether each node of the BT structure is split into blocks of a lower layer and split type information indicating a split type are encoded by the entropy encoder 155 and transmitted to the video decoding device. Meanwhile, a type in which a block of a corresponding node is divided into two blocks having an asymmetric shape may additionally exist. The asymmetric form may include a form in which the block of the corresponding node is divided into two rectangular blocks having a size ratio of 1:3, or a form in which the block of the corresponding node is divided in a diagonal direction may be included.

A CU can have various sizes depending on the QTBT or QTBTTT split from the CTU. Hereinafter, a block corresponding to a CU to be encoded or decoded (ie, a leaf node of QTBTTT) is referred to as a 'current block'. Depending on the adoption of the QTBTTT division, the shape of the current block may be rectangular as well as square.

The prediction unit 120 predicts a current block and generates a prediction block. The prediction unit 120 includes an intra prediction unit 122 and an inter prediction unit 124 .

In general, each current block in a picture can be coded predictively. In general, prediction of a current block uses an intra-prediction technique (using data from a picture containing the current block) or an inter-prediction technique (using data from a picture coded before the picture containing the current block). can be performed Inter prediction includes both uni-prediction and bi-prediction.

The intra predictor 122 predicts pixels in the current block using pixels (reference pixels) located around the current block in the current picture including the current block. A plurality of intra prediction modes exist according to the prediction direction. For example, as shown in FIG. 3A, the plurality of intra prediction modes may include two non-directional modes including a planar mode and a DC mode and 65 directional modes. Depending on each prediction mode, the neighboring pixels to be used and the arithmetic expression are defined differently.

For efficient directional prediction of the rectangular current block, directional modes (numbers 67 to 80 and -1 to -14 intra prediction modes) indicated by dotted arrows in FIG. 3B may be additionally used. These may be referred to as “wide angle intra-prediction modes”. In FIG. 3B , arrows indicate corresponding reference samples used for prediction and do not indicate prediction directions. The prediction direction is opposite to the direction the arrow is pointing. Wide-angle intra prediction modes are modes that perform prediction in the opposite direction of a specific directional mode without additional bit transmission when the current block is rectangular. At this time, among the wide-angle intra prediction modes, some wide-angle intra prediction modes usable for the current block may be determined by the ratio of the width and height of the rectangular current block. For example, wide-angle intra prediction modes (67 to 80 intra prediction modes) having an angle smaller than 45 degrees are usable when the current block has a rectangular shape with a height smaller than a width, and a wide angle having an angle greater than -135 degrees. Intra prediction modes (-1 to -14 intra prediction modes) are available when the current block has a rectangular shape where the width is greater than the height.

The intra prediction unit 122 may determine an intra prediction mode to be used for encoding the current block. In some examples, the intra prediction unit 122 may encode the current block using several intra prediction modes and select an appropriate intra prediction mode to be used from the tested modes. For example, the intra predictor 122 calculates rate-distortion values using rate-distortion analysis for several tested intra-prediction modes, and has the best rate-distortion characteristics among the tested modes. Intra prediction mode can also be selected.

The intra prediction unit 122 selects one intra prediction mode from among a plurality of intra prediction modes, and predicts a current block using neighboring pixels (reference pixels) determined according to the selected intra prediction mode and an arithmetic expression. Information on the selected intra prediction mode is encoded by the entropy encoder 155 and transmitted to the video decoding apparatus.

The inter prediction unit 124 generates a prediction block for a current block using a motion compensation process. The inter-prediction unit 124 searches for a block most similar to the current block in the encoded and decoded reference picture prior to the current picture, and generates a prediction block for the current block using the searched block. Then, a motion vector (MV) corresponding to displacement between the current block in the current picture and the prediction block in the reference picture is generated. In general, motion estimation is performed on a luma component, and a motion vector calculated based on the luma component is used for both the luma component and the chroma component. Motion information including reference picture information and motion vector information used to predict the current block is encoded by the entropy encoding unit 155 and transmitted to the video decoding apparatus.

The inter-prediction unit 124 may perform interpolation on a reference picture or reference block in order to increase prediction accuracy. That is, subsamples between two consecutive integer samples are interpolated by applying filter coefficients to a plurality of consecutive integer samples including the two integer samples. When a process of searching for a block most similar to the current block is performed for the interpolated reference picture, the motion vector can be expressed with precision of decimal units instead of integer sample units. The precision or resolution of the motion vector may be set differently for each unit of a target region to be encoded, for example, a slice, tile, CTU, or CU. When such adaptive motion vector resolution (AMVR) is applied, information on motion vector resolution to be applied to each target region must be signaled for each target region. For example, when the target region is a CU, information on motion vector resolution applied to each CU is signaled. Information on the motion vector resolution may be information indicating the precision of differential motion vectors, which will be described later.

Meanwhile, the inter prediction unit 124 may perform inter prediction using bi-prediction. In the case of bi-directional prediction, two reference pictures and two motion vectors representing positions of blocks most similar to the current block within each reference picture are used. The inter prediction unit 124 selects a first reference picture and a second reference picture from reference picture list 0 (RefPicList0) and reference picture list 1 (RefPicList1), respectively, and searches for a block similar to the current block within each reference picture. A first reference block and a second reference block are generated. Then, a prediction block for the current block is generated by averaging or weighted averaging the first reference block and the second reference block. Further, motion information including information on two reference pictures used to predict the current block and information on two motion vectors is delivered to the encoder 150. Here, reference picture list 0 may include pictures prior to the current picture in display order among restored pictures, and reference picture list 1 may include pictures after the current picture in display order among restored pictures. there is. However, it is not necessarily limited to this, and in order of display, ups and downs pictures subsequent to the current picture may be additionally included in reference picture list 0, and conversely, ups and downs pictures prior to the current picture may be additionally included in reference picture list 1. may also be included.

Various methods may be used to minimize the amount of bits required to encode motion information.

For example, when the reference picture and motion vector of the current block are the same as those of the neighboring block, the motion information of the current block can be delivered to the video decoding apparatus by encoding information capable of identifying the neighboring block. This method is called 'merge mode'.

In the merge mode, the inter prediction unit 124 selects a predetermined number of merge candidate blocks (hereinafter referred to as 'merge candidates') from neighboring blocks of the current block.

Neighboring blocks for deriving merge candidates include a left block (A0), a lower left block (A1), an upper block (B0), and an upper right block (B1) adjacent to the current block in the current picture, as shown in FIG. ), and all or part of the upper left block A2 may be used. Also, a block located in a reference picture (which may be the same as or different from a reference picture used to predict the current block) other than the current picture in which the current block is located may be used as a merge candidate. For example, a block co-located with the current block in the reference picture or blocks adjacent to the co-located block may be additionally used as a merge candidate. If the number of merge candidates selected by the method described above is less than the preset number, a 0 vector is added to the merge candidates.

The inter prediction unit 124 constructs a merge list including a predetermined number of merge candidates using these neighboring blocks. Among the merge candidates included in the merge list, a merge candidate to be used as motion information of the current block is selected, and merge index information for identifying the selected candidate is generated. The generated merge index information is encoded by the encoder 150 and transmitted to the video decoding apparatus.

Merge skip mode is a special case of merge mode. After performing quantization, when all transform coefficients for entropy encoding are close to zero, only neighboring block selection information is transmitted without transmitting residual signals. By using the merge skip mode, it is possible to achieve a relatively high encoding efficiency in low-motion images, still images, screen content images, and the like.

Hereinafter, merge mode and merge skip mode are collectively referred to as merge/skip mode.

Another method for encoding motion information is Advanced Motion Vector Prediction (AMVP) mode.

In the AMVP mode, the inter prediction unit 124 derives predictive motion vector candidates for the motion vector of the current block using neighboring blocks of the current block. Neighboring blocks used to derive predictive motion vector candidates include a left block A0, a lower left block A1, an upper block B0, and an upper right block adjacent to the current block in the current picture shown in FIG. B1), and all or part of the upper left block (A2) may be used. In addition, a block located in a reference picture (which may be the same as or different from the reference picture used to predict the current block) other than the current picture where the current block is located will be used as a neighboring block used to derive motion vector candidates. may be For example, a collocated block co-located with the current block within the reference picture or blocks adjacent to the collocated block may be used. If the number of motion vector candidates is smaller than the preset number according to the method described above, a 0 vector is added to the motion vector candidates.

The inter-prediction unit 124 derives predicted motion vector candidates using the motion vectors of the neighboring blocks, and determines a predicted motion vector for the motion vector of the current block using the predicted motion vector candidates. Then, a differential motion vector is calculated by subtracting the predicted motion vector from the motion vector of the current block.

The predicted motion vector may be obtained by applying a predefined function (eg, median value, average value operation, etc.) to predicted motion vector candidates. In this case, the video decoding apparatus also knows the predefined function. In addition, since a neighboring block used to derive a predicted motion vector candidate is a block that has already been encoded and decoded, the video decoding apparatus also knows the motion vector of the neighboring block. Therefore, the video encoding apparatus does not need to encode information for identifying a predictive motion vector candidate. Therefore, in this case, information on differential motion vectors and information on reference pictures used to predict the current block are encoded.

Meanwhile, the predicted motion vector may be determined by selecting one of the predicted motion vector candidates. In this case, along with information on differential motion vectors and information on reference pictures used to predict the current block, information for identifying the selected predictive motion vector candidate is additionally encoded.

The subtractor 130 subtracts the prediction block generated by the intra prediction unit 122 or the inter prediction unit 124 from the current block to generate a residual block.

The transform unit 140 transforms the residual signal in the residual block having pixel values in the spatial domain into transform coefficients in the frequency domain. The transform unit 140 may transform residual signals in the residual block by using the entire size of the residual block as a transform unit, or divide the residual block into a plurality of subblocks and use the subblocks as a transform unit to perform transformation. You may. Alternatively, the residual signals may be divided into two subblocks, a transform region and a non-transform region, and transform the residual signals using only the transform region subblock as a transform unit. Here, the transformation region subblock may be one of two rectangular blocks having a size ratio of 1:1 based on a horizontal axis (or a vertical axis). In this case, a flag (cu_sbt_flag) indicating that only subblocks have been transformed, directional (vertical/horizontal) information (cu_sbt_horizontal_flag), and/or location information (cu_sbt_pos_flag) are encoded by the entropy encoding unit 155 and signaled to the video decoding device. do. In addition, the size of the transform region subblock may have a size ratio of 1:3 based on the horizontal axis (or vertical axis), and in this case, a flag (cu_sbt_quad_flag) for distinguishing the corresponding division is additionally encoded by the entropy encoder 155 to obtain an image It is signaled to the decryption device.

Meanwhile, the transform unit 140 may individually transform the residual block in the horizontal direction and the vertical direction. For the transformation, various types of transformation functions or transformation matrices may be used. For example, a pair of transformation functions for horizontal transformation and vertical transformation may be defined as a multiple transform set (MTS). The transform unit 140 may select one transform function pair having the highest transform efficiency among the MTS and transform the residual blocks in the horizontal and vertical directions, respectively. Information (mts_idx) on a pair of transform functions selected from the MTS is encoded by the entropy encoding unit 155 and signaled to the video decoding device.

The quantization unit 145 quantizes transform coefficients output from the transform unit 140 using a quantization parameter, and outputs the quantized transform coefficients to the entropy encoding unit 155 . The quantization unit 145 may directly quantize a related residual block without transformation for a certain block or frame. The quantization unit 145 may apply different quantization coefficients (scaling values) according to positions of transform coefficients in the transform block. A quantization matrix applied to the two-dimensionally arranged quantized transform coefficients may be coded and signaled to the video decoding apparatus.

The rearrangement unit 150 may rearrange the coefficient values of the quantized residual values.

The reordering unit 150 may change a 2D coefficient array into a 1D coefficient sequence using coefficient scanning. For example, the reordering unit 150 may output a one-dimensional coefficient sequence by scanning DC coefficients to coefficients in a high frequency region using a zig-zag scan or a diagonal scan. . Depending on the size of the transformation unit and intra prediction mode, instead of zig-zag scan, vertical scan that scans a 2D coefficient array in a column direction and horizontal scan that scans 2D block-shaped coefficients in a row direction may be used. That is, a scan method to be used among zig-zag scan, diagonal scan, vertical scan, and horizontal scan may be determined according to the size of the transform unit and the intra prediction mode.

The entropy encoding unit 155 uses various encoding schemes such as CABAC (Context-based Adaptive Binary Arithmetic Code) and Exponential Golomb to convert the one-dimensional quantized transform coefficients output from the reordering unit 150 to each other. A bitstream is created by encoding the sequence.

In addition, the entropy encoding unit 155 encodes information such as CTU size, CU splitting flag, QT splitting flag, MTT splitting type, and MTT splitting direction related to block splitting so that the video decoding apparatus can divide the block in the same way as the video encoding apparatus. make it possible to divide In addition, the entropy encoding unit 155 encodes information about a prediction type indicating whether the current block is encoded by intra prediction or inter prediction, and encodes intra prediction information (ie, intra prediction) according to the prediction type. mode) or inter prediction information (motion information encoding mode (merge mode or AMVP mode), merge index in case of merge mode, reference picture index and differential motion vector information in case of AMVP mode) are encoded. Also, the entropy encoding unit 155 encodes information related to quantization, that is, information about quantization parameters and information about quantization matrices.

The inverse quantization unit 160 inversely quantizes the quantized transform coefficients output from the quantization unit 145 to generate transform coefficients. The inverse transform unit 165 transforms transform coefficients output from the inverse quantization unit 160 from a frequency domain to a spatial domain to restore a residual block.

The adder 170 restores the current block by adding the restored residual block and the predicted block generated by the predictor 120. Pixels in the reconstructed current block are used as reference pixels when intra-predicting the next block.

The loop filter unit 180 reconstructs pixels in order to reduce blocking artifacts, ringing artifacts, blurring artifacts, etc. caused by block-based prediction and transformation/quantization. perform filtering on The filter unit 180 is an in-loop filter and may include all or part of a deblocking filter 182, a sample adaptive offset (SAO) filter 184, and an adaptive loop filter (ALF) 186. .

The deblocking filter 182 filters the boundary between reconstructed blocks to remove blocking artifacts caused by block-by-block encoding/decoding, and the SAO filter 184 and alf 186 perform deblocking filtering. Additional filtering is performed on the image. The SAO filter 184 and the alf 186 are filters used to compensate for a difference between a reconstructed pixel and an original pixel caused by lossy coding. The SAO filter 184 improves not only subjective picture quality but also coding efficiency by applying an offset in units of CTUs. In contrast, the ALF 186 performs block-by-block filtering. Distortion is compensated for by applying different filters by distinguishing the edge of the corresponding block and the degree of change. Information on filter coefficients to be used for ALF may be coded and signaled to the video decoding apparatus.

The reconstruction block filtered through the deblocking filter 182, the SAO filter 184, and the ALF 186 is stored in the memory 190. When all blocks in one picture are reconstructed, the reconstructed picture can be used as a reference picture for inter-prediction of blocks in the picture to be encoded later.

5 is an exemplary block diagram of a video decoding apparatus capable of implementing the techniques of this disclosure. Hereinafter, referring to FIG. 5, a video decoding device and sub-elements of the device will be described.

The image decoding apparatus includes an entropy decoding unit 510, a rearrangement unit 515, an inverse quantization unit 520, an inverse transform unit 530, a prediction unit 540, an adder 550, a loop filter unit 560, and a memory ( 570) may be configured.

Like the image encoding device of FIG. 1 , each component of the image decoding device may be implemented as hardware or software, or a combination of hardware and software. Also, the function of each component may be implemented as software, and the microprocessor may be implemented to execute the software function corresponding to each component.

The entropy decoding unit 510 determines a current block to be decoded by extracting information related to block division by decoding the bitstream generated by the video encoding apparatus, and provides prediction information and residual signals necessary for restoring the current block. extract information, etc.

The entropy decoding unit 510 determines the size of the CTU by extracting information about the CTU size from a sequence parameter set (SPS) or a picture parameter set (PPS), and divides the picture into CTUs of the determined size. Then, the CTU is divided using the tree structure by determining the CTU as the top layer of the tree structure, that is, the root node, and extracting division information for the CTU.

For example, when a CTU is split using the QTBTTT structure, a first flag (QT_split_flag) related to splitting of QT is first extracted and each node is split into four nodes of a lower layer. In addition, for a node corresponding to a leaf node of QT, a second flag (MTT_split_flag) related to splitting of MTT and split direction (vertical / horizontal) and / or split type (binary / ternary) information are extracted and the corresponding leaf node is MTT split into structures Accordingly, each node below the leaf node of QT is recursively divided into a BT or TT structure.

As another example, when a CTU is split using the QTBTTT structure, a CU split flag (split_cu_flag) indicating whether the CU is split is first extracted, and when the corresponding block is split, a first flag (QT_split_flag) is extracted. may be During the splitting process, each node may have zero or more iterative MTT splits after zero or more repetitive QT splits. For example, the CTU may immediately undergo MTT splitting, or conversely, only QT splitting may occur multiple times.

As another example, when a CTU is split using a QTBT structure, a first flag (QT_split_flag) related to QT splitting is extracted and each node is split into four nodes of a lower layer. And, for a node corresponding to a leaf node of QT, a split flag (split_flag) indicating whether to further split into BTs and split direction information are extracted.

Meanwhile, when the entropy decoding unit 510 determines a current block to be decoded by using tree structure partitioning, it extracts information about a prediction type indicating whether the current block is intra-predicted or inter-predicted. When the prediction type information indicates intra prediction, the entropy decoding unit 510 extracts syntax elements for intra prediction information (intra prediction mode) of the current block. When the prediction type information indicates inter prediction, the entropy decoding unit 510 extracts syntax elements for the inter prediction information, that is, information indicating a motion vector and a reference picture to which the motion vector refers.

In addition, the entropy decoding unit 510 extracts quantization-related information and information about quantized transform coefficients of the current block as information about the residual signal.

The reordering unit 515 converts the sequence of 1-dimensional quantized transform coefficients entropy-decoded in the entropy decoding unit 510 into a 2-dimensional coefficient array (ie, in the reverse order of the coefficient scanning performed by the image encoding apparatus). block) can be changed.

The inverse quantization unit 520 inverse quantizes the quantized transform coefficients and inverse quantizes the quantized transform coefficients using a quantization parameter. The inverse quantization unit 520 may apply different quantization coefficients (scaling values) to the two-dimensionally arranged quantized transform coefficients. The inverse quantization unit 520 may perform inverse quantization by applying a matrix of quantization coefficients (scaling values) from the image encoding device to a 2D array of quantized transformation coefficients.

The inverse transform unit 530 inversely transforms the inverse quantized transform coefficients from the frequency domain to the spatial domain to restore residual signals, thereby generating a residual block for the current block.

In addition, when the inverse transform unit 530 inverse transforms only a partial region (subblock) of a transform block, a flag (cu_sbt_flag) indicating that only a subblock of the transform block has been transformed, and direction information (vertical/horizontal) information (cu_sbt_horizontal_flag) of the transform block ) and/or the location information (cu_sbt_pos_flag) of the subblock, and inversely transforms the transform coefficients of the corresponding subblock from the frequency domain to the spatial domain to restore the residual signals. By filling , the final residual block for the current block is created.

In addition, when the MTS is applied, the inverse transform unit 530 determines transform functions or transform matrices to be applied in the horizontal and vertical directions, respectively, using MTS information (mts_idx) signaled from the video encoding device, and uses the determined transform functions. Inverse transform is performed on the transform coefficients in the transform block in the horizontal and vertical directions.

The prediction unit 540 may include an intra prediction unit 542 and an inter prediction unit 544 . The intra prediction unit 542 is activated when the prediction type of the current block is intra prediction, and the inter prediction unit 544 is activated when the prediction type of the current block is inter prediction.

The intra prediction unit 542 determines the intra prediction mode of the current block among a plurality of intra prediction modes from the syntax element for the intra prediction mode extracted from the entropy decoding unit 510, and references the current block according to the intra prediction mode. The current block is predicted using pixels.

The inter prediction unit 544 determines the motion vector of the current block and the reference picture referred to by the motion vector by using the syntax element for the inter prediction mode extracted from the entropy decoding unit 510, and converts the motion vector and the reference picture. to predict the current block.

The adder 550 restores the current block by adding the residual block output from the inverse transform unit and the prediction block output from the inter prediction unit or intra prediction unit. Pixels in the reconstructed current block are used as reference pixels when intra-predicting a block to be decoded later.

The loop filter unit 560 may include a deblocking filter 562, an SAO filter 564, and an ALF 566 as in-loop filters. The deblocking filter 562 performs deblocking filtering on boundaries between reconstructed blocks in order to remove blocking artifacts generated by block-by-block decoding. The SAO filter 564 and the ALF 566 perform additional filtering on the reconstructed block after deblocking filtering to compensate for the difference between the reconstructed pixel and the original pixel caused by lossy coding. ALF filter coefficients are determined using information on filter coefficients decoded from the non-stream.

The reconstruction block filtered through the deblocking filter 562, the SAO filter 564, and the ALF 566 is stored in the memory 570. When all blocks in one picture are reconstructed, the reconstructed picture is used as a reference picture for inter-prediction of blocks in the picture to be encoded later.

This embodiment relates to encoding and decoding of images (video) as described above. More specifically, in the merge mode and advanced motion vector prediction (AMVP) mode of inter prediction, candidates are added, pruned, and rearranged to the motion vector list at the decoder side based on the template matching method or the bidirectional matching method. Provides a video coding method and apparatus for performing one or more of the following.

The following embodiments may be performed by the inter prediction unit 124 in a video encoding device. In addition, it may be performed by the inter prediction unit 544 in the video decoding device.

The video encoding apparatus may generate signaling information related to the present embodiment in terms of bit rate distortion optimization in encoding of the current block. The image encoding device may encode the image using the entropy encoding unit 155 and transmit it to the image decoding device. The video decoding apparatus may decode signaling information related to decoding of the current block from the bitstream using the entropy decoding unit 510 .

In the following description, the term 'target block' may be used in the same meaning as a current block or a coding unit (CU, Coding Unit), or may mean a partial region of a coding unit.

Also, a value of one flag being true indicates a case in which the flag is set to 1. In addition, a false value of one flag indicates a case in which the flag is set to 0.

I. Intra Block Copy (IBC)

IBC performs intra prediction of a current block by generating a prediction block of the current block by copying a reference block within the same frame using a block vector.

An image encoding apparatus derives an optimal block vector by performing block matching. Here, the block vector represents displacement from the current block to the reference block. In order to increase encoding efficiency, the video encoding apparatus does not transmit the block vector as it is, but divides it into a block vector predictor (BVP) and a block vector difference (BVD), encodes them, It can be transmitted to the video decoding device.

Hereinafter, the spatial resolution of the BVD and the spatial resolution of the block vector are regarded as the same.

In terms of using a block vector, IBC has the characteristics of inter prediction. Accordingly, IBC can be divided into an IBC merge/skip mode and an IBC AMVP mode.

In the case of the IBC merge/skip mode, the video encoding apparatus first constructs an IBC merge list. In terms of encoding efficiency optimization, the video encoding apparatus may select one block vector from among candidates included in the IBC merge list and use it as a block vector predictor (BVP). The video encoding apparatus determines a merge index indicating the selected block vector. However, the video encoding device does not generate BVD. The video encoding apparatus encodes the merge index and transmits it to the video decoding apparatus. The IBC merge list can be constructed in the same way by the video encoding device and the video decoding device. After decoding the merge index, the video decoding apparatus may generate a block vector from the IBC merge list using the merge index.

In the case of the IBC skip mode, the video encoding apparatus uses the same block vector transmission method as the IBC merge mode, but does not transmit a residual block corresponding to a difference between the current block and the prediction block.

In the case of the IBC AMVP mode, in terms of encoding efficiency optimization, the video encoding apparatus determines a motion vector and constructs an IBC AMVP list. The video decoding apparatus determines a candidate index indicating one of the candidate block vectors included in the IBC AMVP list as a BVP. The video encoding apparatus calculates BVD, which is a difference between BVP and motion vector. Thereafter, the video encoding apparatus encodes the candidate index and the BVD and transmits them to the video decoding apparatus.

Meanwhile, the video decoding apparatus decodes the candidate index and the BVD. After obtaining the BVP indicated by the candidate index from the IBC AMVP list, the video decoding apparatus may restore the motion vector by adding the BVP and the BVD.

The following inter prediction techniques are used to improve coding efficiency and improve inter prediction accuracy. These techniques are performed by the inter prediction unit 124 in the video encoding apparatus, but may also be performed by the inter prediction unit 544 in the video decoding apparatus as described above.

II-1. Merge/Skip mode, MMVD, AMVP mode and AMVR

The merge/skip mode includes a regular merge mode, a Merge mode with Motion Vector Difference (MMVD) mode, a Combined Inter and Intra Prediction (CIIP) mode, a Geometric Partitioning Mode (GPM), and a subblock merge ( subblock merge) mode. At this time, the subblock merge mode is divided into a subblock-based temporal motion vector prediction (SbTMVP) and an affine merge mode.

Meanwhile, the advanced motion vector prediction (AMVP) mode includes a regular AMVP (regular AMVP) mode, a symmetric MVD (SMVD) mode, and an affine AMVP mode.

Hereinafter, a method of constructing a merge candidate list of motion information in a normal merge/skip mode will be described. In order to support the merge/skip mode, the inter prediction unit 124 in the video encoding apparatus may configure a merge candidate list by selecting a preset number (eg, 6) of merge candidates.

The inter prediction unit 124 searches for spatial merge candidates. The inter prediction unit 124 searches for spatial merge candidates from neighboring blocks as illustrated in FIG. 4 . Up to four spatial merge candidates can be selected. Spatial merge candidates are also referred to as Spatial MVPs (SMVPs).

The inter prediction unit 124 searches for a temporal merge candidate. The inter-prediction unit 124 includes a block (co-located at the same position as the current block) in a reference picture (which may be the same as or different from the reference picture used to predict the current block) other than the current picture where the target block is located. located block) can be added as a temporal merge candidate. One temporal merge candidate may be selected. The temporal merge candidate is also referred to as Temporal MVP (TMVP).

The inter predictor 124 searches for a history-based motion vector predictor (HMVP) candidate. The inter predictor 124 may store the motion vectors of the previous h (here, h is a natural number) number of CUs in a table and then use them as merge candidates. The size of the table is 6, and the motion vector of the previous CU is stored according to the first-in-first-out (FIFO) method. This indicates that up to 6 HMVP candidates are stored in the table. The inter prediction unit 124 may set recent motion vectors among HMVP candidates stored in the table as merge candidates.

The inter prediction unit 124 searches for a Pairwise Average MVP (PAMVP) candidate. The inter prediction unit 124 may set an average of motion vectors of a first candidate and a second candidate in the merge candidate list as a merge candidate.

When the merge candidate list cannot be filled even after performing all of the above search processes (ie, when the preset number cannot be filled), the inter prediction unit 124 adds a zero motion vector as a merge candidate.

In terms of encoding efficiency optimization, the inter predictor 124 may determine a merge index indicating one candidate in the merge candidate list. The inter predictor 124 may derive a motion vector predictor (MVP) from the merge candidate list using the merge index, and then determine the MVP as the motion vector of the current block. Also, the video encoding apparatus may signal the merge index to the video decoding apparatus.

In the case of the skip mode, the video encoding apparatus uses the same motion vector transmission method as the merge mode, but does not transmit a residual block corresponding to a difference between the current block and the prediction block.

The above-described method of constructing the merge candidate list may be equally performed by the inter prediction unit 544 in the video decoding apparatus. The video decoding apparatus may decode the merge index. The inter prediction unit 544 may derive the MVP from the merge candidate list using the merge index, and then determine the MVP as the motion vector of the current block.

Meanwhile, in the case of using the MMVD technique, the inter predictor 124 may derive the MVP from the merge candidate list using the merge index. For example, the first or second candidate of the merge candidate list may be used as MVP. Also, in terms of encoding efficiency optimization, the inter predictor 124 determines a magnitude index and a distance index. The inter predictor 124 may derive a motion vector difference (MVD) using the magnitude index and the direction index, and then restore the motion vector of the current block by adding the MVD and MVP. Also, the video encoding apparatus may signal the merge index, size index, and direction index to the video decoding apparatus.

The aforementioned MMVD technique may be equally performed by the inter prediction unit 544 in the video decoding apparatus. The video decoding apparatus may decode the merge index, size index, and direction index. After constructing the merge candidate list, the inter predictor 544 may derive the MVP from the merge candidate list using the merge index. The inter predictor 544 may derive the MVD using the magnitude index and the direction index, and then restore the motion vector of the current block by summing the MVD and the MVP.

Hereinafter, a method of constructing a candidate list of motion information in the AMVP mode of inter prediction will be described. In order to support the AMVP mode, the inter prediction unit 124 in the video encoding apparatus may configure a candidate list by selecting a predetermined number (eg, two) of candidates.

The inter prediction unit 124 searches for spatial candidates. The inter prediction unit 124 searches for spatial candidates from neighboring blocks as illustrated in FIG. 4 . Up to two spatial candidates can be selected.

The inter prediction unit 124 searches for temporal candidates. The video encoding apparatus may add a block co-located with the current block in a reference picture (which may be the same as or different from the reference picture used to predict the current block) other than the current picture where the target block is located as a temporal candidate. can One temporal candidate may be selected.

When the candidate list cannot be filled even after performing all of the above search processes (ie, when the preset number cannot be filled), the inter predictor 124 adds a zero motion vector as a candidate.

In terms of encoding efficiency optimization, the inter predictor 124 may determine a candidate index indicating one candidate in the candidate list. The inter predictor 124 may derive an MVP from the candidate list using the candidate index. Also, in terms of encoding efficiency optimization, the inter prediction unit 124 calculates the MVD by subtracting MVP from the motion vector after determining the motion vector. The video encoding apparatus may signal the candidate index and the MVD to the video decoding apparatus.

The above-described method of constructing the AMVP candidate list may be equally performed by the inter prediction unit 544 in the video decoding apparatus. The video decoding apparatus may decode the candidate index and the MVD. The inter predictor 544 may derive an MVP from the candidate list using the candidate index. The inter prediction unit 544 may restore the motion vector of the current block by summing the MVD and the MVP.

Meanwhile, the video encoding apparatus transmits information for determining the spatial resolution of the MVD together with the MVD. When AMVR technology is used, the video encoding apparatus may determine the adaptive spatial resolution of the MVD in terms of bit rate distortion optimization. In this case, the spatial resolution of the MVD and the spatial resolution of the motion vector may be the same.

When using the AMVR technology, the video encoding apparatus notifies the spatial resolution of the MVD by signaling amvr_flag and amvr_precision_idx to the video decoding apparatus. That is, when amvr_flag is signaled as 0, the video decoding apparatus sets the MVD to 1/4-pel spatial resolution. On the other hand, if amvr_flag is not 0, the video decoding apparatus may determine the spatial resolution of the MVD according to amvr_precision_idx. At this time, the spatial resolution of the selectable MVD may vary according to the prediction method to which AMVR is applied. Prediction methods to which AMVR can be applied include a general AMVP mode, an affine AMVP mode, and an IBC AMVP mode.

II-2. Affine Merge Mode and Affine AMVP Mode

Inter prediction is motion prediction that reflects a translation motion model. That is, it is a technique for predicting movement in the horizontal direction (x-axis direction) and vertical direction (y-axis direction). However, in reality, various types of motions such as rotation, zoom-in, or zoom-out may exist in addition to translational motion. Affine motion prediction may reflect these various types of motion.

6A and 6B are exemplary diagrams for explaining affine motion prediction according to an embodiment of the present disclosure.

Two types of models can exist for affine motion prediction. One is, as in the example of FIG. 6A, two control point motion vectors (Control Point Motion Vectors, CPMVs) of the top-left corner and top-right corner of the target block to be currently encoded, that is, 4 It is a model using two parameters. The other, as in the example of FIG. 6B, is a model using three control point motion vectors of the upper-left corner, upper-right corner, and bottom-left corner of the target block, that is, six parameters.

A four-parameter affine model is expressed as shown in Equation 1. The motion at the sample position (x,y) in the target block can be calculated as shown in Equation 1. Here, the location of the top left sample of the target block is assumed to be (0,0).

Also, the six-parameter affine model is expressed as shown in equation (2). The motion at the sample position (x,y) in the target block can be calculated as shown in Equation 2.

Here, (mv _0x ,mv _0y ) is the upper left ear control point motion vector, (mv _1x ,mv _1y ) is the upper right ear control point motion vector, and (mv _2x ,mv _2y ) is the lower left ear control point motion vector. W is the horizontal length of the target block, and H is the vertical length of the target block.

Affine motion prediction may be performed using a motion vector calculated according to Equation 1 or Equation 2 for each sample in the target block. Alternatively, in order to reduce computational complexity, for example, the target block may be divided into subblocks having a size of 4×4, and then performed in units of subblocks.

The motion vectors (mv _x , mv _y ) may be set to have 1/16 sample precision. In this case, the motion vectors (mv _x , mv _y ) calculated according to Equation 1 or 2 may be rounded to 1/16 sample units.

The video encoding apparatus selects an optimal prediction method by performing intra prediction, inter prediction (translational motion prediction), affine motion prediction, and the like, and calculating a rate-distortion (RD) cost. To perform affine motion prediction, the inter prediction unit 124 of the video encoding apparatus determines which of the two types of models to use, and determines two or three control points according to the determined type. The inter prediction unit 124 calculates a motion vector (mv _x , mv _y ) for each of the subblocks in the target block using the control point motion vectors. In addition, a prediction block for each subblock in the target block is generated by performing motion compensation within the reference picture in units of subblocks using motion vectors (mv _x , mv _y ) of each subblock.

The video encoding apparatus encodes affine-related syntax elements including a flag indicating whether affine motion prediction is applied to the target block, type information indicating the type of the affine model, and motion information indicating the motion vector of each control point, and the like, and decodes the video. delivered to the device. Type information and motion information of control points may be signaled when affine motion prediction is performed, and the number of motion vectors of control points determined according to type information may be signaled.

The video decoding apparatus determines the type of the affine model and control point motion vectors using the signaled syntaxes, and uses Equation 1 or 2 to determine the motion vectors for each 4×4 subblock within the target block. Computes (mv _x , mv _y ). If the motion vector resolution information for the affine motion vector of the target block is signaled, the motion vector (mv _x , mv _y ) is corrected to a precision identified by the motion vector resolution information using an operation such as rounding.

The video decoding apparatus generates a prediction block for each subblock by performing motion compensation within a reference picture using a motion vector (mv _x , mv _y ) for each subblock.

In order to reduce the amount of bits required to encode the control point motion vectors, the above-described general inter prediction (translational motion prediction) method may be applied.

As an example, in the case of the affine merge mode, the inter predictor 124 of the video encoding apparatus configures a predefined number (eg, 5) of affine merge candidate lists. First, the inter prediction unit 124 of the image encoding apparatus derives an inherited affine merge candidate from neighboring blocks of a target block. For example, a merge candidate list is generated by deriving a predefined number of inheritance affine merge candidates from neighboring samples A0, A1, B0, B1, and B2 of the target block shown in FIG. 4 . Each of the inheritance affine merge candidates included in the candidate list corresponds to a combination of two or three CPMVs.

The inter prediction unit 124 derives an inherited affine merge candidate from control point motion vectors of neighboring blocks predicted in an affine mode among neighboring blocks of the target block. In some embodiments, the number of merge candidates derived from neighboring blocks predicted in an affine mode may be limited. For example, the inter predictor 124 may derive a total of two inherited affine merge candidates, one from A0 and A1 and one from B0, B1 and B2, from neighboring blocks predicted in the affine mode. The priorities may be the order of A0, A1, and the order of B0, B1, and B2.

Meanwhile, when the total number of merge candidates is three or more, the inter predictor 124 may derive as many constructed affine merge candidates as the insufficient number from translational motion vectors of neighboring blocks.

7 is an exemplary diagram for explaining a method of deriving combinatorial affine merge candidates for affine motion prediction.

The inter prediction unit 124 derives control point motion vectors CPMV1, CPMV2, and CPMV3 from the neighboring block group {B2, B3, A2}, the neighboring block group {B1, B0}, and the neighboring block group {A1, A0}, respectively. . As an example, the priority order within each neighboring block group may be the order of B2, B3, and A2, the order of B1, B0, and the order of A1 and A0. In addition, another control point motion vector CPMV4 is derived from a collocated block T in the reference picture. The inter predictor 124 combines two or three control point motion vectors among the four control point motion vectors to generate as many combined affine merge candidates as the insufficient number. The priority order of combinations is as follows. The elements in each group are arranged in the order of upper left, upper right, lower left control point motion vectors.

{CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4},

{CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}

When the merge candidate list cannot be filled using the inherited affine merge candidate and the combined affine merge candidate, the inter predictor 124 may add a zero motion vector as a candidate.

The inter prediction unit 124 selects a merge candidate from the merge candidate list in terms of encoding efficiency optimization and determines a merge index indicating the merge candidate. The inter prediction unit 124 performs affine motion prediction on the target block using the selected merge candidate. If the merge candidate consists of two control point motion vectors, affine motion prediction is performed using a 4-parameter model. On the other hand, when the merge candidate is composed of three control point motion vectors, affine motion prediction is performed using a 6-parameter model. The video encoding apparatus encodes the merge index and signals it to the video decoding apparatus.

The video decoding apparatus decodes the merge index. The inter prediction unit 544 of the video decoding apparatus constructs a merge candidate list in the same way as the video encoding apparatus, and performs affine motion prediction using control point motion vectors corresponding to the merge candidate indicated by the merge index.

As another example, in the case of the affine AMVP mode, in terms of encoding efficiency optimization, the inter prediction unit 124 of the video encoding apparatus determines the type of affine model for the target block and the corresponding actual control point motion vectors. The inter prediction unit 124 of the video encoding apparatus calculates the MVD, which is the difference between the actual control point motion vector and the MVP of each control point, for each control point, and then encodes the MVD of each control point. In order to derive the MVP of each control point, the inter predictor 124 constructs a predefined number (eg, two) of affine AMVP candidate lists. When the target block is a 4-parameter type, the candidates included in the list each consist of a pair of two control point motion vectors. On the other hand, when the target block is a 6-parameter type, the candidates included in the list each consist of a pair of three control point motion vectors.

Hereinafter, a method of constructing a candidate list in the affine AMVP mode will be described using the example of FIG. 8 . The affine AMVP candidate list can be derived similarly to the method of constructing the affine merge candidate list described above.

8 is a flowchart illustrating a process of searching for an affine AMVP candidate in an affine AMVP mode according to an embodiment of the present disclosure.

The inter prediction unit 124 checks whether the reference picture of the inheritance affine AMVP candidate and the reference picture of the current block are the same (S800). Here, the inherited affine AMVP candidate may be a block predicted in the affine mode among neighboring blocks A0, A1, B0, B1, and B2 of the target block shown in FIG. 4, as in the aforementioned affine merge mode.

When the reference picture of the inherited affine AMVP candidate and the reference picture of the current block are the same, the inter prediction unit 124 adds the corresponding inherited affine AMVP candidate (S802).

If the reference picture of the inherited affine AMVP candidate and the reference picture of the current block are not identical, the inter prediction unit 124 checks whether the reference pictures of all CPMVs of the combined affine AMVP candidate and the reference picture of the current block are identical (S804). . Here, all CPMVs of the combined affine AMVP candidate can be derived from motion vectors of neighboring samples shown in FIG. 7, as in the aforementioned affine merge mode.

When the reference pictures of all CPMVs of the combined affine AMVP candidates are identical to the reference pictures of the current block, the inter prediction unit 124 adds the corresponding combined affine AMVP candidate (S806).

At this time, the affine model type of the target block should be considered. When the affine model type of the target block is a 4-parameter type, the video encoding apparatus derives two control point motion vectors (upper left and upper right control point motion vectors of the target block) using the affine model of the neighboring block. If the affine model type of the target block is a 6-parameter type, three control point motion vectors (upper left, upper right, and lower left control point motion vectors of the target block) are derived using the affine model of the neighboring block.

If the reference pictures of all CPMVs and the reference picture of the current block are not the same, the inter prediction unit 124 adds a translational motion vector as an affine AMVP candidate (S808). The translational motion vectors may be used to predict the CPMV of the current block in the order of mv ₀ , mv ₁ , and mv ₂ .

If the candidate list cannot be filled even after all of the above steps (S800 to S808) are performed (ie, the preset number cannot be filled), the inter predictor 124 adds a zero motion vector as an affine AMVP candidate ( S810).

The inter prediction unit 124 selects one candidate from the affine AMVP list and determines a candidate index indicating the selected candidate. At this time, each control point motion vector of the selected candidate corresponds to the MVP of each control point. The coding efficiency inter predictor 124 determines the actual control point motion vector for each control point of the target block, and then calculates the MVD between the actual control point motion vector and the MVP of the control point. The video encoding apparatus encodes the affine model type of the target block, the candidate index, and the MVD of each control point, and signals it to the video decoding apparatus.

The video decoding apparatus decodes the MVD of the affine model type, candidate index, and each control point. The inter prediction unit 544 of the video decoding apparatus generates an affine AMVP list in the same manner as the video encoding apparatus, and selects a candidate indicated by a candidate index in the affine AMVP list. The inter prediction unit 544 of the video decoding apparatus restores the motion vector of each control point by adding the MVP of each control point of the selected candidate and the corresponding MVD. The inter prediction unit 544 performs affine motion prediction using the restored control point motion vectors.

II-3. Geometric Partitioning Mode (GPM)

9 is an exemplary diagram illustrating triangulation types supported in a geometric segmentation mode according to an embodiment of the present disclosure.

In GPM, the inter prediction unit 124 performs inter prediction based on triangular blocks from which the current block is divided. GPM supports two triangulation types, as illustrated in FIG. 9 . For the two triangular regions in each triangulation type, the inter prediction unit 124 performs inter prediction using different motion information (ie, motion vectors).

The inter prediction unit 124 generates final prediction signals by weighting the prediction signals of each region in order to minimize discontinuity at the boundary between the divided regions. Weights used for generating final prediction signals may be exemplified as shown in FIG. 10 . In the example of FIG. 10 , P ₁ is a predictor of a current block based on motion information of an upper right triangular block, and P ₂ represents a predictor of a current block based on motion information of a lower left triangular block.

11 is an exemplary diagram illustrating a GPM candidate list according to an embodiment of the present disclosure.

When constructing the GPM candidate list, motion information of each divided region is derived from the general merge candidate list, as illustrated in FIG. 11 . If the index of the merge candidate list is an even number, motion information existing in L0 (first reference list) is selected, and if it is an odd number, motion information existing in L1 (second reference list) is selected.

II-4. template matching

12 is an exemplary diagram illustrating template matching in intra prediction according to an embodiment of the present disclosure.

In the template matching (TM) mode, the intra prediction unit 122 in the video encoding apparatus searches for an optimal reference block using a template in the reconstructed region of the current frame, as shown in the example of FIG. 12, and A reference block is applied as a prediction block. A similar template most similar to the current template may be searched by calculating how much the L-shaped template matches the current template, and a block corresponding to the similar template may be used as a prediction block. A search range of the template may be set in advance, and prediction of the current block may be performed based on the preset search range.

Meanwhile, an adaptive reordering of merge candidates with template matching (ARRMC) technique adaptively rearranges merge candidates of inter prediction based on the above-described template matching. The reordering method of merge candidates can be applied to normal merge mode, template matching merge mode, or affine merge mode (excluding SbTMVP candidates).

For example, in the case of the template matching merge mode, the inter prediction unit 124 in the video encoding apparatus constructs a merge candidate list, divides the merge candidates into subgroups having a size of 5, and then converts the merge candidates into templates for each subgroup. It can be rearranged in ascending order (ie, in order of increasing cost) according to the matching cost (hereinafter referred to as TM cost). TM cost can be defined as the sum of absolute differences (SAD) or the sum of squared errors (SSE) between the template samples of the current block and the corresponding reference samples. .

Meanwhile, when a merge candidate uses bi-prediction, the inter predictor 124 may derive reference samples of a merge candidate template according to bi-prediction, as illustrated in FIG. 13 .

II-5. DMVR and multipass decoder side motion vector improvement

Decoder-side motion vector refinement (DMVR) finely adjusts the motion vectors (MV0 and MV1) of bi-directional prediction using Bilateral Matching (BM) technology, resulting in motion on the decoder side. How to improve vectors. Hereinafter, motion vectors of bidirectional prediction are used interchangeably with motion vector pairs.

In bi-directional prediction, the video encoding apparatus searches for a refined motion vector around initial motion vectors generated from reference pictures of reference lists L0 and L1. Here, the initial motion vectors mean two motion vectors MV0 and MV1 of bidirectional prediction. In the BM technique, a BM cost, which is a distortion between two candidate blocks in reference pictures of L0 and L1, is calculated. In this case, SAD or SSE between two candidate blocks may be calculated as the BM cost. As shown in Equation 3, the video encoding apparatus generates motion vector candidates having a minimum BM cost as refined motion vectors.

Here, MV_offset is an offset applied to initial motion vectors according to motion vector refinement, and is a difference between candidate motion vectors and initial motion vectors. This offset may be formed as a sum of an integer offset in units of integer samples and a sub-pixel offset in units of sub-pixel or sub-pel samples. As shown in Equation 3, a mirroring rule is followed for offsets of candidates of two motion vectors.

Meanwhile, multi-pass decoder-side motion vector refinement is a method of improving motion vectors by multi-pass at the decoder side using a BM technique. Hereinafter, multi-pass decoder-side motion vector improvement using BM technology is referred to as multi-pass bilateral matching (MBM).

The video encoding apparatus searches for motion vectors in units of CUs in a first pass among multiple passes of MBM, and searches for motion vectors for each 16×16 subblock in a CU in a second pass. The image encoding apparatus searches for motion vectors for each 8×8 sub-block by applying Bi-directional Optical Flow (BDOF) in the third pass. Motion vectors improved according to this search are stored for prediction of spatial and temporal motion vectors. Here, BDOF is a technique for additionally compensating for motions of predicted samples using bi-directional motion prediction based on the assumption that samples or objects constituting an image move at a constant speed and there is little change in sample values.

Motion vector improvement on the multi-pass decoder side performs the following process in detail.

In the first pass, the video encoding apparatus uses motion vectors (MV0 and MV1) of bi-directional prediction generated from reference pictures of reference lists L0 and L1 as initial values, and improves motion vectors (MV0_pass1 and MV0_pass1 and MV1) around them. MV1_pass1) is created. In this case, the improved motion vectors may be generated as in Equation 4 based on the minimum BM cost for the two reference blocks L0 and L1.

Here, deltaMV has integer sample precision around the initial value, and can be searched according to a 3×3 square search pattern around the initial MV.

In the second pass, the video encoding apparatus applies BM to a 16×16 subblock. For each subblock, the video encoding apparatus generates MV0_pass2 and MV1_pass2 by improving motion vectors around MV0_pass1 and MV1_pass1 obtained from the reference lists L0 and L1 in the first pass. At this time, motion vectors are searched with integer sample precision. Thereafter, the video encoding apparatus generates deltaMV(sbIdx2) as shown in Equation 5 using a subsample unit improvement process according to DMVR technology.

Here, sbIdx2 represents an index of a 16×16 subblock.

In the third pass, the video encoding apparatus derives final motion vectors by applying BDOF to an 8×8 subblock. The video encoding apparatus applies BDOF to motion vectors obtained in the second pass for each 8x8 subblock. MV0_pass3 and MV1_pass3, which are final motion vectors generated in the third pass, are expressed as in Equation (6).

Here, sbIdx3 represents an index of an 8x8 subblock, and bioMV represents a correction value according to application of BDOF.

Hereinafter, for each pass, the MBM cost represents a block matching cost between two blocks indicated by the searched motion vectors. In addition, deltaMV, deltaMV(sbIdx2), and bioMV are collectively referred to as improved values of improved motion vectors or 'improved values'. Also, for the initial motion vectors of the first pass, the improvement value may represent the sum of all or part of deltaMV, deltaMV(sbIdx2), and bioMV.

II-6. Bi-directional matching AMVP-MERGE mode

14 is an exemplary diagram illustrating a bidirectional matching AMVP-MERGE mode according to an embodiment of the present disclosure.

In the bidirectional matching AMVP-MERGE mode (BM AMVP-MERGE mode), when the unidirectional AMVP mode is selected for the reference picture in the LX (X = 0 or 1) direction, the video encoding device is located equidistantly in time. The motion vector of the merge candidate for the reference picture in the direction is improved. When improving, bi-directional matching techniques are used.

For bidirectional matching-based motion vector improvement, the video encoding apparatus generates improved motion vectors by applying multi-pass decoder-side motion vector improvement using the LX-direction AMVP vector and the L1-X direction merge candidate motion vector as initial values. In this case, the AMVP motion vector may be changed as much as the merge candidate motion vector is improved. For example, when the AMVP motion vector enhancement value is MVD0, the enhancement value MVD1 of the merge candidate motion vector may be -MVD0.

III. Addition, pruning and reordering of candidates in the motion vector list

Hereinafter, realization examples according to the present disclosure are described centering on an image encoding device, but may also be applied to an image decoding device.

Hereinafter, the motion vector list includes candidates. In realization example 1 and realization example 3, each candidate may be motion vectors MV0 and MV1 of bidirectional prediction, that is, a motion vector pair. In realization example 2, each candidate may include a motion vector pair or a motion vector of unidirectional prediction as motion information.

<Example 1> Merge and AMVP candidate list modification method using MBM

The video encoding apparatus may use MBM to modify the merge candidate list and the AMVP candidate list as follows.

First, the video encoding apparatus may improve each candidate motion vector pair using 3-step multiple passes of MBM. In this case, the video encoding apparatus may use final motion vectors generated in the third pass, but may selectively add motion vectors for each stage to the AMVP candidate list or the merge candidate list. For example, in the case of a normal merge mode, improved motion vectors of a first pass generated in units of CUs or improved motion vectors of a second path generated in subblocks of 16×16 units may be added to the candidate list. .

When the DMVR mode is applied, the first-pass enhancement motion vectors may be used.

In addition, when the DMVR mode and the affine merge mode are applied, the deltaMVs of the improved motion vectors generated in the first pass may be added to all CPMVs of each combined affine candidate in the affine merge candidate list.

In addition, when a candidate is determined in units of subblocks, such as a subblock-based merge mode or an affine-based motion vector candidate, improved motion vectors of a third pass may be added to the candidate list in addition to the first and second passes. .

Meanwhile, when AMVR is applied, the video encoding apparatus adds only motion vectors greater than or equal to the minimum luma sample unit according to amvr_precision_idx. To this end, multiple passes of MBM may be adaptively selected for the corresponding CU.

As an example, the video encoding apparatus may perform candidate addition, list rearrangement, list pruning, etc. on the AMVP candidate list using MBM as follows.

As described above, the video encoding apparatus generates improved motion vectors MV0_passN and MV1_passN by improving an existing pair of motion vectors MV0 and MV1 using 3-step multiple passes of MBM. Here, the existing motion vector pairs are candidates included in the AMVP candidate list and represent motion vectors of bi-directional prediction. For the existing motion vector pair (MV0, MV1), the video encoding apparatus generates one improved motion vector pair (MV0_passN, MV1_passN), but is not necessarily limited thereto. That is, one or more improved motion vector pairs may be generated for an existing motion vector pair according to the application of multiple passes.

For candidate addition, the video encoding apparatus adds improved motion vectors (MV0_passN, MV1_passN) to the existing AMVP candidate list. When adding to the list, the enhanced motion vectors can be added at a fixed location. For example, improved motion vectors may be located at the end of the list. Alternatively, the improved motion vectors may be located at the highest priority in the list.

Also, for pruning of the AMVP candidate list, the video encoding apparatus may replace or delete existing motion vectors from the list with improved motion vectors. In this case, replacement or deletion may be selected by calculating a cost difference between the improved motion vectors and the existing motion vectors. Here, the cost represents the MBM cost.

An alternative is to replace the existing motion vectors in the list with improved motion vectors if the improved motion vectors are better than the existing motion vectors in terms of cost. Deletion is to not add the improved motion vectors to the list if the improved motion vectors are not better than the existing motion vectors in terms of cost. Alternatively, deletion means leaving the best candidates as large as the size of the list in terms of MBM cost with respect to the candidate list to which the improved motion vector pairs of candidates are added, and removing the remaining candidates. In this case, MBM cost may be calculated after assuming that deltaMV is 0 for existing motion vector pairs.

Also, for rearrangement of the list, the video encoding apparatus may rearrange motion vector candidates of the AMVP candidate list based on the MBM cost. For example, the priority of the candidates may be determined by sorting the candidates in ascending order based on the MBM cost. In addition, the video encoding apparatus may rearrange the candidates based on the MBM cost for the AMVP candidate list to which candidates are added or pruned.

As another example, the video encoding apparatus may perform candidate addition, list rearrangement, list pruning, etc. on a general merge candidate list using MBM as follows.

As described above, the video encoding apparatus generates improved motion vectors MV0_passN and MV1_passN by improving an existing pair of motion vectors MV0 and MV1 using 3-step multiple passes of MBM. Here, the existing motion vector pairs represent motion vectors of bi-directional prediction as candidates included in the merge candidate list.

For candidate addition, the video encoding apparatus adds improved motion vectors (MV0_passN, MV1_passN) to the existing merge candidate list. When adding to the list, the enhanced motion vectors can be added at a fixed location. For example, the improved motion vectors may be located at the end of the list or at the highest priority.

Alternatively, the video encoding apparatus may add the improved motion vectors to a specific rank of the merge candidate list in consideration of a preset condition. The video encoding apparatus may calculate MBM costs for candidates in the list and consider their diversity. For example, the video encoding apparatus calculates a cost difference between a previous candidate of a specific rank and improved motion vectors, and then compares it with a preset threshold. When a specific order is the highest priority, a cost difference between a candidate with the highest priority and the improved motion vectors may be calculated. When the calculated cost difference is greater than or equal to the threshold value, the video encoding apparatus places the improved motion vectors in a specific rank. On the other hand, if the calculated cost difference is less than the threshold value, the video encoding apparatus determines that the improved motion vectors have overlap with the preceding candidate, and repeats the above-described process for the next rank in the specific rank.

Here, the threshold may be set based on the quantization parameter. For example, the threshold may be set as a Lagrangian parameter used for bit rate-distortion optimization.

Also, for pruning of the merge candidate list, the video encoding apparatus may replace or delete existing motion vectors from the list with improved motion vectors. In this case, replacement or deletion may be selected by calculating a cost difference between the improved motion vectors and the existing motion vectors. Here, the cost represents the MBM cost.

An alternative is to replace the existing motion vectors in the list with improved motion vectors if the improved motion vectors are better than the existing motion vectors in terms of cost. Deletion is to not add the improved motion vectors to the list if the improved motion vectors are not better than the existing motion vectors in terms of cost. Alternatively, deletion means leaving the best candidates as large as the size of the list in terms of MBM cost with respect to the candidate list to which the improved motion vector pairs of candidates are added, and removing the remaining candidates. At this time, after assuming that deltaMV is 0 for the existing motion vector, the MBM cost can be calculated.

Alternatively, for rearrangement of the list, the video encoding apparatus may rearrange motion vector candidates of the merge candidate list based on the MBM cost. For example, the priority of the candidates may be determined by sorting the candidates in ascending order based on the MBM cost. Also, the video encoding apparatus may rearrange candidates based on the MBM cost for the merge candidate list to which candidates are added or pruned.

Meanwhile, when the video encoding apparatus stores the encoded motion vectors in the HMVP table, it adds improved motion vectors according to TM, BM or MBM. In addition, the existing HMVP table is updated in a FIFO format, but in this embodiment, motion vectors in the HMVP table may be rearranged based on TM cost or MBM cost.

In addition, the video encoding device may rearrange the GPM candidate list as follows.

In the conventional method, when constructing the GPM candidate list, motion information of each region is derived from the general merge mode candidate list for two regions in which the current block is divided by an oblique line. That is, a candidate list may be constructed using motion information in L0/L1 depending on whether the index is an even number or an odd number. In this embodiment, the video encoding apparatus may configure the candidate list by prioritizing motion information in the merge mode candidate list based on the MBM cost, instead of such a configuration method. In this case, the MBM cost may be calculated based on motion information (ie, motion vectors) of the two regions.

Alternatively, in order to select motion information for each of the two regions, the image encoding apparatus may select a reference block having the smallest TM cost. Unidirectional motion information of each reference block may be selected from a motion information candidate list derived from neighboring blocks of the current block, as in the example of FIG. 15 . In order to improve encoding efficiency, the video encoding apparatus may configure a candidate list by prioritizing motion information based on TM cost.

In the existing GPM, as shown in the example of FIG. 9 , a block may be divided according to two division methods using an oblique line from an upper left to a lower right or an oblique line from an upper right to a lower left. In the template matching according to the present embodiment, only the division method using the oblique line from the upper left side to the lower right side where pixels adjacent to the current block exist is considered.

The video encoding apparatus may signal a flag indicating whether to use the reordering method for GPM.

Alternatively, the video encoding apparatus may use a TM/BM-based method when constructing the GPM motion information candidate list. In this case, the template of the current block uses the top and left reference samples according to a general TM/BM method, but the template of the reference block is a position indicated by motion vectors derived from subblocks adjacent to the boundary of the current block. samples are available. In the GPM template matching search process, the video encoding apparatus performs a matching search on the other triangulated area while maintaining motion information of one triangulated area among the two triangulated areas. The video encoding apparatus may repeat the above-described template matching search process for all general merge mode candidates.

As another example, the video encoding apparatus generates an affine merge/AMVP candidate list using MBM. can be rearranged.

In the affine mode process, the video encoding device configures a candidate list with inherited affine candidates and combined affine candidates. In this case, priorities among candidates in the list may be determined based on the MBM method.

When determining a combined affine candidate, the video encoding apparatus checks whether a motion vector exists in the order of neighboring blocks B2→B3→A2 as in the example of FIG. 7 and sets the first available motion vector as CPMV1. At this time, the video encoding apparatus rearranges the available motion vectors based on the MBM cost, and then determines the CPMV1 according to the priority order. Similarly, the video encoding apparatus checks the motion vectors in the order of B1→B0 to set CPMV2, checks the motion vectors in the order of A1→A0 and sets them to CPMV3, and determines CPMVs (CPMV2 and CPMV3) based on the MBM cost. can

In the case of the affine merge mode, the video encoding apparatus generates improved motion vectors using the first pass of MBM for the current block, and then assigns deltaMVs of the improved motion vectors to all CPMVs of each combination affine candidate in the list. can be counted At this time, a motion vector pair of the current block may be used as an initial value for applying the first pass.

Alternatively, in the case of the affine merge mode, the video encoding apparatus generates improved motion vectors for the current block using the first pass of MBM, and then adds the deltaMVs of the improved motion vectors to CPMVs selected from neighboring blocks. can do.

When determining a combined affine candidate, the video encoding apparatus checks whether a motion vector exists in the order of candidate block B2→B3→A2 as in the example of FIG. 7 and sets the first available motion vector as CPMV1. Similarly, the video encoding apparatus sets CPMV2 by checking the motion vectors in the order of B1 → B0, and sets them to CPMV3 by checking the motion vectors in the order of A1 → A0. At this time, the video encoding apparatus does not use fixed CPMVs, but calculates a regression model for estimating affine parameters using motion vectors of neighboring blocks of the current block, and recalculates modified CPMVs of the current block from the model. . After generating the improved motion vectors for the current block using the first pass of the MBM, the video encoding apparatus may add the deltaMV of the improved motion vectors to the corrected CPMVs.

<Example 2> Merge and AMVP candidate list modification method using TM

The video encoding apparatus performs template matching on each candidate in the AMVP/merge list and calculates a template cost. After that, the video encoding apparatus may rearrange the list in order of decreasing TM cost.

As an example, the video encoding device may rearrange candidates in the general AMVP candidate list based on template matching.

The video encoding apparatus may configure the AMVP candidate list to include up to two prediction candidates by adding the HMVP candidate to the spatial and temporal candidates. Then, the video encoding apparatus rearranges the order of the candidates using the TM cost. Alternatively, when selecting a candidate in the list, the video encoding apparatus may select a candidate based on the TM cost.

The image encoding apparatus may use up to two candidates, one each, as spatial candidates by referring to the left and upper blocks. As shown in the example of FIG. 16, the video encoding apparatus selects one of A _m and A _m+1 located on the left side as a candidate, and selects one of B _-1 , B _n and B _n+1 located on the upper side as a candidate. can Thereafter, the video encoding apparatus proceeds with encoding through bit rate distortion optimization. In this embodiment, the video encoding apparatus may determine the candidates based on the TM cost when determining the left and top candidates.

As another example, the video encoding apparatus may rearrange affine AMVP/merge candidates based on template matching.

In the affine mode process, the video encoding device may configure an affine AMVP candidate list to include an inherited affine candidate, a combined affine candidate, and the like. In this case, the video encoding apparatus may determine the priority between candidates using the TM.

When determining a combined affine candidate, the video encoding apparatus checks whether a motion vector exists in the order of candidate block B2→B3→A2 as in the example of FIG. 7 and sets the first available motion vector as CPMV1. At this time, the video encoding apparatus rearranges the motion vectors based on the TM cost and then determines the CPMV1 according to the priority order. Similarly, the video encoding apparatus may set CPMV2 by checking motion vectors in the order of B1 → B0, and set CPMV3 by checking motion vectors in the order of A1 → A0, and determine CPMVs based on the TM cost.

If there is more than one inheritance affine candidate or combinatorial affine candidate, each can be divided into subgroups. Thereafter, the video encoding apparatus rearranges the list by determining the priority within the subgroup based on template matching.

17 is a flowchart illustrating a process of searching for an affine AMVP candidate, including rearrangement of inherited affine AMVP candidates, according to an embodiment of the present disclosure.

In the example of FIG. 17 , the video encoding apparatus rearranges the inherited affine AMVP candidates in the subgroup by prioritizing the subgroup including the inherited affine AMVP candidates based on template matching (S1703). Since the remaining steps in the example of FIG. 17 are the same as those in the example of FIG. 8, detailed descriptions are omitted.

As another example, the video encoding apparatus may rearrange candidates in the TM AMVP/TM merge candidate list.

The video encoding apparatus calculates a TM cost by performing template matching on each candidate in the TM AMVP/TM merge list. Thereafter, the video encoding apparatus may rearrange the candidates in the list in an order of decreasing TM cost.

The video encoding apparatus may generate improved motion information by improving existing motion information using TM. Here, the existing motion information indicates a motion vector pair of unidirectional prediction or motion vector (MV0, MV1) of bidirectional prediction included in the candidate list.

When the candidate motion information is improved using TM, the video encoding apparatus may calculate the TM cost by reflecting the shape of the current block. For example, if the block is long in the horizontal direction, top samples of the template may be used, and if the block is long in the vertical direction, left samples may be used.

When generating the enhancement motion information using the TM, the video encoding apparatus does not add the enhancement motion information to the list if the TM cost is greater than a preset threshold. At this time, the threshold is calculated in consideration of the TM cost of the first candidate. For example, the threshold may be set by multiplying the TM cost of the first candidate by N (eg, N=5).

As another example, the video encoding apparatus may perform candidate addition, list rearrangement, list pruning, etc. on a normal merge candidate list based on the TM as follows. Hereinafter, motion information indicates a unidirectional predictive motion vector or a bidirectional predictive motion vector pair (MV0, MV1).

First, the video encoding apparatus generates improved motion information by improving existing motion information using the TM.

For candidate addition, the video encoding apparatus adds the improved motion information to the existing merge candidate list. When adding to the list, the enhanced motion information can be added at a fixed location. For example, the improved motion information may be located at the end of the list or at the top of the list.

Alternatively, the video encoding apparatus may add the improved motion information to a specific rank of the merge candidate list in consideration of a preset condition. The video encoding apparatus may calculate TM costs for candidates in the list and consider their diversity. For example, the video encoding apparatus calculates a cost difference between a previous candidate of a specific rank and improved motion information, and then compares the cost difference with a preset threshold. When a specific priority is the highest priority, a cost difference between a candidate with the highest priority and the improved motion information may be calculated. When the calculated cost difference is greater than or equal to a threshold value, the video encoding apparatus positions the improved motion information in a specific rank. On the other hand, if the calculated cost difference is smaller than the threshold value, the video encoding apparatus determines that the improved motion information has overlap with the preceding candidate, and repeats the above-described process for the next rank in the specific rank.

Here, the threshold may be set based on the quantization parameter. For example, the threshold can be set as a Lagrange parameter used for bitrate-distortion optimization.

Also, for pruning of the merge candidate list, the video encoding apparatus may replace or delete existing motion information from the list with improved motion information. In this case, replacement or deletion may be selected by calculating a cost difference between the improved motion information and the existing motion information. Here, the cost represents the TM cost.

The replacement is to replace the existing motion information in the list with the improved motion information when the improved motion information is better than the existing motion information in terms of cost. Deletion means not adding the improved motion information to the list if the improved motion information is inferior to the existing motion information in terms of cost. Alternatively, deletion means leaving the best candidates as large as the size of the list in terms of TM cost and removing the remaining candidates for list candidates including improved motion information. At this time, after assuming that deltaMV is 0 for the existing motion information, the TM cost may be calculated.

Alternatively, for rearrangement of the list, the video encoding apparatus may rearrange motion vector candidates of the merge candidate list based on the TM cost. For example, the priority of the candidates may be determined by sorting the candidates in ascending order based on TM cost.

<Example 3> Method for modifying candidate list in BM AMVP-MERGE mode

Hereinafter, a method of adding motion vector candidates and rearranging the candidate list in the BM AMVP-MERGE mode will be described.

Motion vectors in the L0 direction and the L1 direction according to the BM AMVP-MERGE mode are referred to as MV0_passN and MV1_passN, respectively. MV0_passN and MV1_passN are improved motion vectors generated according to one of MBM's multiple passes.

The video encoding apparatus replaces MV0 and MV1 among existing AMVP/merge candidates and uses MV0_passN and MV1_passN. For example, in the case of HMVP candidates, the replaced motion vectors may be used for motion vector prediction of a subsequent block in addition to the current block.

The video encoding apparatus adds and uses MV0_passN and MV1_passN to MV0 and MV1 among existing AMVP/merge candidates. For example, in the case of HMVP candidates, the added motion vectors may be used for motion vector prediction of a next block in addition to the current block.

After replacement or addition, the video encoding apparatus may rearrange the ranks of the candidates in the list. For example, the video encoding apparatus may give MV0_passN and MV1_passN the highest priority. Alternatively, the ranking of candidates may be adaptively arranged based on BM, MBM, or TM.

The video encoding device may store and use MV0_passN and MV1_passN in HMVP. The video encoding apparatus may separately store MV0_passN and MV1_passN in addition to storing them in the existing HMVP and use them for motion vector prediction of the next block.

The video encoding apparatus uses MV0_passN and MV1_passN as candidate motion vectors for prediction in the L0 and L1 directions, respectively, but may conversely use them as candidate motion vectors for prediction in the L1 and L0 directions. At this time, the signs of the candidate motion vectors may be reversed. For example, -MV0_passN may be used in the L1 direction and -MV1_passN may be used in the L0 direction. This method can be applied in response to all the cases described above.

MV0_passN and MV1_passN are motion vectors improved from MV0 and MV1, which are candidate motion vectors in the original L0 and L1 directions, but MV0_passN may have the same value as MV0 or MV1_passN may have the same value as MV1. These values may be applied correspondingly to all of the above cases.

The video encoding apparatus may determine whether to use the above-described embodiment by using an additional flag in units of prediction units (PUs).

Hereinafter, a method of inter-predicting a current block by an image encoding apparatus or an image decoding apparatus based on MBM multi-passes and MBM cost will be described as illustrated in FIGS. 18 and 19 .

18 is a flowchart illustrating a method of inter-predicting a current block by an image encoding apparatus according to an embodiment of the present disclosure.

The video encoding apparatus determines a candidate index (S1800). Here, the candidate index indicates one of a plurality of candidates in the candidate list, and each candidate represents a motion vector pair of bidirectional prediction.

In case of merge mode, the candidate list may be a general merge candidate list, a GPM candidate list, or an affine merge candidate list. In case of AMVP mode, the candidate list may be a general AMVP candidate list or an affine AMVP candidate list.

The video encoding apparatus generates a candidate list using information surrounding the current block (S1802).

The video encoding apparatus modifies the candidate list based on the multiple passes of MBM and the MBM cost (S1804). Here, among multiple passes, the first pass searches for a current block, the second pass searches for subblocks within the current block, and the third pass searches for motion vector pairs based on the MBM cost for subblocks having a smaller size than the subblocks. . The MBM cost depends on the difference between two blocks indicated by a pair of motion vectors found for each pass.

The video encoding apparatus generates an improved motion vector pair that minimizes MBM cost for each candidate by using multiple passes. The image encoding apparatus may add the generated improved motion vector pair to the candidate list. The video encoding apparatus may perform substitution or deletion as described above by calculating a difference in MBM cost between an improved motion vector pair and a corresponding candidate. Alternatively, the video encoding apparatus may rearrange candidates in the candidate list in ascending order based on the MBM cost.

The video encoding apparatus extracts a motion vector pair of the current block from the modified candidate list using the candidate index (S1806).

The video encoding apparatus generates a prediction block of the current block using the extracted motion vector pair (S1808).

The video encoding apparatus encodes the candidate index (S1810).

19 is a flowchart illustrating a method of inter-predicting a current block by an image decoding apparatus according to an embodiment of the present disclosure.

The video decoding apparatus decodes the candidate index from the bitstream (S1900). Here, the candidate index indicates one of a plurality of candidates in the candidate list, and each candidate represents a motion vector pair of bidirectional prediction.

In case of merge mode, the candidate list may be a general merge candidate list, a GPM candidate list, or an affine merge candidate list. In case of AMVP mode, the candidate list may be a general AMVP candidate list or an affine AMVP candidate list.

The video decoding apparatus generates a candidate list using neighboring information of the current block (S1902).

The video decoding apparatus modifies the candidate list based on the multiple passes of MBM and the MBM cost (S1904). Here, among multiple passes, the first pass searches for a current block, the second pass searches for subblocks within the current block, and the third pass searches for motion vector pairs based on the MBM cost for subblocks having a smaller size than the subblocks. . The MBM cost depends on the difference between two blocks indicated by a pair of motion vectors found for each pass.

The video decoding apparatus generates an improved motion vector pair that minimizes MBM cost for each candidate by using multiple passes. The video decoding apparatus may add the generated improved motion vector pair to the candidate list. The video decoding apparatus may perform replacement or deletion as described above by calculating a difference in MBM cost between an improved motion vector pair and a corresponding candidate. Alternatively, the video decoding apparatus may rearrange candidates in the candidate list in ascending order based on the MBM cost.

The video decoding apparatus extracts a motion vector pair of the current block from the modified candidate list using the candidate index (S1906).

The video decoding apparatus generates a prediction block of the current block using the extracted motion vector pair (S1908).

Hereinafter, a method of inter-predicting the current block by the video encoding apparatus and the video decoding apparatus based on TM and TM cost will be described as illustrated in FIGS. 20 and 21 .

20 is a flowchart illustrating a method of inter-predicting a current block by an image encoding apparatus according to another embodiment of the present disclosure.

The video encoding apparatus determines a candidate index (S2000). Here, the candidate index indicates one of a plurality of candidates in the candidate list, and each candidate includes, as motion information, a motion vector pair of bidirectional prediction or a motion vector of unidirectional prediction.

In case of merge mode, the candidate list may be a general merge candidate list, a GPM candidate list, a TM merge candidate list, or an affine merge candidate list. In case of AMVP mode, the candidate list may be a general AMVP candidate list, a TM AMVP candidate list, or an affine AMVP candidate list.

The video encoding apparatus generates a candidate list using information surrounding the current block (S2002).

The video encoding apparatus modifies the candidate list based on the TM and the TM cost (S2004). Here, the TM searches for a similar template corresponding to the template of the current block in the relief restoration area. The TM cost depends on the difference between samples in the current block's template and samples in similar templates.

The video encoding apparatus may rearrange candidates in the candidate list in ascending order based on the TM cost.

In the case of a normal merge candidate list, the video encoding apparatus generates improved motion information for each candidate that minimizes the TM cost by using the TM. The video encoding device may add the generated enhancement motion information to the candidate list. The video encoding apparatus may perform replacement or deletion as described above by calculating a TM cost difference between the enhancement motion information and the corresponding candidate.

The video encoding apparatus extracts motion information of the current block from the modified candidate list using the candidate index (S2006).

The video encoding apparatus generates a prediction block of the current block using the extracted motion information (S2008).

The video encoding apparatus encodes the candidate index (S2010).

21 is a flowchart illustrating a method of inter-predicting a current block by an image decoding apparatus according to another embodiment of the present disclosure.

The video decoding apparatus decodes the candidate index from the bitstream (S2100). Here, the candidate index indicates one of a plurality of candidates in the candidate list, and each candidate includes, as motion information, a motion vector pair of bidirectional prediction or a motion vector of unidirectional prediction.

In case of merge mode, the candidate list may be a general merge candidate list, a GPM candidate list, a TM merge candidate list, or an affine merge candidate list. In case of AMVP mode, the candidate list may be a general AMVP candidate list, a TM AMVP candidate list, or an affine AMVP candidate list.

The video decoding apparatus generates a candidate list using neighboring information of the current block (S2102).

The video decoding apparatus modifies the candidate list based on the TM and the TM cost (S2104). Here, the TM searches for a similar template corresponding to the template of the current block in the relief restoration area. The TM cost depends on the difference between samples in the current block's template and samples in similar templates.

The video decoding apparatus may rearrange candidates in the candidate list in ascending order based on the TM cost.

In the case of a normal merge candidate list, the video decoding apparatus uses TM to generate improved motion information that minimizes TM cost for each candidate. The video decoding apparatus may add the generated enhancement motion information to the candidate list. The video decoding apparatus may perform replacement or deletion as described above by calculating a TM cost difference between the enhancement motion information and the corresponding candidate.

The video decoding apparatus extracts motion information of the current block from the modified candidate list using the candidate index (S2106).

The video decoding apparatus generates a prediction block of the current block using the extracted motion information (S2108).

In the flow chart/timing diagram of the present specification, it is described that each process is sequentially executed, but this is merely an example of the technical idea of one embodiment of the present disclosure. In other words, those skilled in the art to which an embodiment of the present disclosure belongs may change and execute the order described in the flowchart/timing diagram within the range that does not deviate from the essential characteristics of the embodiment of the present disclosure, or one of each process Since the above process can be applied by performing various modifications and variations in parallel, the flow chart/timing chart is not limited to a time-series sequence.

In the above description, it should be understood that the exemplary embodiments may be implemented in many different ways. Functions or methods described in one or more examples may be implemented in hardware, software, firmware, or any combination thereof. It should be understood that the functional components described in this specification have been labeled "...unit" to particularly emphasize their implementation independence.

Meanwhile, various functions or methods described in this embodiment may be implemented as instructions stored in a non-transitory recording medium that can be read and executed by one or more processors. Non-transitory recording media include, for example, all types of recording devices in which data is stored in a form readable by a computer system. For example, the non-transitory recording medium includes storage media such as an erasable programmable read only memory (EPROM), a flash drive, an optical drive, a magnetic hard drive, and a solid state drive (SSD).

The above description is merely an example of the technical idea of the present embodiment, and various modifications and variations can be made to those skilled in the art without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of the present embodiment, but to explain, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The scope of protection of this embodiment should be construed according to the claims below, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of rights of this embodiment.

124: inter predictor
155: entropy encoding unit
510: entropy decoding unit
544 Inter prediction unit

Claims (17)

In the method of inter-predicting a current block, performed by a video decoding apparatus,
decoding a candidate index from a bitstream, wherein the candidate index indicates one of a plurality of candidates in a candidate list, and each candidate represents a motion vector pair of bi-directional prediction;
generating the candidate list using neighboring information of the current block;
Modifying the candidate list based on multiple passes of MBM (Multi-pass Bilateral Matching) and an MBM cost, wherein a first pass among the multiple passes is the current block, and a second pass is a subroutine within the current block. block, and the third pass searches motion vector pairs for sub-blocks having a smaller size than the sub-blocks based on the MBM cost, and the MBM cost is indicated by the searched motion vector pair for each pass. depends on the difference between the two blocks;
extracting a motion vector pair of the current block from the modified candidate list using the candidate index; and
Generating a prediction block of the current block using the extracted motion vector pair
Characterized in that, the method comprising a. According to claim 1,
The editing step is
and generating an improved motion vector pair that minimizes the MBM cost for each candidate using the multiple passes. According to claim 2,
The editing step is
When the candidate list is a general merge candidate list, the improved motion vector pair is generated using the first pass or the second pass, and when the candidate list is a subblock-based candidate list, the first or second pass In addition to the pass, the method characterized in that the improved motion vector pair is generated using the third pass. According to claim 2,
The editing step is
and adding the improved motion vector pair to a fixed position in the candidate list. According to claim 2,
The editing step is
characterized in that replacement or deletion is performed by calculating a MBM cost difference between the improved motion vector pair and the corresponding candidate. According to claim 5,
The replacement is
and replacing the corresponding candidate with the improved motion vector pair in the candidate list when the improved motion vector pair is better than the corresponding candidate in terms of the MBM cost. According to claim 5,
The deletion is
With respect to the candidate list to which the improved motion vector pairs of the candidates are added, the best candidates in terms of the MBM cost are left as much as the size of the candidate list and the remaining candidates are removed. According to claim 1,
The editing step is
and rearranging the candidates in the candidate list in ascending order based on the MBM cost. According to claim 2,
The editing step is
If the candidate list is a general merge candidate list, the improved motion vector pair is added to a specific rank of the candidate list in consideration of a preset condition. According to claim 9,
The editing step is
After calculating the MBM cost difference between the previous candidate of the specific rank and the improved motion vector pair, if the MBM cost difference is greater than or equal to a preset threshold, the improved motion vector pair is placed in the specific rank, and the MBM cost difference is If it is smaller than the preset threshold, the preset condition is considered again for a rank next to the specific rank. According to claim 2,
The editing step is
When the candidate list is a normal merge candidate list, the improved motion vector pair is added to a history-based motion vector predictor (HMVP) table. According to claim 1,
The editing step is
When the candidate list is a general merge candidate list, motion vector pairs in an HMVP table are rearranged based on the MBM cost. According to claim 1
The editing step is
When the candidate list is a geometric partitioning mode candidate list, the method characterized in that the candidate list is constructed by giving priority to motion information in a general merge candidate list based on the MBM cost. According to claim 1,
The correction step is
When the candidate list is an affine merge candidate list, when determining a combined affine candidate, available motion vectors of neighboring blocks of the current block are rearranged based on the MBM cost, and then the combined affine candidate is configured according to priority order A method characterized by determining control point motion vectors. According to claim 1,
The editing step is
If the candidate list is an affine merge candidate list,
extracting a motion vector pair of the current block from the candidate list using the candidate index;
Generating an improved motion vector pair that minimizes the MBM cost by using the first pass for the motion vector pair of the current block, wherein the improved motion vector pair is the motion vector pair of the current block and an improvement value made up of;
selecting control point motion vectors from neighboring blocks of the current block; and
adding the improvement value to the control point motion vectors;
Characterized in that, the method comprising a. In the method of inter-predicting a current block, performed by an image encoding apparatus,
determining a candidate index, wherein the candidate index indicates one of a plurality of candidates in a candidate list, and each candidate represents a motion vector pair of bi-directional prediction;
generating the candidate list using neighboring information of the current block;
Modifying the candidate list based on multiple passes of MBM (Multi-pass Bilateral Matching) and an MBM cost, wherein a first pass among the multiple passes is the current block, and a second pass is a subroutine within the current block. block, and the third pass searches motion vector pairs for sub-blocks having a smaller size than the sub-blocks based on the MBM cost, and the MBM cost is indicated by the searched motion vector pair for each pass. depends on the difference between the two blocks;
extracting a motion vector pair of the current block from the modified candidate list using the candidate index;
generating a prediction block of the current block using the extracted motion vector pair; and
Encoding the candidate index
Characterized in that, the method comprising a. A computer-readable recording medium storing a bitstream generated by an image encoding method, the image encoding method comprising:
determining a candidate index, wherein the candidate index indicates one of a plurality of candidates in a candidate list, and each candidate represents a motion vector pair of bi-directional prediction;
generating the candidate list using neighboring information of the current block;
Modifying the candidate list based on multiple passes of MBM (Multi-pass Bilateral Matching) and an MBM cost, wherein a first pass among the multiple passes is the current block, and a second pass is a subroutine within the current block. block, and the third pass searches motion vector pairs for sub-blocks having a smaller size than the sub-blocks based on the MBM cost, and the MBM cost is indicated by the searched motion vector pair for each pass. depends on the difference between the two blocks;
extracting a motion vector pair of the current block from the modified candidate list using the candidate index;
generating a prediction block of the current block using the extracted motion vector pair; and
Encoding the candidate index
Characterized in that it comprises, a recording medium.

Images