RU2778099C1 - Method and device for video encoding and decoding, using inter-prediction - Google Patents

Abstract

FIELD: video encoding/decoding.

SUBSTANCE: invention relates to means for video encoding and decoding. In a sequence header, the first high-level syntactic element is encoded in a sequence level for indication of whether both an affine motion prediction and at least one encoding tool are disabled. At least one encoding tool includes accounting adjustment of counts of the affine motion prediction. Depending on the first high-level syntactic element, one or more second high-level syntactic elements are encoded, indicating the establishment of a sequence level or an image level relatively to encoding resolution of each of at least one encoding tool. When the first high-level syntactic element indicates that both the affine motion prediction and at least one encoding tool are not disabled, motion vectors of control points are determined for a target block to be encoded, and counts of the affine prediction are generated for each of sub-blocks by performing the sub-block affine motion prediction.

EFFECT: increase in the efficiency of video encoding.

16 cl, 22 dwg

Description

The field of technology to which the invention relates

[1] The present disclosure relates to video encoding and decoding, and in particular, to an encoding tool for improving inter-prediction compression efficiency.

State of the art

[2] Since the amount of video data is larger than the amount of voice data or still image data, storage or transmission of video data without processing for compression requires a lot of hardware resources, including memory. Accordingly, when storing or transmitting video data, the video data is typically compressed using an encoder for storage or transmission. The decoder then receives the compressed video data, decompresses and plays back the video data. Compression procedures for such video include H.264/AVC and High Efficiency Video Coding (HEVC), which improves coding efficiency by about 40% over H.264/AVC.

[3] However, the image size, resolution, and frame rate gradually increase, and accordingly, the amount of data that needs to be encoded also increases. Accordingly, a new compression procedure having better coding efficiency and higher image quality than the existing compression procedure is required.

[4] Video coding mainly uses predictive coding to increase the efficiency of video compression. There are intra prediction for predicting a target block to be encoded based on previously reconstructed samples in the current picture, and inter prediction for predicting the current block using a previously reconstructed reference picture. In particular, inter-prediction is widely used for video coding because it exhibits superior compression performance compared to intra-prediction.

[5] The present disclosure provides a coding tool for improving the compression efficiency of an existing inter-prediction.

[6] In particular, the present disclosure relates to an encoding tool for improving compression efficiency in inter-prediction and, in one aspect, relates to an encoding tool capable of compensating for various motions of an object, including translational motion.

Disclosure of the essence of the invention

[7] According to one aspect of the present disclosure, a video decoding method for decoding a sequence of encoded pictures on a block-by-block basis is provided. The method includes determining, at a sequence level, whether affine motion prediction is enabled by decoding a first high-level syntax element from a bitstream; determining, at the image level, whether each of the at least one encoding tool is enabled by decoding one or more second high-level syntax elements for each of the at least one encoding tool from the bitstream depending on the first high-level syntax element, wherein at least one coding tool includes a sample-by-sample adjustment of affine motion prediction samples; when setting at the sequence level indicates that affine motion prediction is enabled, determining a plurality of motion vectors of control points of the target block encoded using affine motion prediction, and generating affine prediction samples for each of the sub-blocks in the target block by performing affine motion prediction in a unit of sub-blocks with using motion vectors of control points; and when, based on the picture level establishment, it is determined that affine motion prediction sample-by-sample adjustment is allowed for the current image containing the target block, modifying the sample values in the affine prediction samples by motion compensation according to the sample position in each of the sub-blocks based on the motion vectors of the reference target block points.

[8] According to another aspect of the present disclosure, a video decoding apparatus is provided for decoding a sequence of encoded pictures on a block-by-block basis. The apparatus includes a decoder configured to: determine, at a sequence level, whether affine motion prediction is enabled by decoding a first high-level syntax element from a bitstream; and determining, at the image level, whether each of the at least one encoding tool is enabled by decoding one or more second high level syntax elements for each of the at least one encoding tool from the bitstream depending on the first high level syntax element. In particular, the at least one coding tool includes a sample-by-sample correction of the affine motion prediction samples.

[9] The apparatus also includes a predictor configured to determine, when a sequence-level assertion indicates that affine motion prediction is enabled, a plurality of motion vectors of target block checkpoints encoded using affine motion prediction, and generate affine prediction samples for each sub-block in target block by performing affine motion prediction on a sub-block basis using motion vectors of control points. In response to identifying based on establishing at the image level that sample-by-sample adjustment of affine motion prediction samples is enabled for the current image containing the target block, the predictor is configured to modify affine prediction sample values by motion compensation according to the position of the sample in each of the vector-based sub-blocks. movement of control points of the target block.

[10] In accordance with another aspect of the present disclosure, a method for encoding a sequence of encoded pictures on a block basis is provided. The method includes encoding, in a sequence header, a first high-level syntax element in the sequence level to indicate whether both affine motion prediction and at least one coding tool are disabled, wherein the at least one coding tool includes a count-wise adjusting affine motion prediction readings; depending on the first high level syntax element, encoding one or more second high level syntax elements indicating the establishment of a sequence level or a picture level with respect to the encoding resolution of each of the at least one encoding tool; when the first high-level syntax element indicates that both affine motion prediction and at least one encoder are not disabled, determining the motion vectors of the checkpoints for the target block to be encoded and generating affine prediction samples for each of the sub-blocks by performing a sub-block affine motion prediction using control point motion vectors; and when the sample-by-sample adjustment of the affine motion prediction samples is enabled for the current image containing the target block, modifying the sample values in the affine prediction samples by motion compensation according to the position of the sample in each of the sub-blocks based on the motion vectors of the checkpoints of the target block.

Brief description of the drawings

[11] FIG. 1 is an exemplary block diagram of a video encoding apparatus capable of performing the procedures of the present disclosure.

[12] FIG. 2 shows a block division structure using the QTBTTT structure as an example.

[13] FIG. 3 shows a plurality of intra-prediction modes by way of example.

[14] FIG. 4 shows neighboring blocks around the current block as an example.

[15] FIG. 5 is an exemplary block diagram of a video decoding apparatus capable of implementing the technology of the present disclosure.

[16] FIG. 6 is an exemplary diagram illustrating the dual prediction optical stream concept of the present disclosure.

[17] FIG. 7 is an exemplary diagram illustrating a method for obtaining a gradient for a block boundary reference in a bidirectional optical flow.

[18] FIG. 8A-8B and 9 are exemplary diagrams illustrating the affine motion prediction provided by the present disclosure.

[19] FIG. 10 is an exemplary diagram illustrating a method for deriving affine motion prediction merge candidates from translational motion vectors of neighboring blocks.

[20] FIG. 11A-11c are exemplary diagrams illustrating a method for obtaining a light compensation parameter according to the light compensation embodiment of the present disclosure.

[21] FIG. 12A-12B are exemplary diagrams illustrating a method for dividing a target block into a plurality of areas according to another embodiment of lighting compensation provided by the present disclosure.

[22] FIG. 13 is an exemplary diagram illustrating a method for determining reference samples to be used to obtain compensation parameters when a target block is divided into four regions, according to another embodiment of illumination compensation provided by the present disclosure.

[23] FIG. 14 and 15 are exemplary diagrams illustrating a method for performing light compensation using four area light compensation parameters when a target block is divided into four areas, according to another embodiment of light compensation provided by the present disclosure.

[24] FIG. 16 and 17 are exemplary diagrams illustrating a method for performing illumination compensation for each region when a target block is divided into two regions, according to another embodiment of illumination compensation provided by the present disclosure.

[25] FIG. 18 are exemplary diagrams illustrating a method for performing per-area illumination compensation when a target block is divided into a plurality of areas, more than four areas, according to another embodiment of illumination compensation provided by the present disclosure.

Detailed description

[26] Hereinafter, an embodiment of the present invention will be described in detail with reference to the attached drawings. It should be noted that when adding numerals to the constituent elements in the respective drawings, the same numerals indicate the same elements, even if the elements are shown in different drawings. In addition, in the following description of the present disclosure, a detailed description of known functions and configurations included herein will be omitted so as not to overshadow the subject matter of the present disclosure.

[27] FIG. 1 is an exemplary block diagram of a video encoding device capable of implementing the technology of the present disclosure. Next, the video encoding device and elements of the device will be described with reference to FIG. one.

[28] The video encoding apparatus includes a block separator 110, a predictor 120, a subtractor 130, a transform 140, a quantizer 145, a reorder module 150, an entropy encoder 155, an inverse quantizer 160, an inverse transform 165, an adder 170, a loop filter module 180, and a memory. 190.

[29] Each element of the video encoding device may be implemented in hardware or software, or a combination of hardware and software. The functions of the respective elements may be implemented in software, and the microprocessor may be implemented to perform the software functions corresponding to the respective elements.

[30] One video consists of one or more sequences, including multiple images. Each image is divided into a plurality of regions, and encoding is performed for each region. For example, one image is divided into one or more fragments and/or sections. In particular, one or more tiles may be defined as a group of tiles. Each tile and/or slice is divided into one or more coding tree units (CTUs). And each CTU is divided into one or more coding units (CU) in a tree structure. Information applied to each CU is encoded as a CU syntax, and information applied to CUs included in one common CTU is encoded as a CTU syntax. In addition, information applicable to all blocks in one slice as a whole is encoded as slice header syntax, and information applicable to all blocks constituting one or more pictures is encoded in a picture parameter set (PPS) or picture header.

[31] In addition, information referenced by a sequence consisting of a plurality of pictures in general is encoded in a sequence parameter set (SPS). In addition, information applied to a single tile or group of tiles in a common tile image may be encoded as a tile or group of tiles header syntax. The syntaxes included in the SPS, PPS, slice header, and tile or tile group header may be referred to as high-level syntaxes.

[32] The block divider 110 is configured to determine a size unit (CTU) of the coding tree. CTU size information (CTU size) is encoded as SPS set or PPS set syntax and transmitted to the video decoding apparatus. In addition, the block divider 110 is configured to divide each image constituting a video into a plurality of CTUs having a predetermined size, and then recursively divide the CTUs using a tree structure. In the tree structure, the leaf node serves as a coding unit (CU), which is the basic coding unit.

[33] The tree structure can be a quadtree (QuadTree (QT)), in which a node (or parent node) is divided into four subnodes (or child nodes) of the same size, a binary tree (BinaryTree (BT)) in which a node is divided into two a subnode, a TernaryTree (TT) in which a node is divided into three subnodes in a 1:2:1 ratio, or a structure formed by a combination of two or more of a QT structure, a BT structure, and a TT structure. For example, the structure quadtree and binary tree (QTBT), or a quadtree and binary tree structure ternary tree (QTBTTT) Here, BTTT may be collectively referred to as a multiple type tree (MTT).

[34] FIG. 2 shows the QTBTTT partition tree structure as an example. As shown in FIG. 2, the CTU may be initially divided in the QT structure. A QT split may be repeated until the split box size reaches the minimum leaf node size MinQTSize allowed in the QT structure. The first flag (QT_split_flag) indicating whether each node of the QT structure is split into four lower layer nodes is encoded by the entropy encoder 155 and signaled to the video decoder. If the leaf node of the QT structure is equal to or less than the maximum root node block size (MaxBTSize) allowed in the BT structure, it may be further divided into one or more BT structures or TT structures. The BT structure and/or the TT structure may have multiple split directions. For example, there may be two directions, namely the direction in which the node block is split horizontally, and the direction in which the block is split vertically.

[35] As shown in FIG. 2, when MTT splitting starts, a second flag (mtt_split_flag) indicating whether the nodes are split, a flag indicating the splitting direction (vertical or horizontal) in case of splitting, and/or a flag indicating the type of splitting (binary or ternary) are encoded by the entropy encoder 155 and signaled to the video decoding device. Alternatively, before encoding the first flag (QT_split_flag) indicating whether each node is split into 4 lower layer nodes, a CU split flag (split_cu_flag) indicating whether the node is split may be encoded. When the value of the CU split flag (split_cu_flag) indicates that no split is performed, the node block becomes a leaf node in the split tree structure and serves a coding unit (CU), which is a basic coding unit. When the value of the CU split flag (split_cu_flag) indicates that the split is done, the video encoder starts encoding the flags in the manner described above, starting from the first flag.

[36] When QTBT is used as another example of a tree structure, there can be two types of partitioning, which are a type of horizontal partitioning of a block into two blocks of the same size (i.e., symmetrical horizontal partitioning) and a type of vertical partitioning of a block into two blocks of the same size (i.e. symmetrical vertical separation). A split flag (split_flag) indicating whether each node of the BT structure is divided into a lower layer block and split type information indicating the split type are encoded by the entropy encoder 155 and signaled to the video decoding apparatus. An additional type of splitting the node block into two asymmetric blocks is possible. The asymmetric split type may include a block split type into two rectangular blocks with an aspect ratio of 1:3 or a diagonal split type of a node block.

[37] In addition, CUs may have different sizes according to the division of CTU into QTBT or QTBTTT. Hereinafter, a block corresponding to a CU (ie, a QTBTTT leaf node) to be encoded or decoded is referred to as a "current block". Because the QTBTTT division is used, the shape of the current block can be square or rectangular. Predictor 120 predicts the current block to generate a prediction block. Predictor 120 includes intra predictor 122 and inter predictor 124.

[38] The intra prediction unit 122 is configured to predict samples in the current block using samples (reference samples) located around the current block in the current picture including the current block. There are many intra-prediction modes according to prediction directions. For example, as shown in FIG. 3, the plurality of intra prediction modes may include two non-directional modes, which include a planar mode and a DC mode, and 65 directional modes. Adjacent samples and the equation used are defined differently for each prediction mode.

[39] Intra predictor 122 may be configured to determine the intra prediction mode to be used when encoding the current block. In some examples, intra predictor 122 may be configured to encode the current block using multiple intra prediction modes and select an appropriate intra prediction mode to use from the tested modes. For example, intra-predictor 122 may be configured to calculate rate-distortion values using rate-distortion analysis of several tested intra-prediction modes and select the intra-prediction mode that has the best rate-distortion performance among the tested modes.

[40] The intra predictor 122 is configured for one intra prediction mode among the plurality of intra prediction modes, and predicts the current block using neighboring pixels (reference pixels) and an equation determined according to the selected intra prediction mode. Information about the selected intra-prediction mode is encoded by the entropy encoder 155 and transmitted to the video decoding device. Inter-predictor 124 is configured to predict a prediction block for the current block by motion compensation. In addition, the inter-predictor 124 is configured to search for a block most similar to the current block in a reference picture that was encoded and decoded earlier than the current picture, and generates a prediction block for the current block using the found block.

[41] The inter-predictor is then configured to generate a motion vector corresponding to an offset between the current block in the current picture and the prediction block in the reference picture. In general, motion estimation is performed for the luminance component, and a motion vector calculated from the luminance component is used for both the luminance component and the chrominance component. Motion information including reference picture information and motion vector information used to predict the current block is encoded by the entropy encoder 155 and transmitted to the video decoding apparatus.

[42] Inter-predictor 124 may be configured to perform interpolation based on a reference image or reference block to improve prediction accuracy. In other words, sub-samples between two consecutive integer samples are interpolated by applying the filter coefficients to a plurality of consecutive integer samples, including two integer samples. When a search operation for a block most similar to the current block is performed on the interpolated reference picture, the motion vector may be expressed with a fractional unit of count precision level rather than with an integer unit of count precision level.

[43] The motion vector accuracy or resolution may be set differently for each target encoded area, for example, for each unit such as slice, tile, CTU, or CU. When such adaptive motion vector resolution is applied, motion vector resolution information to be applied to each target area must be transmitted for each target area. For example, when the target area is a CU, motion vector resolution information applied to each CU is transmitted. The motion vector resolution information may be information indicating the accuracy of the motion vector difference, which will be described later.

[44] Inter predictor 124 may be configured to perform inter prediction using dual prediction. In bidirectional prediction, inter predictor 124 uses two reference pictures and two motion vectors representing the positions of blocks most similar to the current block in the respective reference pictures. The inter predictor 124 is configured to select the first reference picture and the second reference picture from reference picture list 0 (RefPicList0) and reference picture list 1 (RefPicList1), respectively, to search for blocks similar to the current block in the respective reference pictures, and generate the first a support block and a second support block. Inter predictor 124 is then configured to generate a prediction block for the current block by averaging or weighting the first reference block and the second reference block. Inter predictor 124 is then configured to send motion information, including information about two reference pictures and two motion vectors used to predict the current block, to encoder 150.

[45] In particular, RefPicList0 may be composed of images preceding the current image in display order among recovered images, and RefPicList1 may be composed of images following the current image in display order among recovered images. However, the embodiments are not limited to this. Pre-restored pictures following the current picture in display order may be further included in RefPicList0, and conversely, pre-restored pictures preceding the current picture may be further included in RefPicList1.

[46] Various techniques can be used to minimize the number of bits required to encode motion information. For example, when the reference picture and the motion vector of the current block are the same as the reference picture and the motion vector of the neighboring block, the motion information of the current block can be transmitted to the decoding apparatus by encoding information for identifying the neighboring block. This method is called "merge mode".

[47] In the merge mode, inter-predictor 124 selects a predetermined number of merge candidate blocks (hereinafter referred to as "merge candidates") from among neighboring blocks of the current block. As shown in FIG. 4, all or part of the left block L, the top block A, the top right block AR, the bottom left block BL, and the top left block AL that are adjacent to the current block in the current picture can be used as adjacent blocks to obtain merge candidates. In addition, a block located in a reference picture (which may be the same or different from the reference picture used to predict the current block) other than the current picture in which the current block is located may be used as a merging candidate. For example, a merged block that is in the same position as the current block, or blocks adjacent to the merged block in the reference picture, may further be used as merge candidates.

[48] The inter predictor 124 configures a merge list including a predetermined number of merge candidates using such neighboring blocks. The inter predictor 124 is configured to select a merge candidate to use as motion information for the current block from among the merge candidates included in the merge list, and generate merge index information to identify the selected candidates. The generated merge index information is encoded by the encoder 155 and transmitted to a decoder.

[49] Another way to encode traffic information is the AMVP mode. In the AMVP mode, inter predictor 124 is configured to obtain predicted motion vector candidates for the motion vector of the current block using adjacent blocks of the current block. All or part of the left block L, the top block A, the top right block AR, the bottom left block BL, and the top left block AL that are adjacent to the current block in the current image in FIG. 2 may be used as neighboring blocks used to obtain predicted motion vector candidates. In addition, a block located in a reference picture (which may be the same or different from the reference picture used to predict the current block) other than the current picture including the current block may be used as neighboring blocks used to obtain the predicted motion vector candidates. For example, a merged block that is in the same position as the current block, or blocks adjacent to the merged block in the reference picture, may be used.

[50] The inter-predictor 124 is configured to obtain predicted motion vector candidates using the motion vectors of adjacent blocks and determine a predicted motion vector for the motion vector of the current block using the predicted motion vector candidates. Then, the motion vector difference is calculated by subtracting the predicted motion vector from the motion vector of the current block.

[51] The predicted motion vector can be obtained by applying a predetermined function (eg, a function for calculating the median, mean, and the like) to the predicted motion vector candidates. In particular, the video decoding apparatus also "knows" the predefined function. Since the neighboring blocks used to obtain the predicted motion vector candidates have already been encoded and decoded, the video decoder also already "knows" the motion vectors of the neighboring blocks. Accordingly, the video encoding device does not need to encode information to identify the predicted motion vector candidates. Therefore, in this case, motion vector difference information and information about a reference picture used to predict the current block are encoded.

[52] The predicted motion vector may be determined by selecting any one of the predicted candidate motion vectors. Specifically, information for identifying a selected motion vector predicted candidate is further coded along with motion vector difference information and reference picture information to be used for prediction of the current block.

[53] Subtractor 130 is configured to subtract a prediction block generated by intra predictor 122 or inter predictor 124 from the current block to generate a residual block. Transformer 140 may be configured to divide the residual block into one or more sub-blocks and apply a transform to one or more sub-blocks, thereby converting the residual values of the transform blocks from the reporting domain to the frequency domain. In the frequency domain, transformed blocks are referred to as coefficient blocks or transform blocks containing one or more transform coefficient values. A 2D transform kernel can be used for the transform, and a 1D transform kernel can be used for the horizontal transform and vertical transform, respectively. Transform kernels can be based on Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), or other similar

[54] Transformer 140 may be configured to transform the residual signals in the residual block using the entire residual block size as the transform unit. Alternatively, the residual block may be divided into a plurality of sub-blocks, and the residual signals in the sub-blocks may be transformed using the sub-block as the transformation unit.

[55] The transducer 140 may be configured to individually transform the residual block in the horizontal direction and the vertical direction. Various types of transformation functions or transformation matrices can be used for the transformation. For example, pairs of transform functions for horizontal transform and vertical transform may be defined as a multiple transform set (MTS). Transformer 140 may be configured to select one pair of transform functions having the highest transform efficiency in the MTS and transform the residual block in the horizontal and vertical directions, respectively. Information (mts_idx) about the pair of transform functions selected in the MTS is encoded by the entropy encoder 155 and transmitted to the video decoder.

[56] The quantizer 145 is configured to quantize the transform coefficients output from the transform 140 using the quantization parameters and output the quantized transform coefficients to the entropy encoder 155. For some blocks or frames, the quantizer 145 may be configured to directly quantize the corresponding residual block without transform . The quantizer 145 may be configured to apply different quantization factors (scaling values) according to the positions of the transform coefficients in the transform block. A matrix of quantized coefficients applied to the two-dimensionally arranged quantized transform coefficients may be encoded and signaled to a video decoder.

[57] The reordering module 150 may be configured to resort the coefficient values for the quantized residual value. Reordering module 150 may be configured to convert a two-dimensional array of coefficients into a one-dimensional sequence of coefficients by scanning the coefficients. For example, reordering module 150 may be configured to scan the coefficients from the forward current (DC) coefficient to the coefficient in the high frequency domain using zigzag scanning or diagonal scanning to output a one-dimensional coefficient sequence. Depending on the size of the transform unit and the intra-prediction mode, a vertical scan, in which a 2D array of coefficients is scanned in the column direction, or a horizontal scan, in which 2D block coefficients are scanned in the runoff direction, may be used instead of a zigzag scan. In other words, the scan mode used can be determined among zigzag scan, diagonal scan, vertical scan, and horizontal scan according to the size of the transform unit and the intra prediction mode.

[58] The entropy encoder 155 is configured to encode the one-dimensional quantized transform coefficients output from the reordering module 150 using various coding procedures such as context-dependent adaptive binary arithmetic code (CABAC) and Exponential Golomb code to generate a bitstream. The entropy encoder 155 is configured to encode information such as CTU size, CU split flag, QT split flag, MTT split type, and MTT split direction that are associated with block split, so that the video decoder can split the block in the same way as in the encoder. video.

[59] In addition, the entropy encoder 155 is configured to encode prediction type information indicating whether the current block is encoded by intra prediction or inter prediction, and encodes intra prediction information (i.e., intra prediction mode information) or inter-prediction information (fusion index for the fusion mode, reference picture index information, and motion vector difference for the AMVP mode) according to the prediction type. The entropy encoder 155 also encodes quantization-related information, that is, quantization parameter information and quantization matrix information.

[60] The inverse quantizer 160 is configured to inversely quantize the quantized transform coefficients output from the quantizer 145 to generate the transform coefficients. The inverse transform 165 is configured to transform the transform coefficients output from the inverse quantizer 160 from the frequency domain to the spatial domain and to reconstruct the residual block.

[61] In addition, the adder 170 is configured to add the recovered residual block to the prediction block generated by the predictor 120 to recover the current block. The samples in the reconstructed current block are used as reference samples when performing intra-prediction of the next block. The loop filter module 180 is configured to filter the reconstructed samples to reduce blocking artifacts, ringing artifacts, and blur artifacts generated due to block prediction and transform/quantization. Loop filter module 180 may include at least a deblocking filter 182, an adaptive sample offset (SAO) filter 184, and an adaptive loop filter (ALF) 186.

[62] The deblocking filter 182 is configured to filter the boundary between the reconstructed blocks to remove the blocking artifact caused by block-by-block encoding/decoding, and the SAO filter 184 performs additional filtering of the deblocking filtered video. The SAO filter 184 is a filter used to compensate for the difference between the reconstructed sample and the original sample caused by lossy coding, and performs filtering such that it adds an appropriate offset to each reconstructed sample. The ALF 186 is configured to filter the target sample to be filtered by applying filter coefficients to the target sample and neighboring samples of the target sample. The ALF 186 may be configured to divide the samples included in the image into predefined groups and then determine one filter to be applied to the corresponding group to perform filtering differentially in each group. Information about the filter coefficients to be used for the ALF may be encoded and transmitted to the video decoder.

[63] The reconstructed blocks filtered through the loop filter module 180 are stored in the memory 190. When all blocks in one image are restored, the reconstructed image can be used as a reference image for inter-prediction of blocks in the image to be subsequently encoded.

[64] FIG. 5 is an exemplary functional block diagram of a video decoding apparatus capable of implementing the technology of the present disclosure. Next, the video decoding device and elements of the device will be described with reference to FIG. 5. The video decoding apparatus may include an entropy decoder 510, a reordering module 515, an inverse quantizer 520, an inverse transform 530, a predictor 540, an adder 550, a loop filter module 560, and a memory 570.

[65] Similar to the video encoding device of FIG. 1, each element of a video decoding apparatus may be implemented in hardware, software, or a combination of hardware and software. In addition, the function of each element may be implemented in software, and the microprocessor may be implemented to perform a software function corresponding to each element.

[66] The entropy decoder 510 is configured to determine the current block to be decoded by decoding the bitstream generated by the video encoding apparatus and extracting information related to block separation, and extracting prediction information and residual signal information, etc., necessary to restore the current block. In addition, the entropy decoder 510 is configured to extract CTU size information from a sequence parameter set (SPS) or a picture parameter set (PPS), determine the CTU size, and partition the image into CTUs of a certain size. Then, the decoder is configured to determine the CTU as the topmost level, that is, the root node, of the tree structure, and extract CTU split information to split the CTU using the tree structure.

[67] For example, when the CTU is split using the QTBTTT structure, the first flag (QT_split_flag) associated with QT split is extracted to split each node into four sublayer nodes. For a node corresponding to a QT leaf node, a second flag (MTT_split_flag) and split direction (vertical/horizontal) and/or split type (binary/ternary) information related to the MTT split are extracted to split the corresponding leaf node in the MTT structure. Accordingly, each node below a QT leaf node is recursively split in a BT or TT structure.

[68] As another example, when the CTU is split using the QTBTTT structure, a CU split flag (split_cu_flag) indicating whether the CU should be split can be retrieved. When the corresponding block is split, the first flag (QT_split_flag) can be extracted. In a split operation, zero or more recursive MTT splits may occur for each node after zero or more recursive QT splits. For example, the CTU may be configured to directly undergo an MTT separation without a QT separation, or repeatedly undergo only a QT separation.

[69] As another example, when the CTU is split using the QTBT structure, the first flag (QT_split_flag) associated with the QT split is extracted, and each node is split into four lower layer nodes. Then, a split flag (split_flag) indicating whether the node corresponding to the QT leaf node is further split in BT and split direction information is extracted.

[70] Once the current block to be decoded is determined by splitting in the tree structure, the entropy decoder 510 is configured to extract prediction type information indicating whether the current block is intra-predicted or inter-predicted. When the prediction type information indicates intra prediction, the entropy decoder 510 is configured to extract a syntax element for intra prediction information (intra prediction mode) for the current block. When the prediction type information indicates inter prediction, the entropy decoder 510 is configured to extract a syntax element for inter prediction information, that is, information indicating a motion vector and a reference picture referenced by the motion vector.

[71] The entropy decoder 510 is also configured to extract quantized transform coefficient information of the current block as quantization related information and residual signal information. Reordering module 515 may be configured to change the sequence of one-dimensional quantized transform coefficients entropy decoded by entropy decoder 510 into a two-dimensional array of coefficients (ie, a block) in the reverse order of the coefficient scan performed by the video encoder.

[72] The inverse quantizer 520 is configured to inversely quantize the quantized transform coefficients using a quantization parameter. The inverse quantizer 520 may be configured to apply different quantization factors (scaling values) to the quantized transform coefficients placed in two dimensions. The inverse quantizer 520 may be configured to perform inverse quantization by applying a matrix of quantization coefficients (scaling values) from the video encoder to a two-dimensional array of quantized transform coefficients.

[73] The inverse transform 530 is configured to inversely transform the inversely quantized transform coefficients from the frequency domain to the spatial domain to recover the residual signals, thereby generating a reconstructed residual block for the current block. In addition, when MTS is applied, the inverse transform 530 is configured to determine the transform functions or transformation matrices to be applied in the horizontal and vertical directions, respectively, using the MTS information (mts_idx) transmitted from the video encoder, and use the determined transform functions to reverse transformation of the transformation coefficients in the transformation block in the horizontal and vertical directions.

[74] Predictor 540 may include an intra predictor 542 and an inter predictor 544. Intra predictor 542 is activated when the prediction type of the current block is intra prediction, and inter predictor 544 is activated when the prediction type of the current block is inter prediction. The intra predictor 542 is configured to determine the intra prediction mode of the current block among the plurality of intra prediction modes based on the syntax element for the intra prediction mode extracted from the entropy decoder 510, and predict the current block using reference samples around the current block according to the mode. intra-predictions.

[75] The inter predictor 544 is configured to determine the motion vector of the current block and the reference picture referenced by the motion vector using the syntax element for the intra prediction mode extracted from the entropy decoder 510 and predict the current block based on the motion vector and the reference Images. The adder 550 is configured to reconstruct the current block by adding a residual block output from the inverse transform and a prediction block output from the inter-predictor or intra-predictor. The samples in the reconstructed current block are used as reference samples in intra-prediction of the next block to be decoded.

[76] Loop filter module 560 may include at least one of deblocking filter 562, SAO filter 564, and ALF filter 566. Deblocking filter 562 filters the boundary between reconstructed blocks to remove blocking artifacts caused by block decoding. The SAO filter 564 is configured to perform filtering by adding the reconstructed block after the deblocking filtering to the respective offsets to compensate for the difference between the reconstructed sample and the original sample caused by lossy coding. The ALF 566 is configured to filter the target sample to be filtered by applying filter coefficients to the target sample and neighboring samples of the target sample. The ALF 566 can be configured to divide the samples in the image into predetermined groups and then determine one filter to be applied to the corresponding group to perform filtering differentially in each group. The filter factor ALF is determined based on the filter factor information decoded from the bitstream.

[77] The reconstructed block, filtered through the loop filter module 560, is stored in the memory 570. When all blocks in one picture are reconstructed, the reconstructed picture can be used as a reference picture for inter-prediction of blocks in the picture to be subsequently encoded.

[78] The following disclosure relates to coding tools for improving inter-prediction compression performance that can be controlled by inter-predictor 124 of a video encoder and inter-predictor 544 of a video decoder. The term "target block" as used herein may have the same meaning as the terms "current block" or "coding unit (CU)" used above, or may mean a partial area of the CU.

[79] I. Bi-directional optical flow

[80] Bidirectional optical flow is a technique for additionally compensating for the movements of reports predicted using bidirectional motion prediction, provided that the samples or objects that make up the video move at a constant speed and that the values of the samples change little.

[81] FIG. 6 is an exemplary diagram illustrating the basic concept of BIO. It is assumed that the bidirectional motion vectors MV0 and MV1 pointing to the respective regions (i.e., reference blocks) most similar to the target block to be encoded in the current picture have been determined in the reference pictures Ref0 and Ref1 by (normal) bidirectional motion prediction for target block. The two motion vectors have values representing the motion of the entire target block. In the example in FIG. 6, P0 is a sample in reference picture Ref0 indicated by motion vector MV0 and corresponding to sample P in the target block, and P1 is a sample in reference picture Ref1 indicated by motion vector MV1 and corresponding to sample P in the target block.

[82] In addition, it is assumed that the movement for the reference P in FIG. 6 is slightly different from the general movement of the target block. For example, when an object located at sample A in Ref0 in FIG. 6 moves to sample B in Ref1 via sample P in the target block of the current picture, sample A and sample B can have values very similar to each other. Also, in this case, the point in Ref0 most similar to the P sample in the target block is not P0 indicated by the bidirectional motion vector MV0, but the A sample offset from P0 by the predefined offset vector (vxt0, vyt1). The point in Ref1 that most closely resembles the reference P in the target block is not P1, indicated by the bidirectional motion vector MV1, but reference B, which has been shifted from P1 by the given offset vector (-vxt0, -vyt1). In particular, t0 and t1 denote the distances Ref0 and Ref1 along the time axis with respect to the current image, respectively, and are calculated based on a picture order counter (POC). Hereinafter (vx, vy) is called "optical flow".

[83] When predicting the sample value P of the current block in the current picture, the two reference samples A and B provide more accurate prediction than the reference samples P0 and P1 indicated by the bidirectional motion vectors MV0 and MV1. Specifically, I(0)(i,j) denotes the value of the sample in the reference picture Ref0 indicated by the motion vector MV0 as corresponding to the sample (i,j) in the target block, and I(1)(i,j) denotes the value of the sample in the reference picture Ref1 indicated by the motion vector MV1 as corresponding to the sample (i,j) in the target block.

[84] The value of sample A in the reference picture Ref0 indicated by the motion vector BIO (vx, vy) as corresponding to the sample in the target block may be determined as I(0) (i+ vxt0, j +vyt0), and the value of sample B in the reference image Ref1 can be defined as I(1)(i -vxt1, j-vyt1). Here, when the linear approximation is performed using only the first order term of the Taylor series, A and B can be expressed as Equation 1.

Equation 1

[85] where Ix(k) and Iy(k) (k = 0, 1) are the gradient values in the horizontal and vertical directions at position (i, j) in Ref0 and Ref1. In addition, t0 and t1 denote the time axis distances for Ref0 and Ref1 relative to the current image, respectively, and are calculated based on POC: t0 = POC(current) - POC(Ref0) and t1 = POC(Ref1) - POC(current) .

[86] The bi-directional optical flow (vx, vy) of each sample in a block is defined as the solution that minimizes Δ, which is defined as the difference between sample A and sample B. Here, Δ can be determined in Equation 2 using a linear approximation from A and B obtained from Equation 1.

Equation 2

[87] For simplicity, the position (i, j) of the reference is omitted in each term of Equation 2.

[88] To implement a more reliable estimate of the optical flow, it is assumed that the movement is locally consistent with neighboring samples. For the BIO, the motion vector for the sample (i, j) currently to be predicted, the differences Δ in Equation 2 are taken into account for all samples (i', j') present in a mask Ω of a certain size centered on the sample (i , j). That is, the optical flow for the current sample (i, j) can be defined as a vector minimizing the objective function Φ(vx, vy), which is the sum of the squared differences Δ[i', j'] obtained for the corresponding samples in the mask Ω, as shown in Equation 3.

Equation 3

[89] Bidirectional optical flow according to the present disclosure can be applied in the case where one of the two reference pictures used for bidirectional prediction precedes the current picture in the display order, the other follows the current picture, and the distances from the two reference pictures to the current picture are equal to each other. to each other, that is, the differences in the picture order counter (POC) between each reference picture and the current picture are equal to each other. Therefore, t0 and t1 can be ignored.

[90] In addition, the bidirectional optical flow according to the present disclosure can only be applied to the luminance components. The bi-directional optical flow according to the present disclosure is performed on a sub-block basis, not a pixel basis, for a target block to which bi-directional prediction is applied. In the following description, for simplicity, it is assumed that the sub-unit, which may have different sizes of 2x2, 4x4 and 8x8, has a size of 4x4.

[91] Before executing the optical stream, the inter-predictor 124 of the video encoder generates two reference blocks for the target block using the aforementioned bidirectional prediction. The first reference block of the two reference blocks is a block composed of predicted samples generated from reference picture Ref0 using the first motion vector MV0 of the target block, and the second reference block is a block composed of predicted samples generated from reference picture Ref1 using motion vector MV1.

[92] The inter-predictor 124 is configured to calculate (vx, vy), called optical flow, for each of the 4x4 subblocks constituting the target block using the horizontal and vertical gradient values of the predicted samples in the first reference block and in the second reference block. The optical flow (vx, vy) is determined such that the difference between the predicted samples from the reference picture Ref0 and the predicted samples from the reference picture Ref1 is minimized. Inter-predictor 124 is configured to obtain a sample offset for modifying the bidirectional predicted samples for the 4x4 subblock using the (vx, vy) computed for the 4x4 subblock and gradients of the predicted samples in the 4x4 subblock.

[94] In particular, the inter-predictor 124 calculates the horizontal and vertical gradients of the sample values at position (i, j) using Equation 4.

Equation 4

where k is 0 or 1 and I (0) (i,j) and I (1) (i,j) denote the values of the samples at position (i,j) in the first reference block and in the second reference block, respectively. Also, Shift1 is a value derived from the bit depth of the luma component, for example, shift1 = max(6, bitDepth-6).

[95] To obtain the gradient of the sample located on the boundary of each reference block, samples outside the boundary of the first reference block and the second reference block are needed. Accordingly, as shown in FIG. 7, each anchor block is extended one column to the left and right, and one row up and down. To reduce the amount of computation, each sample in the extended portion can be filled with a sample or an integer sample at the nearest position in the reference block. In addition, gradients at a sample position outside the boundary of each reference block can be padded with gradients corresponding to samples at the nearest position.

[96] Inter-predictor 124 is configured to calculate S1, S2, S3, S5, and S6 corresponding to autocorrelation and cross-correlation of gradients using horizontal and vertical gradients in a 6x6 window spanning a 4x4 subblock as shown in FIG. 7.

Equation 5

[97] where Ω denotes a window spanning a subblock. In addition, as shown in Equation 6 below, Ψx(i,j) denotes the sum of the horizontal gradient values at position (i,j) at the first reference block and the second reference block, Ψy(i,j) denotes the sum of the vertical gradient values at position (i,j) in the first reference block and in the second reference block, and θ(i,j) denotes the difference between the reference value at position (i,j) in the second reference block and the reference value at position (i,j) in the first base block.

Equation 6

[98] where na and nb are values derived from bit depth and have the values min(1, bitDepth-11) and min(4, bitDepth-8).

[99] The inter-predictor 124 is configured to calculate the optical flows (vx, vy) for the 4x4 subblock using Equation 7 based on S1, S2, S3, S5, and S6.

Equation 7

, и

.

является функцией округления, и

.[100] where ,

, and

.

is the rounding function, and

.

[101] The sample offset for modifying the predicted sample at each sample position (x, y) in the 4x4 subblock in the target block can be calculated using the optical flow calculated for the 4x4 subblock and the gradient values at the sample position (x, y) as shown in Equation 8. In Equation 8, rnd() represents a rounding operation.

Equation 8

[102] The inter predictor 124 is configured to generate a final predicted sample pred(x, y) using the sample offset b(x, y) at position (x, y) and the predicted samples I(0)(x,y) and I (1) (x,y) in the first reference block and in the second reference block, as in Equation 9.

Equation 9

[103] where the offset is Max(3, 15-BitDepth) and Ooffset is the value for the round operation and is half the offset.

[104] As described above, the bidirectional optical flow method uses sample values predicted using motion information (two motion vectors and two reference images) used for bidirectional prediction. Therefore, the inter-predictor 544 of the video decoder can also be configured to implement bidirectional optical flow in the same manner as the video encoder using motion information (motion vectors, reference pictures) for bidirectional prediction received from the video encoder. This does not require the video encoding device to signal additional information for the bidirectional optical flow process to the video decoding device.

[105] II. Affine motion prediction

[106] The above-described inter-prediction is a motion prediction reflecting a forward motion model. In other words, it is a method for predicting movement in the horizontal direction (x-axis direction) and the vertical direction (y-axis direction). However, in practice, in addition to translational motion, there may be various types of motion such as rotation, zooming in or out. One aspect of the present disclosure provides for affine motion prediction capable of covering such different types of motion.

[107] FIG. 8A-8B are exemplary diagrams illustrating affine motion prediction. For affine motion prediction, there can be two types of models. One of them is a model using the motion vectors of the two control points of the upper left corner and the upper right corner of the target block to be encoded at the moment, that is, four parameters, as shown in FIG. 8A. Another model is a model using the motion vectors of the three control points of the upper left corner, the upper right corner and the lower left corner of the target block, that is, six parameters, as shown in FIG. 8B.

[108] The four-parameter affine model is expressed by Equation 10. The movement at the position (x, y) of the sample in the target block can be calculated from Equation 10. Here, the position of the above left sample of the target block is assumed to be (0, 0).

Equation 10

[109] A six-parameter affine model is expressed by Equation 11. Movement at the position (x, y) of the reference in the target block can be calculated from Equation 11.

Equation 11

[110] where (mv0x, mv0y) is the motion vector of the control point in the upper left corner, (mv1x, mv1y) is the motion vector of the control point in the upper right corner, and (mv2x, mv2y) is the motion vector of the control point in the lower left corner. W is a constant given by the horizontal length of the target box, and H is a constant given by the vertical length of the target box.

[111] Affine motion prediction may be performed for each sample in the target block using the motion vector calculated by equation 10 or 11. Alternatively, to reduce computational complexity, prediction may be performed for each sub-block separated from the target block, as shown in fig. 9. For example, the sub-block size may be 4x4, 2x2, or 8x8. In the exemplary embodiment below, affine motion prediction for the target block is performed based on the 4x4 sub-block. This example is for convenience of explanation only, and the present disclosure is not limited thereto.

[112] In subblock-based affine motion prediction, the motion vector (affine motion vector) of each subblock is calculated by substituting the center position of each subblock into (x, y) in equation 10 or 11. In particular, the center position may be the actual center point of the subblock or the lower right position of the center point reference. For example, in the case of a 4x4 subblock in which the bottom left sample coordinates are (0, 0), the position of the center of the subblock may be (1.5, 1.5) or (2, 2). The predictive block for each sub-block is generated using the affine motion vector (mvx, mvy) of the sub-block.

[113] The motion vectors (mvx,mvy) can be accurate to 1/16 of a sample. In particular, the motion vector (mvx, mvy) calculated from Equation 10 or 11 may be rounded up to 1/16 units of reference. Adaptive motion vector resolution can be applied to affine motion prediction as in conventional inter-prediction. In particular, information about the resolution of the motion vector of the target block, that is, the accuracy of the motion vector, is transmitted for each target block.

[114] Affine motion prediction can be performed not only for the luminance component, but also for the chrominance component. In the case of a 4:2:0 video format, when affine motion prediction for the luma component is performed based on a 4x4 subblock, affine motion prediction for the chrominance component may be performed based on a 2x2 subblock. The motion vector (mvx, mvy) of each chrominance component subblock may be derived from the motion vector of the corresponding luma component. Alternatively, the sub-block size for affine motion prediction for the chrominance component may be the same as for the luma component.

[115] When affine motion prediction for the luminance component is performed based on the 4x4 subblock, affine motion prediction for the chrominance component is also performed based on the 4x4 subblock. In this case, since the 4x4 chrominance subblock corresponds to four 4x4 luma subblocks, the motion vector (mvx,mvy) for the chrominance subblock can be calculated by calculating the average value of the chroma component motion vectors from the four corresponding luma component subblocks.

[116] The video encoding device performs intra-prediction, inter-prediction (progressive motion prediction), affine motion prediction, and the like. and calculates the rate-distortion (RD) cost to select the optimal prediction method. To perform affine prediction, the video coder inter-predictor 124 determines which of the two types of models to use, and determines two or three breakpoints according to the determined type. The inter predictor 124 calculates a motion vector (mvx, mvy) for each of the 4x4 subblocks in the target block using the motion vectors of the control points. It then performs motion compensation on the reference picture on a sub-block basis using the motion vector (mvx,mvy) of each sub-block to generate a predicted block for each sub-block in the target block.

[117] The entropy encoder 155 of the video coding apparatus is configured to encode associated affine syntax elements, including a flag indicating whether affine motion prediction is applied to the target block, type information indicating the type of the affine model, and motion information indicating the motion vector of each reference block. point, and transfer it to the video decoding device. The type information and motion information about the control points may be transmitted when affine motion prediction is performed, and as many control point motion vectors as determined according to the type information may be transmitted. In addition, when adaptive motion vector resolution is applied, motion vector resolution information about the affine motion vector of the target block is transmitted.

[118] The video decoder is configured to determine the affine model type and checkpoint motion vectors using the signaled syntax elements and calculate the motion vector (mvx, mvy) for each 4x4 subblock in the target block using Equation 10 or 11. When the resolution information is reported motion vector about the affine motion vector of the target block, the motion vector (mvx, mvy) is adjusted to the precision determined by the motion vector resolution information through operations such as rounding.

[119] The video decoding apparatus is configured to generate a predicted block for each sub-block by performing motion compensation in a reference picture using a motion vector (mvx, mvy) for each sub-block. In order to reduce the number of bits required to encode the motion vectors of the control points, the method used in the above-described conventional inter-prediction (forward motion prediction) can be applied.

[120] As an example, in the merge mode, the inter-predictor 124 of the video encoder is configured to obtain the motion vector of each control point from neighboring blocks of the target block. For example, inter predictor 124 is configured to generate a list of merge candidates by obtaining a predetermined number of merge candidates from adjacent samples L, BL, A, AR, and AL of the target block illustrated in FIG. 4. Each of the merge candidates included in the list corresponds to a pair of motion vectors from two or three control points.

[121] First, the inter-predictor 124 is configured to derive a merge candidate from the affine-predicted neighboring blocks motion vectors of control points among neighboring blocks. In some embodiments, the number of merge candidates derived from neighboring blocks predicted in affine mode may be limited. For example, inter-predictor 124 can be configured to produce two affine-predicted neighbor merge candidates: one from L and BL and one from A, AR, and AL. Priority can be assigned in the order L and BL and in the order A, AR and AL. When the total number of merge candidates is greater than or equal to 3, inter-predictor 124 may be configured to obtain the required number of merge candidates from the translation vectors of neighboring blocks.

[122] FIG. 10 is an exemplary diagram illustrating a method for deriving affine motion prediction merge candidates from translational motion vectors of neighboring blocks. Inter-predictor 124 is configured to obtain the reference point motion vectors CPMV1, CPMV2, CPMV3 from neighboring block group {B2, B3, A2}, neighboring block group {B1, B0}, and neighboring block group {A1, A0 }, respectively. For example, the priorities in each group of neighboring blocks may be assigned in the order B2, B3, A2, the order B1 and B0, and the order A1 and A0. In addition, another control point motion vector CPMV4 is obtained from the co-located block T in the reference picture. The inter-predictor 124 is configured to generate as many merge candidates as needed by combinations of two or three checkpoint motion vectors among the four checkpoint motion vectors. Combination priorities are assigned as shown below. The elements in each group are listed in order of motion vectors of top-left, top-right, and bottom-left control points.

{CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}

[123] The inter predictor 124 is configured to select a merge candidate from the merge candidate list and perform affine motion prediction for the target block. When the selected candidate consists of two control point motion vectors, motion affine prediction is performed using a four-parameter model. On the other hand, when the selected candidate consists of three control point motion vectors, motion affine prediction is performed using a six-parameter model. The entropy encoder 155 of the video encoder is configured to encode index information indicative of a merge candidate selected from the merge candidates in the merge candidate list and signal it to the video decoder.

[124] The entropy decoder 510 of the video decoder is configured to decode index information signaled from the video encoder. The inter-predictor 544 of the video decoder lists the merge candidates in the same manner as the video encoder, and performs affine motion prediction using the motion vectors of the checkpoints corresponding to the merge candidate indicated by the index information.

[125] As another example, in the AMVP mode, the video encoder inter-predictor 124 is configured to determine the type of affine model and control point motion vectors for the target block. Next, the inter predictor 124 is configured to calculate motion vector differences, which are the differences between the actual motion vectors of the control points of the target block and the predicted motion vectors of the respective control points, and transmit the motion vector differences that respectively correspond to the control points.

[126] Accordingly, the video encoder inter-predictor 124 configures a list of a predetermined number of affine AMVPs. When the target block is a 4-parameter type, each of the candidates included in the list consists of a motion vector pair of two control points. On the other hand, when the target block is of a 6-parameter type, each candidate included in the list is composed of a motion vector pair of three control points. The list of affine AMVPs can be obtained using the motion vectors of checkpoints or the translational motion vectors of adjacent blocks in a manner similar to the method of creating the merge candidate list described above.

[127] However, in order to obtain candidates for inclusion in the affine AMVP list, there may be a restriction to consider only neighboring blocks that rely on the same reference image as the target block among neighboring blocks in FIG. four.

[128] In addition, in the AMVP mode, the affine model type of the target block should be taken into account. When the target block affine model type is a 4-parameter type, the video encoding device is configured to obtain two control point motion vectors (upper left control point motion vector and upper right control point motion vector of the target block) using the neighboring block affine model. When the target block affine model type is a 6-parameter type, the device obtains three control point motion vectors (upper left corner, upper right corner, and lower left control point motion vector of the target block) using the neighbor block affine model.

[129] When a neighboring block is of a 4-parameter type, two or three checkpoint motion vectors are predicted according to the target block's affine model type using the two checkpoint motion vectors of the neighboring block. For example, the affine neighbor block model expressed by Equation 10 may be used. In Equation 10, (mv0x, mv0y) and (mv1x, mv1y) are replaced by the motion vectors of the control point at the upper left and upper right corner of the neighboring block, respectively. In particular, the horizontal length of the neighboring block is substituted into W. The predicted motion vector for each control point of the target block can be obtained by entering in (x, y) the difference between the position of the corresponding control point of the target block and the position of the upper left corner of the neighboring block.

[130] When the neighbor block is of the 6-parameter type, two or three checkpoint motion vectors are predicted according to the target block affine model type using the three checkpoint motion vectors of the neighbor block. For example, the affine neighbor block model expressed by Equation 11 may be used. In Equation 11, (mv0x,mv0y), (mv1x, mv1y) and (mv2x, mv2y) are replaced by the control point motion vectors of the upper left corner, upper right corner and lower left corner of the neighboring block , respectively. In particular, W and H are replaced by the horizontal length and vertical length of the neighboring block, respectively. The predicted motion vector for each control point of the target block can be obtained by entering in (x, y) the difference between the position of the corresponding control point of the target block and the position of the upper left corner of the neighboring block.

[131] The video encoder inter-predictor 124 is configured to select one candidate in the affine AMVP list and generate a motion vector difference between the motion vector of each actual breakpoint and the predicted motion vector of the corresponding checkpoint of the selected candidate. The entropy encoder 155 of the video encoder is configured to encode type information indicative of the target block affine model type, index information indicative of a candidate selected among the candidates in the affine AMVP list, and a motion vector difference corresponding to each breakpoint, and to signal this in video decoding device.

[132] The inter-predictor 544 of the video decoder is configured to determine the type of affine model using the information signaled from the video encoder and generate a motion vector difference for each breakpoint. Then, the inter predictor is configured to generate the AMVP affine list in the same manner as the video encoder and select the candidate indicated by the index information transmitted in the AMVP affine list. The inter-predictor 544 of the video decoder is configured to calculate the motion vector of each breakpoint by adding the predicted motion vector of each breakpoint of the selected candidate and the corresponding motion vector difference.

[133] III. Block-wise adjustment of affine motion prediction samples

[134] Sub-block affine motion prediction for a target block has been described above. Another aspect of the present disclosure relates to adjusting, on a sample basis, sample values of predicted samples generated by sub-block affine motion prediction. Movement by the position of each sample is additionally compensated in each sub-block, which forms the basis of affine motion prediction.

[135] When the sample values in any sub-block generated as a result of sub-block affine motion prediction for the target block are equal to I(x, y), the video coding device calculates horizontal and vertical gradients gx (i, j) and gy (i, j) at each reading position. Equation 12 can be used to calculate gradients.

Equation 12

[136] Count offset ΔI(i, j) for adjusting the prediction count is calculated by the following equation.

Equation 13

[137] where Δmv(i,j) denotes the motion offset, i.e. the difference between the affine motion vector at the reference (i, j) and the affine motion vector at the center position of the subblock, and can be calculated by Equation 10 or 11 depending on the type affine model of the target block. In other words, Δmv(i, j) can be calculated from Equation 10 or 11 by subtracting the motion vector given when the position of the center of the subblock is substituted into (x, y) from the motion vector given when (i, j) is substituted into ( x, y). In addition, Δmv(i, j) can be calculated from the equation that is obtained by replacing (x, y) in Equations 11 and 12 with a horizontal and vertical offset from the subblock center position to the reference position (i,j) and removing the last term " +mv0x" and "+mv0y". The center position may be the actual center point of the subblock, or may be the bottom right reference position of the center point.

[138] The motion vector of each control point of the target block used to calculate Δmv(i, j) and the difference between the reference position (i, j) and the position of the center of the sub-block are the same for all sub-blocks. Accordingly, the values of Δmv(i, j) can only be calculated for one subblock, such as the first subblock, and can be reused for other subblocks.

[139] The technique according to the present disclosure is based on the assumption that the object is moving at a constant speed and the change in the values of the reference is smooth. Therefore, the variation in the horizontal direction and the amount of change in the vertical direction are obtained by multiplying the x(Δmvx) component and the y (Δmvy) component of Δmv(i, j) by the horizontal and vertical reading gradient values, respectively. The offset ΔI(i,j) of the reference is calculated by adding two variants of the reference. The final value of the predicted count is calculated as follows.

Equation 14

[140] When sample-by-sample correction is applied to affine motion prediction samples, the inter-predictor 124 of the video coding device and the inter-predictor 544 of the video decoder perform the above-described process to correct sample values of the predicted samples generated by affine motion prediction. The gradient values are obtained from the predicted samples generated by affine motion prediction, and Δmv(i, j) is obtained from the motion vectors of the control points of the target block. Therefore, the video encoding device is not required to signal additional information for the process of the present technique to the video decoding device.

[141] IV. Local Illumination Compensation for Affine Motion Prediction Samples

[142] The local illumination compensation technique is an encoding technique that compensates for the change in illumination between a target block and a predicted block using a linear model. The video encoder inter-predictor 124 is configured to determine the reference block in the reference picture using the motion vector (translational motion vector) of the target block and obtain linear model parameters for illumination compensation using pre-reconstructed samples around (on the top and left sides) of the reference block and pre-recovered samples around (on the top and left sides) of the target block.

[143] When the pre-recovered samples around the reference block are denoted as x, and the corresponding pre-recovered samples around the target block are denoted as y, the parameters "A" and "b" are output as in equation 15, so that the sum of the squared differences between y and ( Ax + b) is minimized.

Equation 15

[144] The final predicted samples are generated by applying weight A and offset b to the samples in the predicted block (reference block) generated by the motion vector of the target block, as shown in Equation 16. In Equation 16, pred[x][y] is the predicted sample in position (x, y) generated by the motion vector of the target block, and predLIC[x][y] is the final predicted count after light compensation.

Equation 16

[145] Another aspect of the present disclosure relates to a technique for combining a lighting compensation technique with affine motion prediction.

[146] As described above, when sub-block affine motion prediction is applied to a target block, a motion vector is generated for each of the sub-blocks. Light compensation parameters may be obtained using the respective motion vector for each sub-block, and then illumination compensation may be performed in the sub-block module using the respective motion vector. This, however, not only increases the computational complexity, but also causes a latency problem. Since the reconstructed samples in each subblock are needed to compensate for the illumination of the next subblock, the illumination compensation process for the subblock must be suspended until the adjacent subblock is restored (i.e., until both the prediction block and the residual block for subblock). The present disclosure is directed to solving such problems.

[147] FIG. 11A-11C show various examples of determining the position of a reference block to obtain a lighting compensation parameter according to an embodiment of the present disclosure. In this embodiment, one set of illumination compensation parameters (A, b) for the target block is output, and the same parameter is applied to all sub-blocks in the target block. That is, the entire target block is modified with one set of light compensation parameters.

[148] As shown in FIG. 11A, inter-predictor 124 of the video encoder may be configured to determine the position of a reference block in a reference picture using the affine motion vector of a sub-block located at the upper left corner of the target block or the motion vector of a checkpoint at the upper left corner of the target block. Pre-recovered readings around a specific reference block are used to obtain parameters. Alternatively, as shown in FIG. 11B, the position of the reference block may be determined using the affine motion vector of the center subblock in the target block. Once the position of the reference block is determined, the illumination compensation parameter is derived using the pre-recovered samples adjacent to the top and left sides of the reference block and the corresponding pre-recovered samples adjacent to the top and left sides of the target block.

[149] As another example, a plurality of sub-blocks in a target block can be used. As shown in FIG. 11C, inter predictor 124 determines a reference subblock corresponding to each border subblock using the affine motion vectors of the subblocks (border subblocks) located on the border in the target block. The samples for obtaining the illumination compensation parameter are extracted from previously reconstructed samples adjacent to the boundary sub-blocks in the target block and the corresponding reference sub-blocks, respectively. For sub-blocks located at the top boundary in the target block and the corresponding reference sub-blocks, the samples are taken from the previously reconstructed samples adjacent to the top side. For sub-blocks located on the left boundary in the target block and the corresponding reference sub-blocks, the samples are taken from the previously reconstructed samples adjacent to the left side.

[150] In the present embodiment, since the illumination compensation parameter is derived using pre-built samples around the target block and pre-built samples around the reference block, the aforementioned delay problem can be solved. However, applying only one illuminance compensation parameter to a target block to which sub-block affine motion prediction is applied can potentially result in lower coding efficiency. Another embodiment, described below, results in an increase in coding efficiency in exchange for a slight increase in complexity.

[151] According to another embodiment of the present disclosure, the video encoder inter-predictor 124 divides the target block into a plurality of regions. Each region includes at least one subblock, which is the unit in which affine motion prediction is performed. Inter-predictor 124 is configured to calculate illumination compensation parameters for a plurality of areas, and performs illumination compensation in units of areas or sub-blocks in the target block based on the calculated illumination compensation parameters.

1. Separation of the target block

[152] The inter-predictor 124 of the video encoder is configured to divide the target block into a plurality of areas, each of which includes one or more sub-blocks, which are units in which affine motion prediction is performed.

[153] The target block may be divided using the motion vectors of the checkpoints of the target block. Fig. 12A-12B are examples of target block division using checkpoint motion vectors. Referring to FIG. 12A, when the angle between the upper-left reference point motion vector MV0 and the upper-right reference point motion vector MV1 of the target block is an obtuse angle (more than 90 degrees), the target block is vertically divided. For example, the target block can be divided into two regions. Otherwise, the target block is not split vertically. Referring to FIG. 12B, when the angle between the upper left reference point motion vector MV0 and the lower left reference point motion vector MV2 is an obtuse angle, the target block is horizontally divided, for example, into two halves. Otherwise, the target block is not split horizontally. When the affine model type of the target block corresponds to 4 parameters, the motion vector MV2 of the lower left corner reference point can be obtained by inputting the reference coordinates of the lower left corner of the target block in (x, y) in Equation 1.

[154] Based on the angle formed by the motion vectors of the checkpoint, the inter predictor 124 either skips splitting the target block or splits the target block into two regions (left region and right region in the case of a vertical split, and upper region and lower region in the case of a horizontal split). ) or 4 areas. When the dot product of two vectors is less than 0, the angle is obtuse. Accordingly, the inter-predictor 124 can determine whether the angle between the motion vectors of the reference point is an obtuse angle based on the dot product of the two vectors.

[155] Although the above example shows that the target block is split when the angle between the motion vectors of the two control points is an obtuse angle, the block can be split when the angle is an acute angle (angle less than 90 degrees). In addition, the size of the target block may be taken into account when splitting the target block. For example, when the horizontal length exceeds the set threshold, the target block is split vertically. When the vertical length exceeds a preset threshold, the target block is split horizontally. Both the size of the target block and the angle between the motion vectors of the control point can be used to split the target block.

[156] For example, the video encoder may be configured to determine whether to split a block based on the angle between the motion vectors of the control points only when the horizontal and/or vertical length of the target block is greater than a predetermined threshold. As another example, when the horizontal length exceeds a predetermined threshold, the video encoding device may be configured to determine whether to vertically split the block based on the angle between the top left and top right control point motion vectors of the target block. In addition, when the vertical length exceeds a predetermined threshold, the device can be configured to determine whether to split the block vertically based on the angle between the top left corner and bottom left control point motion vectors of the target block.

[157] Although it has been described that the target block is divided into a plurality of regions based on the motion vectors of the checkpoints of the target block, the embodiments are not limited to this. For example, the affine motion vectors of the top left sub-box, the top right sub-box, and the bottom left sub-box in the target box can be used instead of the top left anchor point motion vector MV0, the top right anchor point motion vector MV1, and the lower left corner anchor motion vector MV2.

[158] While FIG. 12A-12B show that the target block is split in half in the horizontal or vertical direction, the embodiments are not limited to this. For example, the target block may be divided into a plurality of blocks in the horizontal direction and/or a plurality of blocks in the vertical direction. Alternatively, the target block may be divided into sub-blocks in which affine motion prediction has been performed. In this case, one subblock constitutes one area.

[159] Each split area from the target block must be greater than or equal to the size of the sub-block in which affine motion prediction was performed. In other words, each area must consist of one or more sub-blocks. For example, if the sub-block size is 4x4, the area size is 2m x 2n (where m and n are integers greater than or equal to 2).

[160] When the target block is not divided, the illumination compensation parameters can be obtained according to the first embodiment described above. Next, in the case where the target block is divided, a method for obtaining illumination compensation parameters will be described.

2. Get Light Compensation Parameters and Light Compensation

[161] The video encoding device is configured to calculate illumination compensation parameters corresponding to the plurality of areas. Light compensation is performed in units of areas or sub-blocks in the target block using a plurality of light compensation parameters. The video encoding device is configured to determine, for each of the plurality of pre-built sample areas (reference samples) in the current picture and the reference picture, to be used to obtain the compensation parameters. Reference samples in the current image are selected from pre-built samples adjacent to the target block. For the area separated from the left border of the target block, pre-constructed samples adjacent to the left side of the target block are selected as reference ones. For the area separated from the upper boundary of the target block, pre-constructed readings adjacent to the upper side of the target block are selected as reference ones.

[162] The reference samples in the reference image may be selected using the motion vectors of the control points of the target block or one or more motion vectors derived from the motion vectors of the control points. Next, a method for obtaining illumination compensation parameters and performing illumination compensation according to various types of separation of the target block will be described in detail.

[163] 2-1. When the target block is split into 4 areas

[164] First, a case will be described as an example where a target block having a horizontal length and a vertical length equal to W and H is divided in the horizontal and vertical directions, and thus consists of four regions. The video encoding device is configured to determine prebuilt samples (reference samples) in the current picture and the reference picture to be used to obtain compensation parameters for each area.

Fig. 13 is an exemplary diagram illustrating a method for determining reference samples to be used to obtain compensation parameters. For the upper left region in the target block, the video encoder selects T0 to Tw/2-1 near the upper side of the upper left region and L0 to LH/2-1 near the left side of the upper left region in the current picture as reference samples, which will be used to obtain compensation parameters. Then, a reference area corresponding to the top left area of the reference image is determined using the representative motion vector of the top left area, and then the prebuilt samples adjacent to the top and left sides of the reference area are selected as the reference samples. In particular, the representative motion vector of the upper left region may be the motion vector MV0 of the reference point of the upper left corner of the target block.

[166] For the upper right region, the video encoder is configured to select Tw/2 to Tw-1 near the upper side of the upper right region as at least some reference samples to be used to obtain the compensation parameters. In particular, L0 - LH /2-1 adjacent to the left side of the target block may be additionally selected as reference samples. Since there are no prebuilt samples adjacent to the left side of the upper right region, L0 to LH/2-1 on the left side of the target block can be filled in as reference samples on the left side of the upper right region.

[167] Then, the video encoding device is configured to determine the reference area corresponding to the top right area in the reference picture using the representative motion vector of the top right area, and selects prebuilt samples adjacent to the top side of the reference area as the reference samples used to obtain compensation parameters. When L0 to LH/2-1 adjacent to the left side of the target block are further selected, the video coding apparatus may further select prebuilt samples adjacent to the left side of the reference area as reference samples. Specifically, the upper right area representative motion vector may be the upper right corner reference point motion vector MV1 of the target block.

[168] For the lower left region, the video encoder selects LH/2 to LH-1 near the left side of the lower left region as at least some reference samples. Additionally, T0 to Tw/2-1 adjacent to the top side of the target block can be selected. Then, the video encoding device is configured to determine the reference area corresponding to the bottom left area in the reference image using the representative motion vector of the bottom left area, and select prebuilt samples adjacent to the left side of the reference area as the reference samples. When T0 to Tw/2-1 adjacent to the top side of the target block is further selected, the video encoder may be configured to further select prebuilt samples adjacent to the top side of the reference area. Here, the representative motion vector of the upper left region may be the motion vector MV1 of the reference point of the lower left corner of the target block.

[169] Once the reference samples for each of the upper left, upper right, and lower left regions in the target block are determined, the video encoding device is configured to obtain the illumination compensation parameter for each region by appropriately inputting the reference samples selected from the reference image, and the reference samples selected from the current image, in x and y Equations 15. The illumination compensation parameter for the lower right region can be obtained from the illumination compensation parameters of the upper left, upper right, and lower left regions. For example, the light compensation parameter of the lower right region may be set to the average of the light compensation parameters of the upper right and lower left regions.

[170] Alternatively, the video encoding device may be configured to determine a representative motion vector for the lower right region and obtain a light compensation parameter for the lower right region using the representative motion vector. The representative motion vector for the lower right region may be the motion vector of the control point in the lower right corner of the target block. The video encoder may be configured to obtain the motion vector of the target block's lower right corner reference point by replacing the coordinate (W-1, H-1) of the lower right corner sample (lower right corner sample of the lower right region) in the target block for (x, y) in Equation 10 or 11.

[171] There are no pre-built samples adjacent to the bottom right of the target block. Therefore, Tw/2 to Tw-1 and LH/2 to LH-1 adjacent to the top and left sides of the target block are used as reference samples to obtain the compensation parameters. Then, a reference area in the reference image is determined based on the motion vector of the control point of the lower right corner of the target block, and pre-built samples adjacent to the top and left sides of the reference area are selected as reference samples. The video encoder obtains the illumination compensation parameter for the lower right region using Equation 15 with selected reference samples.

[172] In the above example, a representative motion vector of each area is used to determine the reference samples in the reference image to obtain the compensation parameter. Reference point motion vectors (i.e. parameters for affine motion prediction) are used as an example. However, the embodiments are not limited to this. The representative motion vector of each region may be a motion vector obtained from the motion vectors of the control points of the target block.

[173] In an exemplary embodiment, the representative motion vector of each region may be a motion vector of a predetermined position obtained from motion vectors of control points of the target block. For example, the 4x4 affine motion vectors of the top left, top right, bottom left, and bottom right sub-blocks in a target block previously generated by affine motion prediction can be used as representative top left, top right, bottom left, and bottom right motion vectors, respectively. As another example, the affine motion vector at the center point of each of the top left, top right, bottom left, and bottom right regions may be used as a representative motion vector of each region. The affine motion vector at the center point of each region can be generated by replacing (x, y) with the position of the center of each region in Equation 10 or 11.

[174] In another exemplary embodiment, the average of the affine motion vectors of the sub-blocks in each region may be used as a representative motion vector of the corresponding region. As another way of determining the reference samples to be used in the reference image to obtain the compensation parameters, instead of the representative motion vector of each area, the affine motion vectors of the sub-blocks in each area can be used. In other words, the 4x4 reference blocks at the positions indicated by the affine motion vectors of the 4x4 sub-blocks in each region may be defined in the reference image, and the prebuilt pixels adjacent to the top and left sides of the 4x4 reference blocks may be selected as reference pixels.

[175] FIG. 14 is an exemplary diagram for describing a method for performing light compensation using light compensation parameters for four areas of the target block. Each area includes one or more sub-blocks, which are the units in which the affine transformation is performed. In the example in FIG. 14, the upper left region TL, upper right region TR, lower left region BL, and lower right region BR of the target block each each include four subblocks (4×4 subblock). The video encoding device is configured to perform illuminance compensation for predictive samples in the corresponding area (predictive samples generated by affine motion prediction) using the illuminance compensation parameter of each area. Each sub-block may be compensated using the illumination compensation parameter of the area to which the sub-block belongs. For example, four sub-blocks included in the upper left region are compensated using the illumination compensation parameters (A0, b0) of the upper left region.

[176] In some other embodiments, the video encoder may be configured to interpolate the illumination compensation parameters that respectively correspond to sub-blocks in each region using the illumination compensation parameters of the four regions TL, TR, BL, and BR, and then perform illumination compensation on each sub-block using an interpolated light compensation parameter. As described above, the illumination compensation parameter of each area can be obtained from the representative motion vector of the area. However, the representative motion vector may not reflect the motion of all sub-blocks in the respective area.

[177] For example, the illumination compensation parameters of the respective areas can be obtained from the motion vectors of control points of the top left, top right, bottom left, and bottom right corners of the target block, or affine motion vectors from 4x4 top left, top right, bottom left, and bottom right subblocks. target block. In particular, the movement of the top left, top right, bottom left, and bottom right 4x4 subblocks may have a greater effect on obtaining the illumination compensation parameters. Accordingly, the video encoding device may be configured to obtain the illumination compensation parameters of the respective sub-blocks by interpolation using the illumination compensation parameters of the respective areas.

[178] FIG. 15 is an exemplary diagram illustrating a method for interpolating the illumination compensation parameters of sub-blocks using the illumination compensation parameters of the respective areas. (A0, b0) - (A3, b3) - illumination compensation parameters of the corresponding areas, obtained from the motion vectors of the control points of the upper left corner, upper right corner, lower left corner and lower right corner of the target block, respectively, or from the 4x4 affine motion vectors of the upper left, top right, bottom left, and bottom right subblocks of the target block. In addition, it is assumed that (A0, b0) is the illumination compensation parameter of the top left 4x4 subblock in the top left region of the target block. Similarly, it is assumed that (A1, b1) to (A3, b3) are the light compensation parameters of the top right 4x4 sub-box in the upper right region, the light compensation parameter of the lower left 4x4 sub-box in the lower left region, and the light compensation parameter of the lower right 4x4 sub-box in lower right area. In other words, it is assumed that (A0, b0) - (A3, b3) are the compensation parameters of the four corner sub-blocks of the target block.

[179] The video coding device interpolates the compensation parameters of other sub-blocks using the compensation parameters of the four corner sub-blocks. Bilinear interpolation using four corner subblocks or linear interpolation can be performed. For example, the sub-blocks in the first row of the target block may be interpolated using (A0, b0) and (A2, b2). The sub-blocks in the first column of the target block can be interpolated using (A0, b0) and (A2, b2). In this way, the compensation parameters of all sub-blocks are obtained.

[180] In particular, the filter factor or weight factor used for interpolation may be set according to the difference between the positions of the corner subblocks and the positions of the target subblock (or the distance between the corner subblocks and the target subblock). A smaller difference in position is given more weight.

[181] Alternatively, the filter coefficient or weight used for interpolation may be set in accordance with the difference between the motion vectors used to obtain the corner subblock compensation parameters (the motion vectors of the upper left, upper right, lower left, and lower right reference points). corners, or affine motion vectors of the top left, top right, bottom left, and bottom right 4x4 subblocks of the target block) and the affine motion vector of the target subblock. A smaller difference in motion vectors is assigned more weight.

[182] 2.2 When the target block is divided into two areas

[183] Even when the target block is divided into two areas, the illumination compensation parameter of each area can be obtained in the same manner as when the target block is divided into four areas. For example, referring to FIG. 16, the video encoding device selects, as reference samples in the current picture to obtain the upper region compensation parameter, the prebuilt samples T0 to Tw-1 adjacent to the upper side of the upper region, and the prebuilt samples L0 to LH/2-1 adjacent with the left side of the top area. Then, the video encoding apparatus is configured to determine the reference area in the reference image using the representative motion vector of the top area, and select prebuilt samples adjacent to the top and left sides of the reference area as the reference samples.

[184] The representative motion vector of the top region may be the motion vector of the control point of the upper left corner of the target block. Alternatively, it may be a motion vector derived from the motion vectors of the control points of the target block. For example, an affine motion vector of the top left 4x4 subblock in the top region may be used, or an affine motion vector corresponding to the center point of the top region may be used. In another example, the average of the affine motion vectors of the sub-blocks in each region may be used as a representative motion vector of the corresponding region.

[185] The video coding device is configured to select as reference samples in the current picture to obtain a lower area compensation parameter of T0 - Tw-1 pre-built samples adjacent to the top side of the target block and LH/2 - LH-1 pre-built samples, adjacent to the left side of the bottom area. Then, the video encoding apparatus is configured to determine the reference area in the reference image using the representative motion vector of the bottom area, and select prebuilt samples adjacent to the top and left sides of the reference area as the reference samples.

[186] The representative motion vector of the lower region may be the motion vector of the control point of the lower left corner of the target block. Alternatively, it may be a motion vector derived from the motion vectors of the control points of the target block. For example, an affine motion vector of the lower left 4x4 subblock in the lower region may be used, or an affine motion vector corresponding to the center point of the lower region may be used. As another example, the average of the affine motion vectors of the sub-blocks in the bottom region can be used as a representative motion vector of the bottom region.

[187] In the above example, a representative motion vector of each area is used as a method for determining reference samples in a reference image used to obtain a compensation parameter. However, the affine motion vectors of the sub-blocks in each region may be used instead of the representative motion vector of each region. That is, for each region, 4x4 reference blocks at the positions indicated by the affine motion vectors of the 4x4 sub-blocks can be determined in the reference image, and prebuilt pixels adjacent to the top and left sides of the 4x4 reference blocks can be selected as reference pixels.

[188] For each region, the video encoding device is configured to input the reference samples selected in the reference picture and the reference samples selected in the current picture in x and y in Equation 15, respectively, to obtain compensation parameters for the respective regions. In addition, the video encoding apparatus performs illumination compensation for prediction samples in the corresponding area (prediction samples generated by affine motion prediction) using the illumination compensation parameter of each area. Each sub-block may be compensated using the illumination compensation parameter of the area to which the sub-block belongs.

[189] Alternatively, the video coding device may be configured to interpolate the illumination compensation parameters that correspond to sub-blocks using the illumination compensation parameters of the respective areas, and to perform illumination compensation for each sub-block using the interpolated parameters, as in the case where the target block is divided into four areas.

[190] Interpolation may be applied when each region's representative motion vector, used to obtain the compensation parameter, is associated with a specific position in each region. For example, the control point motion vectors of the top left and bottom left corners of the target block or the affine motion vectors of the top left and bottom left 4x4 subblocks of the target block can be used as representative motion vectors. Specifically, the compensation parameters of the two regions are considered as the illumination compensation parameters of the upper left and lower left corner subblocks TL and BL in the target block, as shown in FIG. 17.

[191] The video encoding device may be configured to interpolate the compensation parameter of the target sub-block by applying a weighting factor to the compensation parameters (A0, b0) of the upper region and the compensation parameters (A2, b2) of the lower region. In particular, the weight may be assigned according to the distance between the top left and bottom left corner subblocks in the target block and the target subblock. Alternatively, it can be set according to the difference between the motion vectors used to obtain the compensation parameters of the two regions (the motion vectors of the target block's upper left and lower left corner reference point, or the affine motion vectors of the target block's upper left and lower left 4x4 subblock) and the affine motion vector of the target subblock. As the difference between the motion vectors decreases, the weight increases.

[192] Even when the target block is vertically divided into two regions, the same method as when the target block is horizontally divided into two regions can be used. In particular, the representative motion vectors of the two regions (left region and right region) may be, for example, the motion vectors of the control point of the upper left corner and the upper right corner of the target block, or the affine motion vectors of the motion vectors of the upper left and upper right 4x4 subblocks of the target block. Alternatively, the affine motion vector of the center point of each region or the average of the affine motion vectors of the sub-blocks of each region may be used as a representative motion vector of each region. Other operations are the same as in the case of the horizontal split described above, or can be explicitly derived from it, so their further detailed description is omitted.

[193] 2.3 When the target block is split into more than 4 subblocks

[194] As described above, the target block may be divided into 2m and 2n blocks (where m and n are integers greater than or equal to 2) in the horizontal and vertical directions. For example, the target block may be divided into sub-blocks, which are units in which affine motion prediction is performed. This case is illustrated in FIG. 18. Referring to FIG. 18, the video encoding device outputs the illumination compensation parameters for the upper left region TL, upper right region TR, lower left region BL, and lower right region BR of the target block using the method described in section 2.1.

[195] For areas other than these four areas, the illumination compensation parameters of the remaining areas can be obtained by applying the interpolation described in section 2.1. Alternatively, for the remaining regions, the illumination compensation parameters can be obtained in the same manner as for the upper left region TL, upper right region TR, lower left region BL, and lower right region BR. For example, for area A, the video coding device selects prebuilt samples located on the top side of area A and adjacent to the target block, and prebuilt samples located on the left side of area A and adjacent to the target block, as reference pixels in the current image. In addition, reference samples in the reference image are selected using the motion vector of the control point in the upper left corner of area A, the affine motion vector of the 4x4 sub-block located in the upper left corner of area A, or the affine motion vector of the center point of area A. Light compensation parameter of area A derived using the selected reference readings.

[196] The illumination compensation technique described above is a technique using a motion vector that can be obtained using pre-built samples and affine motion prediction. Accordingly, the video decoding apparatus may be configured to perform light compensation in the same manner as the video encoding apparatus. The video encoding device is not required to signal additional information for the illumination compensation process to the video decoding device.

[197] The illumination compensation technique described above can be used in conjunction with other coding tools to compensate for motion of a target block in units of subblocks, in addition to affine motion prediction. For example, this can be applied to bidirectional optical flow. The video encoding device and the video decoding device may be configured to generate prediction samples for each sub-block by performing bi-directional optical flow on a sub-block basis for the target block, and may further be configured to perform the above illumination compensation on the prediction samples in the target block. The illumination compensation technique applied to the bidirectional optical flow is the same as the illumination compensation technique described above, except for the following.

[198] In the illumination compensation technique described above, the motion vectors of the control point of the target block or the affine motion vectors of sub-blocks are used to separate the target block and calculate a representative motion vector of each area separated from the target block. However, in the case of bidirectional optical flow, there is neither a reference point motion vector nor an affine motion vector. Accordingly, when the illumination compensation technique is applied to a bidirectional optical flow, the optical flows (vx, vy) of the subblocks are used instead of the motion vectors of the control points or the affine motion vectors of the subblocks.

[199] V. Syntax Signaling

[200] Another aspect of the present disclosure relates to a method for signaling high-level syntaxes for controlling the on/off of the various means described above. The above-described affine motion prediction, affine motion prediction sample-based correction, adaptive motion vector resolution, and illumination compensation are coding tools used to improve video coding efficiency. However, for specific content such as, for example, screen content, the aforementioned various encoding tools may not improve compression performance. Accordingly, signaling based on a coding unit whether to apply each coding means or deciding based on a coding unit whether to use each coding means may rather degrade coding efficiency or increase computational complexity. The present disclosure discloses a signaling technique for efficiently managing the coding tools described above.

[201] To encode a sequence of images, the video encoding device is configured to encode the first high-level syntax element of the sequence indicating whether a plurality of encoding tools are disabled. The first element of the high-level syntax may be encoded in the sequence header, i.e. SPS. In particular, the plurality of coding tools includes affine motion prediction and also includes at least one of sample-by-sample correction for affine motion prediction samples, adaptive motion vector resolution, and illumination compensation.

[202] When the first element of the high-level syntax indicates that the entire set of encoding tools is disabled, no encoding tools are applied to the encoding of the sequence. In other words, the encoding tools described above are not applied to any images in the sequence. When the first high-level syntax element indicates that the encoding tool set is not disabled, the encoding tool set may be used to encode the corresponding sequence. For example, affine motion prediction can be applied to all images in a sequence. Accordingly, the video encoding apparatus may encode a coding unit flag indicating whether or not affine motion prediction encoding has been applied to each of the coding units and transmit it to the video decoding apparatus.

[203] When the coding unit flag indicates that affine motion prediction has been applied to the corresponding coding unit, the video decoding apparatus may be configured to perform the above-described affine motion prediction on the coding unit. When the first high-level syntax element indicates that the coding tool set is not disabled, the video encoding device may be configured to encode one or more second high-level syntax elements indicating whether each coding tool is valid in at least a portion of the coding tool set at the sequence level, or at the image level.

[204] In some embodiments, to encode one or more second high-level syntax elements, the video encoder encodes, for each encoding tool, a sequence level flag indicating whether the corresponding encoding tool is allowed in the SPS. When the sequence level flag indicates that the corresponding encoding tool is not allowed, the corresponding encoding tool is not used to encode images belonging to the sequence. In addition, the video encoding device is configured to encode a picture level flag indicating whether a corresponding encoding tool is allowed in the picture header (ie, PPS) of each of the pictures belonging to the sequence, depending on the value of the sequence level flag.

[205] For example, when the sequence level flag indicates that the corresponding encoding tool is allowed at the sequence level, an image level flag is encoded to indicate whether the corresponding encoding tool is allowed at the image level. When the first high-level syntax element indicates that the entire set of encoding tools is disabled, or the sequence level flag indicates that the corresponding encoding facility is not allowed at the sequence level, the image level flag is not separately encoded. In particular, the video decoding apparatus is configured to automatically set the picture level flag to a value indicating that the corresponding encoding tool is not allowed.

[026] Image level flags for coding means other than affine motion prediction may be transmitted depending on the image level flag for affine motion prediction. When the image level flag for affine motion prediction indicates that affine motion prediction is not enabled at the image level, the image level flags for other encoding tools are not encoded, and other encoding tools are not allowed at the image level. When the image level flags for other encoding tools are not included in the bitstream, the video decoding apparatus is configured to set each image level flag to a value indicating that the corresponding encoding tools are not allowed in the image level.

[207] The video decoder may be configured to set at the image level whether to enable the corresponding encoding means from the image level flag signaled from the video encoding device or the image level flag whose value is set automatically. The second high-level syntax element for affine motion prediction may be omitted. In particular, whether affine motion prediction is enabled at the sequence level is controlled by the first high-level syntax. The video decoder is configured to decode the first high-level syntax from the video encoder and then determine whether to enable sequence-level affine motion prediction.

[208] In accordance with the syntax structure described above, the video encoding device may be configured to adaptively determine whether to allow the use of coding tools at the sequence level and the picture level. In addition, the video decoder may be configured to adaptively determine whether to enable sequence level and picture level encoding tools by decoding the syntaxes transmitted from the video encoding device.

[209] In some other embodiments, to encode one or more second high-level syntax elements, the image level flag of the above embodiment may be replaced with a slice level flag. In other words, when the sequence level flag indicates that the corresponding encoder is allowed at the sequence level, the video encoder encodes in the slice header a slice level flag to indicate whether the corresponding encoder is allowed at the slice level. When the sequence level flag indicates that the corresponding encoding tool is not allowed, the slice level flag is not encoded. The video encoding device may be configured to adaptively determine whether to allow the use of encoding tools at the sequence level and the slice level. The video decoder may be configured to determine whether to allow the use of each tool at the slice level with the slice level flag.

[210] In some other embodiments, the sequence level flag may be omitted to encode one or more second high-level syntax elements. A picture level flag or a slice level flag is signaled depending on the first element of the high level syntax. When the first element of the high-level syntax indicates that multiple encoding tools are allowed, an image level flag or a slice level flag is encoded in the image header or slice header.

[211] It should be understood that the exemplary embodiments described above may be implemented in many different ways. The functions described in one or more examples may be implemented in hardware, software, firmware, or any combination thereof. It should be understood that the functional components described here have been labeled "module" to further emphasize their implementation independence.

[212] Various methods or functions described herein may be implemented as instructions stored on a non-volatile storage medium that can be read and executed by one or more processors. A non-transitory recording medium includes, for example, all types of recording devices in which data is stored in a form readable by a computer system. For example, non-transitory recording media includes media such as erasable programmable read-only memory (EPROM), flash drives, optical drives, magnetic hard disks, and solid state drives (SSDs).

[213] While the exemplary embodiments have been described for illustrative purposes, those skilled in the art will appreciate that various modifications and changes are possible without departing from the spirit and scope of the embodiments. Exemplary embodiments have been described for brevity and clarity. Accordingly, one of ordinary skill in the art will appreciate that the scope of the embodiments is not limited to the embodiments expressly described above, but includes the claims and their equivalents.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority of Korean Patent Applications Nos. 10-2019-0070633, 10-2019-0082010 and 10-2020-0072332, filed respectively on June 14, 2019, July 8, 2019, and June 15, 2020, all contents of which are incorporated herein by reference.

Claims (47)

1. A video decoding method for decoding a sequence of encoded images on a block basis, comprising: establishing at the sequence level whether affine motion prediction is enabled by decoding the first high-level syntax element from the bitstream; determining at the image level whether each of the at least one coding tool is enabled by decoding one or more second high level syntax elements for each of the at least one coding tool from the bitstream depending on the first high level syntax element, wherein at least at least one coding tool includes a sample-by-sample adjustment of affine motion prediction samples; when setting at the sequence level indicates that affine motion prediction is enabled, determining a plurality of motion vectors of checkpoints of the target block encoded using affine motion prediction, and generating affine prediction samples for each of the sub-blocks in the target block by performing affine motion prediction in a unit of sub-blocks with using motion vectors of control points; and when, based on the setting at the image level, it is determined that affine motion prediction sample correction per sample is allowed for the current image containing the target block, modifying the sample values in the affine prediction samples by motion compensation according to the position of the sample in each of the sub-blocks based on the motion vectors of the checkpoints target block. 2. The method of claim 1, wherein the first high level syntax element is decoded from a sequence header. 3. The method of claim 2, wherein decoding the one or more second high-level syntax elements to incrementally update affine motion prediction samples includes: when the first high-level syntax element indicates that affine motion prediction is enabled, decoding a sequence level flag indicating whether per-sample adjustment of affine motion prediction samples is enabled from the sequence header; and depending on the sequence level flag, decoding a picture level flag indicating whether per-sample correction of affine motion prediction samples is allowed from a picture header of each of the pictures belonging to the sequence. 4. The method of claim 3, wherein when the first high-level syntax element indicates that affine motion prediction is not enabled, or the sequence level flag indicates that per-sample correction is not enabled, the picture level flag is set to a value indicating that per-sample correction not allowed, without decoding from the image header. 5. The method of claim 1, wherein when the first high-level syntax element indicates that affine motion prediction is not enabled, any of the affine motion prediction and at least one coding means are not enabled for all pictures in the sequence. 6. The method of claim 1, wherein modifying the sample values for the affine prediction samples includes: generating a motion offset resulting from a difference between a reference position in the target sub-block to be modified and the center of the target sub-block, using motion vectors of control points of the target block; generating horizontal and vertical gradients with respect to the position of the reference in the target sub-block; generating a sample bias for the affine prediction sample corresponding to the position of the sample in the target sub-block using horizontal and vertical gradients and a motion bias; and adding a sample offset to the affine prediction sample value corresponding to the position of the sample. 7. The method of claim 6, wherein the motion offset generated for a reference position in the target sub-block is reused as a motion offset for the same reference position in another sub-block of the target block. 8. A video decoding device for decoding a sequence of encoded images on a block basis, comprising: decoder configured: set at the sequence level whether affine motion prediction is enabled by decoding the first high-level syntax element from the bitstream; and determine at the image level whether each of the at least one coding tool is allowed by decoding one or more second high level syntax elements for each of the at least one coding tool from the bitstream depending on the first high level syntax element, wherein at least at least one coding tool includes a sample-by-sample adjustment of affine motion prediction samples; and a predictor configured when setting at the sequence level indicates that affine motion prediction is enabled, determine a plurality of motion vectors of target block checkpoints encoded using affine motion prediction, and generate affine prediction samples for each of the sub-blocks in the target block by performing affine motion prediction in a unit of sub-blocks using control point motion vectors, wherein, based on the setting at the image level, it is determined that affine motion prediction sample correction per sample is allowed for the current image containing the target block, the predictor is configured to modify the sample values in the affine prediction samples by motion compensation according to the position of the sample in each of the vector-based sub-blocks. movement of control points of the target block. 9. The apparatus of claim 8, wherein the first high level syntax element is decoded from the sequence header. 10. The apparatus of claim 9, wherein the decoder is configured to decode one or more second high-level syntax elements to incrementally update affine motion prediction samples by: when the first high-level syntax element indicates that affine motion prediction is enabled, decoding a sequence level flag indicating whether per-sample adjustment of affine motion prediction samples is enabled from the sequence header; and depending on the sequence level flag, decoding a picture level flag indicating whether per-sample correction of affine motion prediction samples is allowed from a picture header of each of the pictures belonging to the sequence. 11. The apparatus of claim 10, wherein when the first high-level syntax element indicates that affine motion prediction is not enabled or the sequence level flag indicates that per-sample correction is not enabled, the decoder is configured to set the picture level flag to a value indicating that per-sampling adjustment is not allowed, without decoding from the picture header. 12. The apparatus of claim 8, wherein the predictor is configured to modify the affine prediction sample values by: generating a motion offset resulting from a difference between a reference position in the target sub-block to be modified and the center of the target sub-block, using motion vectors of control points of the target block; generating horizontal and vertical gradients with respect to the position of the reference in the target sub-block; generating a sample offset for the affine prediction sample corresponding to the position of the sample in the target sub-block using horizontal and vertical gradients and a motion bias; and adding a sample offset to the affine prediction sample value corresponding to the position of the sample. 13. The apparatus of claim 12, wherein the motion offset generated for a reference position in the target sub-block is reused as a motion offset for the same reference position in another sub-block of the target block. 14. A method for encoding a sequence of encoded images on a block basis, including: encoding, in the sequence header, the first high-level syntax element in the sequence level, to indicate whether both affine motion prediction and at least one coding tool are disabled, wherein the at least one coding tool includes a sample-wise adjustment of affine prediction samples movement; depending on the first high level syntax element, encoding one or more second high level syntax elements indicating the establishment of a sequence level or a picture level with respect to the encoding resolution of each of the at least one encoding tool; when the first high-level syntax element indicates that both affine motion prediction and at least one encoder are not disabled, determining the motion vectors of the checkpoints for the target block to be encoded and generating affine prediction samples for each of the sub-blocks by performing a sub-block affine motion prediction using control point motion vectors; and when sample-by-sample adjustment of affine motion prediction samples is enabled for the current image containing the target block, modifying the sample values in the affine prediction samples by motion compensation according to the position of the sample in each of the sub-blocks based on the motion vectors of the checkpoints of the target block. 15. The method of claim 14, wherein encoding the at least one second high-level syntax element for sample-wise updating of affine motion prediction samples includes: when the first high-level syntax element indicates that affine motion prediction is enabled, encoding a sequence level flag indicating whether per-sample adjustment of affine motion prediction samples is enabled in the sequence header; and depending on the sequence level flag, encoding a picture level flag indicating whether per-count correction is allowed in the picture header of each of the pictures belonging to the sequence. 16. The method of claim 14, wherein modifying the sample values for the affine prediction samples includes: generating a motion offset resulting from a difference between a reference position in the target sub-block to be modified and the center of the target sub-block, using motion vectors of control points of the target block; generating horizontal and vertical gradients with respect to the position of the reference in the target sub-block; generating a sample bias for the affine prediction sample corresponding to the position of the sample in the target sub-block using horizontal and vertical gradients and a motion bias; and adding a sample offset to the affine prediction sample value corresponding to the position of the sample.

Images