RU2802844C2 - Method and device for encoding and decoding video using inter-prediction - Google Patents

Abstract

FIELD: means for video coding and decoding.

SUBSTANCE: it is determined at the sequence level whether affine motion prediction is enabled by decoding the first high-level syntax element. It is determined, at the image level, whether each of the at least one encoding tool is enabled by decoding one or more second high-level syntax elements. At least one coding tool includes a count-by-count correction of affine motion prediction samples. When the affine motion prediction is enabled, the predictor defines two or three vectors of checkpoints movement of the target block encoded using affine motion prediction are determined, and affine prediction samples are generated for each of the subblocks in the target block by performing affine motion prediction in a unit of subblocks. When sample-by-sample adjustment of affine motion prediction samples is enabled, the predictor modifies the sample values in the affine prediction samples by motion compensation.

EFFECT: increased efficiency of video coding.

13 cl, 22 dwg

Description

Field of technology to which the invention relates

[1] The present disclosure relates to video encoding and decoding and, in particular, to an encoding tool for improving inter-prediction compression efficiency.

State of the art

[2] Since the volume of video data is larger than the volume of speech data or still image data, storing or transmitting video data without processing for compression requires a lot of hardware resources, including memory. Accordingly, when storing or transmitting video data, the video data is typically compressed using an encoder for storage or transmission. The decoder then receives the compressed video data, decompresses and plays back the video data. Compression procedures for such video include H.264/AVC and High Efficiency Video Coding (HEVC), which improves encoding efficiency over H.264/AVC by approximately 40%.

[3] However, image size, resolution and frame rate are gradually increasing, and accordingly the amount of data that needs to be encoded is also increasing. Accordingly, a new compression procedure having better coding efficiency and higher image quality than the existing compression procedure is required.

[4] In video coding, predictive coding is mainly used to improve video compression efficiency. There is intra-prediction for predicting the target block to be encoded based on the previously reconstructed samples in the current image, and inter-prediction for predicting the current block using the previously reconstructed reference image. In particular, inter-prediction is widely used for video encoding because it exhibits superior compression performance compared to intra-prediction.

[5] The present disclosure provides an encoding tool for improving the compression efficiency of existing inter-prediction.

[6] In particular, the present disclosure relates to an encoding tool for improving inter-prediction compression efficiency and, in one aspect, relates to an encoding tool capable of compensating for various object motions, including translational motion.

Disclosure of the invention

[7] In accordance with one aspect of the present disclosure, a video decoding method is provided for decoding a sequence of encoded images on a block-by-block basis. The method includes determining, at the sequence level, whether affine motion prediction is enabled by decoding a first high-level syntax element from the bitstream; determining, at the image level, whether each of the at least one encoding tool is enabled by decoding one or more second high-level syntax elements for each of the at least one encoding tool from the bitstream depending on the first high-level syntax element, wherein the at least one encoding tool includes sample-by-sample adjustment of the affine motion prediction samples; when the sequence-level setting indicates that affine motion prediction is enabled, determining a plurality of target block reference point motion vectors encoded using the affine motion prediction, and generating affine prediction samples for each of the subblocks in the target block by performing affine motion prediction in the unit of subblocks with using motion vectors of control points; and when, based on the image-level determination, it is determined that sample-by-sample adjustment of the affine motion prediction samples is allowed for the current image containing the target block, modifying the sample values in the affine prediction samples by motion compensation in accordance with the position of the sample in each of the sub-blocks based on the reference motion vectors points of the target block.

[8] According to another aspect of the present disclosure, a video decoding apparatus for decoding a sequence of encoded images on a block-by-block basis is provided. The apparatus includes a decoder configured to: set, at the sequence level, whether affine motion prediction is enabled by decoding a first high-level syntax element from the bitstream; and determining, at the image level, whether each of the at least one encoding tool is enabled by decoding one or more second high-level syntax elements for each of the at least one encoding tool from the bitstream depending on the first high-level syntax element. In particular, the at least one encoding tool includes sample-by-sample updating of the affine motion prediction samples.

[9] The apparatus also includes a predictor configured to determine when a sequence-level assertion indicates that affine motion prediction is enabled, a plurality of target block reference point motion vectors encoded using the affine motion prediction, and generate affine prediction samples for each sub-block in target block by performing affine motion prediction on a sub-block basis using control point motion vectors. In response to the identification based on the determination at the image level that sample-by-sample adjustment of affine motion prediction samples is allowed for the current image containing the target block, the predictor is configured to modify the values of the affine prediction samples by motion compensation according to the position of the sample in each of the sub-blocks based on the vectors movements of control points of the target block.

[10] In accordance with another aspect of the present disclosure, a method for encoding a sequence of encoded images on a block-by-block basis is provided. The method includes encoding, in a sequence header, a first high-level syntax element in the sequence level to indicate whether both affine motion prediction and at least one encoding tool are disabled, wherein the at least one encoding tool includes a count-by-count adjustment of affine motion prediction samples; depending on the first high-level syntax element, encoding one or more second high-level syntax elements indicating the establishment of a sequence level or an image level with respect to the resolution of each of the at least one encoding tool; when the first high-level syntax element indicates that both the affine motion prediction and at least one encoding tool are not disabled, determining motion vectors of reference points for the target block to be encoded, and generating affine prediction samples for each of the sub-blocks by performing a sub-block affine motion prediction using motion vectors of control points; and when sample-by-sample adjustment of the affine motion prediction samples is enabled for the current image containing the target block, modifying the sample values in the affine prediction samples by motion compensation according to the position of the sample in each of the sub-blocks based on the motion vectors of the target block's reference points.

Brief description of drawings

[11] FIG. 1 is an exemplary block diagram of a video encoding apparatus configured to implement the procedures of the present disclosure.

[12] FIG. 2 shows an example of a block division structure using a QTBTTT structure.

[13] FIG. 3 shows a variety of intra-prediction modes as an example.

[14] FIG. 4 shows the neighboring blocks around the current block as an example.

[15] FIG. 5 is an exemplary block diagram of a video decoding apparatus configured to implement the technology of the present disclosure.

[16] FIG. 6 is an exemplary diagram illustrating the dual prediction optical flow concept provided by the present disclosure.

[17] FIG. 7 is an exemplary diagram illustrating a method for obtaining a gradient for sampling a block boundary in bidirectional optical flow.

[18] FIG. 8A-8B and 9 are exemplary diagrams illustrating the affine motion prediction provided by the present disclosure.

[19] FIG. 10 is an exemplary diagram illustrating a method for obtaining fusion candidates for affine motion prediction from translational motion vectors of neighboring blocks.

[20] FIG. 11A to 11C are exemplary diagrams illustrating a method for obtaining an illumination compensation parameter according to an illumination compensation embodiment provided by the present disclosure.

[21] FIG. 12A-12B are exemplary diagrams illustrating a method for dividing a target block into multiple areas according to another embodiment of illumination compensation provided by the present disclosure.

[22] FIG. 13 is an exemplary diagram illustrating a method for determining reference samples to be used to obtain compensation parameters when a target block is divided into four areas, according to another illumination compensation embodiment provided by the present disclosure.

[23] FIG. 14 and 15 are exemplary diagrams illustrating a method for performing illumination compensation using illumination compensation parameters for four areas when a target block is divided into four areas, according to another embodiment of illumination compensation provided by the present disclosure.

[24] FIG. 16 and 17 are exemplary diagrams illustrating a method for performing illumination compensation for each area when a target block is divided into two areas, according to another embodiment of illumination compensation provided by the present disclosure.

[25] FIG. 18 are exemplary diagrams illustrating a method for performing illumination compensation for each area when a target block is divided into a plurality of areas more than four areas, according to another embodiment of illumination compensation provided by the present disclosure.

Detailed description

[26] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. It should be noted that when adding numerals to constituent elements in the respective drawings, the same numerals designate the same elements even if the elements are shown in different drawings. Moreover, in the following description of the present disclosure, detailed description of known functions and configurations included herein will be omitted so as not to detract from the subject matter of the present disclosure.

[27] FIG. 1 is an exemplary block diagram of a video encoding device configured to implement the technology of the present disclosure. Next, the encoding device and the elements of the device will be described with reference to FIG. 1.

[28] The video encoding apparatus includes a block separator 110, a predictor 120, a subtractor 130, a transformer 140, a quantizer 145, a reordering module 150, an entropy encoder 155, an inverse quantizer 160, an inverse transformer 165, an adder 170, a loop filter module 180, and a memory. 190.

[29] Each element of a video encoding device may be implemented as hardware or software, or a combination of hardware and software. The functions of the corresponding elements may be implemented as software, and the microprocessor may be implemented to perform the software functions corresponding to the corresponding elements.

[30] A single video consists of one or more sequences containing multiple images. Each image is divided into multiple regions, and coding is performed for each region. For example, one image is divided into one or more fragments and/or sections. In particular, one or more tiles may be defined as a group of tiles. Each tile and/or slice is divided into one or more coding tree units (CTUs). And each CTU is divided into one or more coding units (CU) in a tree structure. Information applied to each CU is encoded as CU syntax, and information applied to CUs included in one common CTU is encoded as CTU syntax. In addition, information applied to all blocks in one slice as a whole is encoded as slice head syntax, and information applied to all blocks composing one or more images is encoded in picture parameter set (PPS) or image header.

[31] In addition, information referenced by a sequence consisting of a plurality of images in general is encoded in a sequence parameter set (SPS). In addition, information applied to one tile or group of tiles in the overall tile image may be encoded as tile or tile group header syntax. The syntaxes included in the SPS, PPS, slice header, and tile or tile group header may be referred to as high-level syntaxes.

[32] The block separator 110 is configured to determine a coding tree unit (CTU) size. CTU size information (CTU size) is encoded as SPS set or PPS set syntax and transmitted to the video decoding device. In addition, the block divider 110 is configured to divide each image constituting a video into a plurality of CTUs having a predetermined size, and then recursively divide the CTUs using a tree structure. In a tree structure, a leaf node serves as a coding unit (CU), which is the basic coding unit.

[33] The tree structure can be a quadtree (QT), in which a node (or parent node) is divided into four subnodes (or child nodes) of the same size, a binary tree (BT), in which a node is divided into two subnode, a TernaryTree (TT) in which a node is divided into three subnodes in a 1:2:1 ratio, or a structure formed by a combination of two or more of a QT structure, a BT structure, and a TT structure. For example, a structure may be used , including a quadtree and a binary tree (QTBT), or a structure including a quadtree and a binary tree, a ternary tree (QTBTTT).Here, the BTTT may be collectively referred to as a multiple type tree (MTT).

[34] In FIG. Figure 2 shows the tree structure of the QTBTTT partition as an example. As shown in FIG. 2, CTU can be initially divided into QT structure. QT splitting can be repeated until the split block size reaches the minimum leaf node MinQTSize block size allowed in the QT structure. The first flag (QT_split_flag) indicating whether each node of the QT structure is divided into four lower-level nodes is encoded by the entropy encoder 155 and signaled to the video decoding device. If a leaf node of a QT structure is equal to or less than the maximum root node block size (MaxBTSize) allowed in a BT structure, it may be further divided into one or more BT structures or TT structures. The BT structure and/or the TT structure may have multiple division directions. For example, there may be two directions, namely the direction in which the node block is split horizontally and the direction in which the block is split vertically.

[35] As shown in FIG. 2, when MTT splitting begins, a second flag (mtt_split_flag) indicating whether nodes are split, a flag indicating the direction of splitting (vertical or horizontal) in case of splitting, and/or a flag indicating the type of splitting (binary or ternary) are encoded by entropy encoder 155 and are signaled to the video decoding device. Alternatively, before encoding the first flag (QT_split_flag) indicating whether each node is split into 4 bottom layer nodes, a CU split flag (split_cu_flag) indicating whether the node is split may be encoded. When the value of the CU split flag (split_cu_flag) indicates that no split is being performed, the node block becomes the leaf node in the split tree structure and serves a coding unit (CU), which is the basic coding unit. When the value of the CU split flag (split_cu_flag) indicates that splitting is complete, the video encoding device begins encoding the flags in the manner described above, starting with the first flag.

[36] When QTBT is used as another example of a tree structure, there can be two types of partitioning, which are a type of horizontal partitioning of a block into two blocks of equal size (i.e. symmetrical horizontal partitioning) and a type of vertical partitioning of a block into two blocks of equal size (i.e. symmetrical vertical division). A split flag (split_flag) indicating whether each node of the BT structure is divided into a lower layer block and split type information indicating the split type are encoded by the entropy encoder 155 and signaled to the video decoding device. An additional type of splitting a node block into two asymmetric blocks is possible. The asymmetrical division type may include a block division type into two rectangular blocks with an aspect ratio of 1:3 or a node block diagonal division type.

[37] In addition, CUs can have different sizes according to the division of the CTU into QTBT or QTBTTT. In the following, the block corresponding to the CU (i.e., leaf node QTBTTT) to be encoded or decoded is called the “current block”. Because QTBTTT partitioning is used, the shape of the current block can be square or rectangular. Predictor 120 predicts the current block to generate a prediction block. Predictor 120 includes intra-predictor 122 and inter-predictor 124.

[38] The intra prediction unit 122 is configured to predict samples in the current block using samples (reference samples) located around the current block in the current image including the current block. There are many intra-prediction modes according to the prediction directions. For example, as shown in FIG. 3, the plurality of intra-prediction modes may include two omnidirectional modes, which include planar mode and DC mode, and 65 directional modes. The neighboring samples and the equation used are defined differently for each prediction mode.

[39] Intra predictor 122 may be configured to determine the intra prediction mode to be used when encoding the current block. In some examples, intra-predictor 122 may be configured to encode the current block using multiple intra-prediction modes and select an appropriate intra-prediction mode from the tested modes to use. For example, intra-predictor 122 may be configured to calculate rate-distortion values using rate-distortion analysis of several tested intra-prediction modes and select the intra-prediction mode that has the best rate-distortion characteristics among the tested modes.

[40] The intra predictor 122 is configured for one intra prediction mode from a plurality of intra prediction modes, and predicts the current block using neighboring pixels (reference pixels) and an equation determined according to the selected intra prediction mode. Information about the selected intra-prediction mode is encoded by the entropy encoder 155 and transmitted to the video decoding device. The inter-predictor 124 is configured to predict a prediction block for the current block by motion compensation. In addition, inter-predictor 124 is configured to search for a block most similar to the current block in a reference picture that was encoded and decoded earlier than the current picture, and generates a prediction block for the current block using the found block.

[41] The inter-predictor is then configured to generate a motion vector corresponding to the offset between the current block in the current image and the prediction block in the reference image. In general, motion estimation is performed on the luminance component, and the motion vector calculated based on the luminance component is used for both the luminance component and the chrominance component. Motion information including reference image information and motion vector information used for predicting the current block is encoded by the entropy encoder 155 and transmitted to the video decoding apparatus.

[42] Inter-predictor 124 may be configured to perform interpolation based on a reference image or reference block to improve prediction accuracy. In other words, the subsamples between two consecutive integer samples are interpolated by applying the filter coefficients to a plurality of consecutive integer samples, including the two integer samples. When a search operation for a block most similar to the current block is performed on an interpolated reference image, the motion vector can be expressed at a fractional unit precision level rather than an integer unit precision level.

[43] The precision or resolution of the motion vector can be set differently for each target coded area, for example, for each unit such as slice, tile, CTU or CU. When such adaptive motion vector resolution is applied, motion vector resolution information that should be applied to each target region must be transmitted for each target region. For example, when the target region is a CU, information about the motion vector resolution applied to each CU is transmitted. The motion vector resolution information may be information indicating the accuracy of the motion vector difference, which will be described later.

[44] Inter-predictor 124 may be configured to perform inter-prediction using dual prediction. In bidirectional prediction, inter-predictor 124 uses two reference pictures and two motion vectors representing the positions of blocks most similar to the current block in the corresponding reference pictures. The inter-predictor 124 is configured to select a first reference picture and a second reference picture from reference picture list 0 (RefPicList0) and reference picture list 1 (RefPicList1), respectively, to search for blocks similar to the current block in the corresponding reference pictures, and generate the first a support block and a second support block. Inter-predictor 124 is then configured to generate a prediction block for the current block by averaging or weighting the first reference block and the second reference block. Inter-predictor 124 is then configured to transmit motion information, including information about two reference pictures and two motion vectors used to predict the current block, to encoder 150.

[45] Specifically, RefPicList0 may consist of images preceding the current image in display order among the restored images, and RefPicList1 may consist of images following the current image in display order among the restored images. However, the embodiments are not limited to this. Pre-reconstructed images following the current image in display order may be further included in RefPicList0, and conversely, pre-reconstructed images preceding the current image may be further included in RefPicList1.

[46] Various techniques can be used to minimize the number of bits required to encode motion information. For example, when the reference image and motion vector of the current block are the same as the reference image and motion vector of the adjacent block, the motion information of the current block can be transmitted to the decoding device by encoding information to identify the adjacent block. This method is called "merge mode".

[47] In the merge mode, the inter-predictor 124 selects a predetermined number of merge candidate blocks (hereinafter referred to as “merge candidates”) from among the neighboring blocks of the current block. As shown in FIG. 4, all or part of the left block L, the top block A, the top right block AR, the bottom left block BL and the top left block AL, which are adjacent to the current block in the current image, can be used as neighboring blocks to obtain merge candidates. In addition, a block located in a reference picture (which may be the same or different from the reference picture used to predict the current block) other than the current picture in which the current block is located can be used as a merge candidate. For example, a merged block that is in the same position as the current block, or blocks adjacent to a merged block in the reference image, can further be used as fusion candidates.

[48] Inter-predictor 124 configures a merge list including a predetermined number of merge candidates using such neighboring blocks. The inter-predictor 124 is configured to select a merge candidate for use as traffic information for the current block from among the merge candidates included in the merge list, and generate merge index information for identifying the selected candidates. The generated merge index information is encoded by the encoder 155 and transmitted to the decoding device.

[49] Another way to encode motion information is the AMVP mode. In AMVP mode, inter-predictor 124 is configured to obtain predicted motion vector candidates for the motion vector of the current block using neighboring blocks of the current block. All or part of a left block L, an upper block A, an upper right block AR, a lower left block BL and an upper left block AL that are adjacent to the current block in the current image in FIG. 2 can be used as neighboring blocks used to obtain predicted motion vector candidates. In addition, a block located in a reference picture (which may be the same or different from the reference picture used to predict the current block) other than the current picture including the current block can be used as neighboring blocks used to obtain predicted candidate motion vectors. For example, a registered block that is in the same position as the current block, or blocks adjacent to a registered block in the reference image, may be used.

[50] The inter-predictor 124 is configured to obtain predicted motion vector candidates using the motion vectors of neighboring blocks and determine a predicted motion vector for the motion vector of the current block using the predicted motion vector candidates. The motion vector difference is then calculated by subtracting the predicted motion vector from the motion vector of the current block.

[51] The predicted motion vector can be obtained by applying a predetermined function (eg, a function for calculating the median, average value, etc.) to the predicted motion vector candidates. In particular, the video decoding device also "knows" the predefined function. Since the neighboring blocks used to obtain the predicted motion vector candidates have already been encoded and decoded, the video decoder also already knows the motion vectors of the neighboring blocks. Accordingly, the video encoding apparatus does not need to encode information to identify predicted motion vector candidates. Therefore, in this case, motion vector difference information and reference image information used for predicting the current block are encoded.

[52] The predicted motion vector can be determined by selecting any one of the predicted motion vector candidates. Specifically, information for identifying the selected predicted motion vector candidate is further encoded together with motion vector difference information and reference image information, which are to be used for predicting the current block.

[53] Subtractor 130 is configured to subtract a prediction block generated by intra-predictor 122 or inter-predictor 124 from the current block to generate a residual block. Converter 140 may be configured to divide a residual block into one or more subblocks and apply a transform to one or more subblocks, thereby converting the residual values of the transform blocks from the reporting domain to the frequency domain. In the frequency domain, transformed blocks are called coefficient blocks or transform blocks containing one or more transform coefficient values. The two-dimensional transformation kernel can be used for transformation, and the one-dimensional transformation kernels can be used for horizontal transformation and vertical transformation, respectively. The transform kernels may be based on discrete cosine transform (DCT), discrete sine transform (DST), or the like.

[54] Converter 140 may be configured to convert residual signals in a residual block using the entire size of the residual block as a conversion unit. Alternatively, the residual block may be divided into a plurality of subblocks, and the residual signals in the subblocks may be converted using the subblock as a conversion unit.

[55] The converter 140 may be configured to individually convert the residual block in the horizontal direction and the vertical direction. Various types of transformation functions or transformation matrices can be used for transformation. For example, pairs of transform functions for a horizontal transform and a vertical transform may be defined as a multiple transform set (MTS). Converter 140 may be configured to select one pair of transform functions having the greatest conversion efficiency in the MTS, and transform the residual block in the horizontal and vertical directions, respectively. Information (mts_idx) about the pair of transform functions selected in the MTS is encoded by the entropy encoder 155 and transmitted to the video decoding device.

[56] Quantizer 145 is configured to quantize transform coefficients output from transformer 140 using quantization parameters, and output the quantized transform coefficients to entropy encoder 155. For some blocks or frames, quantizer 145 may be configured to directly quantize the corresponding residual block without transform. . Quantizer 145 may be configured to apply different quantization factors (scale values) according to the positions of the transform coefficients in the transform block. A matrix of quantized coefficients applied to the two-dimensionally arranged quantized transform coefficients may be encoded and signaled to a video decoding device.

[57] Reordering module 150 may be configured to reorder coefficient values for the quantized residual value. The reordering module 150 may be configured to convert a two-dimensional array of coefficients into a one-dimensional sequence of coefficients by scanning the coefficients. For example, reordering module 150 may be configured to scan coefficients from a forward current (DC) coefficient to a high-frequency domain coefficient using a zigzag scan or a diagonal scan to output a one-dimensional sequence of coefficients. Depending on the size of the transform unit and the intra-prediction mode, vertical scanning, in which a two-dimensional array of coefficients is scanned in the column direction, or horizontal scanning, in which two-dimensional block coefficients are scanned in the drain direction, can be used instead of zigzag scanning. In other words, the scanning mode used can be determined among zigzag scanning, diagonal scanning, vertical scanning and horizontal scanning according to the size of the transformation unit and the intra-prediction mode.

[58] Entropy encoder 155 is configured to encode one-dimensional quantized transform coefficients output from reordering unit 150 using various encoding procedures such as context-sensitive adaptive binary arithmetic code (CABAC) and exponential Golomb code to generate a bit stream. The entropy encoder 155 is configured to encode information such as CTU size, CU partition flag, QT partition flag, MTT partition type, and MTT partition direction, which are associated with block partitioning, so that the video decoding apparatus can partition the block in the same way as in the encoding apparatus. video.

[59] In addition, the entropy encoder 155 is configured to encode prediction type information indicating whether the current block is encoded by intra-prediction or inter-prediction, and encodes intra-prediction information (i.e., intra-prediction mode information) or inter-prediction information (fusion index for the fusion mode, reference image index information, and motion vector difference for the AMVP mode) according to the prediction type. The entropy encoder 155 also encodes quantization-related information, that is, quantization parameter information and quantization matrix information.

[60] The inverse quantizer 160 is configured to inversely quantize the quantized transform coefficients output from the quantizer 145 to generate transform coefficients. The inverse converter 165 is configured to convert the conversion coefficients output from the inverse quantizer 160 from the frequency domain to the spatial domain, and to reconstruct the residual block.

[61] In addition, the adder 170 is configured to add the reconstructed residual block to the prediction block generated by the predictor 120 to recover the current block. The samples in the reconstructed current block are used as reference samples when performing intra-prediction of the next block. Loop filter module 180 is configured to filter the reconstructed samples to reduce blocking artifacts, ringing artifacts, and blur artifacts generated due to block-by-block prediction and transform/quantization. Loop filter module 180 may include at least a deblocking filter 182, an adaptive sample offset (SAO) filter 184, and an adaptive loop filter (ALF) 186.

[62] A deblocking filter 182 is configured to filter the boundary between reconstructed blocks to remove blocking artifact caused by block encoding/decoding, and an SAO filter 184 performs additional filtering on the deblocking filtered video. The SAO filter 184 is a filter used to compensate for the difference between the reconstructed sample and the original sample caused by lossy encoding, and performs filtering such that it adds a corresponding offset to each reconstructed sample. ALF 186 is configured to perform filtering on a target sample to be filtered by applying filter coefficients to the target sample and adjacent samples of the target sample. ALF 186 may be configured to divide the samples included in the image into predefined groups, and then determine a single filter to be applied to the corresponding group to perform filtering differentially in each group. Information about the filter coefficients to be used for the ALF may be encoded and transmitted to the video decoding device.

[63] The reconstructed blocks filtered through the loop filter unit 180 are stored in the memory 190. When all the blocks in one image are reconstructed, the reconstructed image can be used as a reference image for inter-prediction of blocks in an image to be subsequently encoded.

[64] FIG. 5 is an exemplary functional block diagram of a video decoding apparatus configured to implement the technology of the present disclosure. Next, the video decoding device and elements of the device will be described with reference to FIG. 5. The video decoding apparatus may include an entropy decoder 510, a reordering module 515, an inverse quantizer 520, an inverse transformer 530, a predictor 540, an adder 550, a loop filter module 560, and a memory 570.

[65] Similar to the video encoding apparatus of FIG. 1, each element of a video decoding apparatus may be implemented as hardware, software, or a combination of hardware and software. In addition, the function of each element may be implemented in software, and a microprocessor may be implemented to execute the software function corresponding to each element.

[66] The entropy decoder 510 is configured to determine the current block to be decoded by decoding the bit stream generated by the video encoding apparatus and extracting block division related information and extracting prediction information and residual signal information and the like. necessary to restore the current block. In addition, entropy decoder 510 is configured to extract CTU size information from a Sequence Parameter Set (SPS) or Picture Parameter Set (PPS), determines the CTU size, and splits the image into CTUs of a certain size. The decoder is then configured to determine the CTU as the topmost level, that is, the root node, of the tree structure, and retrieve CTU partitioning information to partition the CTU using the tree structure.

[67] For example, when a CTU is split using the QTBTTT structure, the first flag (QT_split_flag) associated with the QT split is retrieved to split each node into four sublayer nodes. For a node corresponding to a leaf QT node, a second flag (MTT_split_flag) and split direction (vertical/horizontal) and/or split type (binary/ternary) information related to the MTT split are retrieved to split the corresponding leaf node in the MTT structure. Accordingly, each node below the leaf node QT is recursively split into a BT or TT structure.

[68] As another example, when a CTU is split using a QTBTTT structure, a CU split flag (split_cu_flag) may be retrieved indicating whether the CU should be split. When the corresponding block is split, the first flag (QT_split_flag) can be retrieved. In a split operation, zero or more recursive MTT splits can occur for each node after zero or more recursive QT splits. For example, a CTU may be configured to directly undergo MTT partitioning without QT partitioning, or to repeatedly undergo only QT partitioning.

[69] As another example, when a CTU is split using a QTBT structure, the first flag (QT_split_flag) associated with the QT split is retrieved, and each node is split into four lower-level nodes. Then, a split flag (split_flag) is retrieved indicating whether the node corresponding to the QT leaf node is split further into BT and split direction information.

[70] Once the current block to be decoded is determined by partitioning in the tree structure, the entropy decoder 510 is configured to extract prediction type information indicating whether the current block is intra-predicted or inter-predicted. When the prediction type information indicates intra-prediction, the entropy decoder 510 is configured to extract a syntax element for intra-prediction information (intra-prediction mode) for the current block. When the prediction type information indicates inter-prediction, the entropy decoder 510 is configured to extract a syntax element for the inter-prediction information, that is, information indicating the motion vector and the reference image referenced by the motion vector.

[71] The entropy decoder 510 is also configured to extract information about the quantized transform coefficients of the current block as quantization-related information and residual signal information. Reordering module 515 may be configured to change the sequence of one-dimensional quantized transform coefficients entropy decoded by entropy decoder 510 into a two-dimensional coefficient array (ie, block) in the reverse order of the coefficient scan performed by the video encoding device.

[72] Inverse quantizer 520 is configured to inversely quantize the quantized transform coefficients using a quantization parameter. The inverse quantizer 520 may be configured to apply different quantization factors (scaling values) to the quantized transform coefficients located in two dimensions. Inverse quantizer 520 may be configured to perform inverse quantization by applying a matrix of quantization coefficients (scale values) from the video encoder to a two-dimensional array of quantized transform coefficients.

[73] The inverse converter 530 is configured to inversely convert the inversely quantized transform coefficients from the frequency domain to the spatial domain to restore the residual signals, thereby generating a reconstructed residual block for the current block. In addition, when MTS is applied, the inverter 530 is configured to determine transform functions or transform matrices to be applied in the horizontal and vertical directions, respectively, using MTS information (mts_idx) transmitted from the video encoding device, and use the determined transform functions to inverse conversion of the conversion coefficients in the conversion block in the horizontal and vertical directions.

[74] Predictor 540 may include intra predictor 542 and inter predictor 544. Intra predictor 542 is activated when the prediction type of the current block is intra prediction, and inter predictor 544 is activated when the prediction type of the current block is inter-prediction. prediction. The intra predictor 542 is configured to determine the intra prediction mode of the current block among the plurality of intra prediction modes based on the syntax element for the intra prediction mode extracted from the entropy decoder 510, and predict the current block using reference samples around the current block in accordance with the mode intra-predictions.

[75] The inter-predictor 544 is configured to determine the motion vector of the current block and a reference image referenced by the motion vector using an intra-prediction mode syntax element extracted from the entropy decoder 510, and predict the current block based on the motion vector and the reference Images. The adder 550 is configured to reconstruct the current block by adding a residual block output from the inverter and a prediction block output from the inter-predictor or intra-predictor. The samples in the reconstructed current block are used as reference samples in intra-prediction of the block to be decoded next.

[76] Loop filter module 560 may include at least one of deblocking filter 562, SAO filter 564, and ALF filter 566. A deblocking filter 562 filters the boundary between reconstructed blocks to remove blocking artifacts caused by block-by-block decoding. The SAO filter 564 is configured to perform filtering by adding the reconstructed block after deblocking filtering to the corresponding offsets to compensate for the difference between the reconstructed sample and the original sample caused by lossy encoding. ALF 566 is configured to perform filtering of a target sample to be filtered by applying filter coefficients to the target sample and adjacent samples of the target sample. ALF 566 can be configured to separate samples in an image into predefined groups, and then determine a single filter to be applied to the corresponding group to perform filtering differentially in each group. The filter factor ALF is determined based on the filter factor information decoded from the bit stream.

[77] The reconstructed block filtered through the loop filter unit 560 is stored in the memory 570. When all blocks in one image are reconstructed, the reconstructed image can be used as a reference image for inter-prediction of blocks in the image to be subsequently encoded.

[78] The following disclosure relates to encoding tools for improving inter-prediction compression characteristics that can be controlled by video encoding device inter-predictor 124 and video decoding device inter-predictor 544. As used herein, the term "target block" may have the same meaning as the terms "current block" or "coding unit (CU)" used above, or may mean a partial area of the CU.

[79] I. Bidirectional optical flow

[80] Bidirectional optical flow is a method of additionally compensating for motions of reports predicted using bidirectional motion prediction, provided that the samples or objects composing the video move at a constant speed and that the sample values change little.

[81] FIG. 6 is an example diagram illustrating the basic concept of BIO. It is assumed that the bidirectional motion vectors MV0 and MV1 pointing to the corresponding regions (i.e. reference blocks) most similar to the target block to be encoded in the current image have been determined in the reference images Ref0 and Ref1 by (normal) bidirectional motion prediction for target block. The two motion vectors have values representing the motion of the entire target block. In the example in FIG. 6,_P0 is the sample in the reference image Ref0 indicated by the motion vector MV0 and corresponding to the sample P in the target block, and P₁ represents the sample in the reference image Ref₁, indicated by the motion vector MV1 and corresponding to the sample P in the target block.

[82] In addition, it is assumed that the motion for reference P in FIG. 6 is slightly different from the general movement of the target block. For example, when an object located at sample A in Ref0 in FIG. 6 moves to reference B in Ref₁ through sample P in the target block of the current image, sample A and sample B can have values very similar to each other. Moreover, in this case, the point in Ref₀, most similar to the sample P in the target block is not P₀, indicated by the bidirectional motion vector MV0, and the reference A, shifted from_P0 to a predefined displacement vector (v_xt₀,v_yt₁). The point in Ref1 most similar to sample P in the target block is not P₁, indicated by the bidirectional motion vector MV1, and the reference B, which has been shifted from P₁ to a given displacement vector (-v_xt₀, -v_yt₁). In particular, t₀ and t₁ indicate distances Ref₀ and Ref₁ along the time axis with respect to the current image, respectively, and are calculated based on the Picture Order Counter (POC). Next (v_x, v_y) is called "optical flow".

[83] When predicting the sample value P of the current block in the current image, the two reference samples A and B provide a more accurate prediction than the reference samples P ₀ and P ₁ indicated by the bidirectional motion vectors MV ₀ and MV ₁ . Specifically, I ⁽⁰⁾ (i,j) denotes the value of the sample in the reference image Ref0 indicated by the motion vector MV ₀ as corresponding to the sample (i,j) in the target block, and I ⁽¹⁾ (i,j) denotes the value a sample in the reference image Ref ₁ indicated by the motion vector MV ₁ as corresponding to a sample (i,j) in the target block.

[84] The sample value A in the reference image Ref ₀ indicated by the motion vector BIO (v _x , v _y ) as corresponding to the sample in the target block can be defined as I ⁽⁰⁾ (i+ v _x t ₀ , j +v _y t ₀ ), and the sample value B in the reference image Ref1 can be defined as I ⁽¹⁾ (i - v _x t ₁ , j - v _y t ₁ ). Here, when the linear approximation is performed using only the first order term of the Taylor series, A and B can be expressed as Equation 1.

Equation 1

[85] where I_x ^(k) and I_y ^(k) (k = 0, 1) - gradient values in horizontal and vertical directions at position (i, j) in Ref₀ and Ref₁. Besides,_t0 And_t1 denote the time axis distances of Ref0 and Ref1 relative to the current image, respectively, and are calculated based on POC: t₀ = POC(current) - POC(Ref0) and t₁ = POC(Ref1) - POC (current).

[86] The bidirectional optical flow (v _x , v _y ) of each sample in the block is defined as the solution that minimizes Δ, which is defined as the difference between sample A and sample B. Δ can be determined in Equation 2 using the linear approximation from A and B obtained from Equation 1.

Equation 2

[87] For simplicity, the position (i, j) of the sample is omitted in each term of Equation 2.

[88] To realize more reliable optical flow estimation, the motion is assumed to be locally consistent with neighboring samples. For BIO, the motion vector for the sample (i, j) that is currently to be predicted, the differences Δ in Equation 2 are taken into account for all samples (i', j') present in a mask Ω of a certain size centered on the sample (i , j). That is, the optical flow for the current sample (i, j) can be defined as a vector that minimizes the objective function Φ(v _x , v _y ), which is the sum of squared differences Δ[i', j'] obtained for the corresponding samples in the mask Ω , as shown in Equation 3.

Equation 3

[89] Bidirectional optical flow according to the present disclosure can be applied in the case where one of the two reference pictures used for bidirectional prediction precedes the current picture in the display order, the other follows the current picture, and the distances from the two reference pictures to the current picture are equal to each other. each other, that is, the differences in picture order counter (POC) between each reference picture and the current picture are equal to each other. Therefore, t ₀ and t ₁ can be ignored.

[90] Moreover, the bidirectional optical flow according to the present disclosure can only be applied to luminance components. The bidirectional optical flow of the present disclosure is performed on a sub-block basis, rather than a pixel-based basis, for the target block to which the bidirectional prediction is applied. In the following description, for simplicity, it is assumed that the subblock, which can have various sizes of 2x2, 4x4 and 8x8, has a size of 4x4.

[91] Before performing the optical flow, the inter-predictor 124 of the video encoding device generates two reference blocks for the target block using the above-mentioned bidirectional prediction. The first reference block of the two reference blocks is a block consisting of predicted samples generated from the reference image Ref0 using the first motion vector MV ₀ of the target block, and the second reference block is a block consisting of predicted samples generated from the reference image Ref ₁ using the motion vector MV ₁ .

[92] Inter-predictor 124 is configured to calculate (v _x , v _y ), called optical flow, for each of the 4x4 sub-blocks constituting the target block using the horizontal and vertical gradient values of the predicted samples in the first reference block and in the second reference block . The optical flow (v _x , v _y ) is determined such that the difference between the predicted samples from the reference image Ref ₀ and the predicted samples from the reference image Ref ₁ is minimized. Inter-predictor 124 is configured to obtain a sample offset to modify bidirectional predicted samples for the 4x4 subblock using (v _x , v _y ) computed for the 4x4 subblock and gradients of the predicted samples in the 4x4 subblock.

[93] Specifically, inter-predictor 124 calculates the horizontal and vertical gradients of sample values at position (i, j) using Equation 4.

Equation 4

[94] where k is 0 or 1, and I ⁽⁰⁾ (i,j) and I ⁽¹⁾ (i,j) denote the sample values at position (i,j) in the first reference block and in the second reference block, respectively. Additionally, Shift1 is a value derived from the bit depth of the luma component, for example shift1 = max(6, bitDepth-6).

[95] To obtain the gradient of a sample located at the boundary of each reference block, samples are required beyond the boundary of the first reference block and the second reference block. Accordingly, as shown in FIG. 7, each reference block is expanded one column left and right and one row up and down. To reduce the amount of computation, each sample in the extended part can be filled with a sample or an entire sample at the nearest position in the reference block. Additionally, gradients at a sample position outside the boundary of each reference block can be complemented by gradients corresponding to samples at a nearby position.

[96] Inter-predictor 124 is configured to calculate S1, S2, S3, S5 and S6 corresponding to autocorrelation and cross-correlation gradients using horizontal and vertical gradients in a 6x6 window spanning a 4x4 subblock, as shown in FIG. 7.

Equation 5

[97] where Ω denotes the window spanning the subblock. Moreover, as shown in Equation 6 below, Ψx(i,j) denotes the sum of the horizontal gradient values at position (i,j) in the first reference block and the second reference block, Ψy(i,j) denotes the sum of the vertical gradient values at position (i,j) in the first reference block and in the second reference block, and θ(i,j) denotes the difference between the sample value at position (i,j) in the second reference block and the sample value at position (i,j) in the first support block.

Equation 6

[98] where n _a and n _b are values derived from the bit depth and have the values min(1, bitDepth-11) and min(4, bitDepth-8).

[99] Inter-predictor 124 is configured to calculate optical flows (v _x , v _y ) for a 4x4 subblock using Equation 7 based on S1, S2, S3, S5 and S6.

Equation 7

[100] where, , And . is a rounding function, and .

[101] The sample offset for modifying the predicted sample at each sample position (x, y) in a 4x4 sub-block in the target block can be calculated using the optical flow calculated for the 4x4 sub-block and the gradient values at the sample position (x, y) as shown in Equation 8. In Equation 8, rnd() is a rounding operation.

Equation 8

[102] The inter-predictor 124 is configured to generate a final predicted sample pred(x, y) using the sample offset b(x, y) at position (x, y) and the predicted samples I ⁽⁰⁾ (x,y) and I ⁽¹⁾ (x,y) in the first reference block and in the second reference block as in Equation 9.

Equation 9

[103] where the offset is Max(3, 15-BitDepth) and _Ooffset is the value for the rounding operation and is half the offset.

[104] As described above, the bidirectional optical flow method uses sample values predicted using motion information (two motion vectors and two reference images) used for bidirectional prediction. Therefore, the inter-predictor 544 of the video decoding device can also be configured to implement bidirectional optical flow in the same manner as the video encoding device, using motion information (motion vectors, reference pictures) for bidirectional prediction received from the video encoding device. This does not require the video encoding device to signal additional information for the bidirectional optical flow process to the video decoding device.

[105] II. Affine motion prediction

[106] The above inter-prediction is a motion prediction reflecting a translational motion model. In other words, it is a method for predicting motion in the horizontal direction (x-axis direction) and vertical direction (y-axis direction). However, in practice, in addition to translational motion, there can be various types of motions, such as rotation, zooming in, and zooming out. One aspect of the present disclosure provides affine motion prediction capable of covering these different types of motions.

[107] FIG. 8A-8B are exemplary diagrams illustrating affine motion prediction. For affine motion prediction, there can be two types of models. One is a model that uses the motion vectors of two reference points of the upper left corner and the upper right corner of the target block that is currently to be encoded, that is, four parameters, as shown in FIG. 8A. Another model is a model using the motion vectors of three control points of the upper left corner, upper right corner and lower left corner of the target block, that is, six parameters, as shown in FIG. 8B.

[108] The four-parameter affine model is expressed by Equation 10. The motion at the position (x, y) of the sample in the target block can be calculated by Equation 10. Here, the position of the above left sample of the target block is assumed to be (0, 0).

Equation 10

[109] The six-parameter affine model is expressed by Equation 11. The motion at the reference position (x, y) in the target block can be calculated by Equation 11.

Equation 11

[110] where (mv _0x , mv _0y ) is the movement vector of the control point in the upper left corner, (mv _1x , mv _1y ) is the movement vector of the control point in the upper right corner, and (mv _2x , mv _2y ) is the movement vector of the control dots in the lower left corner. W is a constant determined by the horizontal length of the target block, and H is a constant determined by the vertical length of the target block.

[111] Affine motion prediction can be performed for each sample in the target block using the motion vector calculated from Equation 10 or 11. Alternatively, to reduce computational complexity, prediction can be performed for each sub-block separated from the target block, as shown in fig. 9. For example, the subblock size could be 4x4, 2x2 or 8x8. In the exemplary embodiment below, affine motion prediction for a target block is performed based on a 4x4 subblock. This example is provided for convenience of explanation only, and the present disclosure is not limited to it.

[112] In sub-block-based affine motion prediction, the motion vector (affine motion vector) of each sub-block is calculated by substituting the center position of each sub-block into (x, y) in Equation 10 or 11. Specifically, the center position may be the actual center point of the sub-block or lower right position of the center point reference. For example, in the case of a 4x4 subblock in which the bottom left reference coordinates are (0, 0), the position of the center of the subblock could be (1.5, 1.5) or (2, 2). A predicted block for each sub-block is generated using the affine motion vector (mv _x , mv _y ) of the sub-block.

[113] Motion vectors (mv _x , mv _y ) can have an accuracy of 1/16th of a sample. In particular, the motion vector (mv _x , mv _y ) calculated from equation 10 or 11 can be rounded to 1/16 units. Adaptive motion vector resolution can be applied to affine motion prediction as in conventional inter-prediction. In particular, information about the resolution of the motion vector of the target block, that is, the accuracy of the motion vector, is transmitted for each target block.

[114] Affine motion prediction can be performed not only on the luminance component, but also on the chrominance component. In the case of a 4:2:0 video format, when affine motion prediction for the luma component is performed based on a 4x4 sub-block, affine motion prediction for the chrominance component may be performed based on a 2x2 sub-block. The motion vector (mv _x , mv _y ) of each chroma component subblock can be obtained from the motion vector of the corresponding luminance component. Alternatively, the affine motion prediction subblock size for the chroma component may be the same as for the luminance component.

[115] When affine motion prediction for the luma component is performed based on the 4x4 subblock, affine motion prediction for the chrominance component is also performed based on the 4x4 subblock. In this case, since a 4x4 sub-block for the chrominance component corresponds to four 4x4 sub-blocks for the luma component, the motion vector (mv _x , mv _y ) for the chrominance component sub-block can be calculated by calculating the average of the motion vectors for the chrominance component from the four corresponding luma component sub-blocks.

[116] The video encoding apparatus performs intra-prediction, inter-prediction (translational motion prediction), affine motion prediction, and the like. and calculates the rate-distortion (RD) cost to select the optimal prediction method. To perform affine prediction, the inter-predictor 124 of the video encoding device determines which of two types of models to use and determines two or three control points according to the determined type. Inter-predictor 124 calculates a motion vector (mv _x , mv _y ) for each of the 4x4 subblocks in the target block using the motion vectors of the control points. It then performs motion compensation on the reference image on a sub-block basis using the motion vector (mv _x ,mv _y ) of each sub-block to generate a predicted block for each sub-block in the target block.

[117] The entropy encoder 155 of the video encoding device is configured to encode associated affine syntax elements, including a flag indicating whether the affine motion prediction is applied to the target block, type information indicating the type of the affine model, and motion information indicating the motion vector of each reference point, and transmitting this to the video decoding device. Type information and motion information about reference points may be transmitted when affine motion prediction is performed, and as many motion vectors of reference points as determined according to the type information may be transmitted. In addition, when adaptive motion vector resolution is applied, motion vector resolution information about the affine motion vector of the target block is transmitted.

[118] The video decoding apparatus is configured to determine the affine model type and motion vectors of the reference points using the signaled syntax elements and calculate the motion vector (mv _x , mv _y ) for each 4x4 sub-block in the target block using equation 10 or 11. When motion vector resolution information about the affine motion vector of the target block is reported, the motion vector (mv _x , mv _y ) is adjusted to the accuracy determined by the motion vector resolution information through operations such as rounding.

[119] The video decoding apparatus is configured to generate a predicted block for each sub-block by performing motion compensation on the reference image using a motion vector (mv _x , mv _y ) for each sub-block. To reduce the number of bits required to encode the motion vectors of the reference points, a method that is used in the above-described conventional inter-prediction (translational motion prediction) can be applied.

[120] As an example, in the fusion mode, the inter-predictor 124 of the video encoding device is configured to obtain the motion vector of each reference point from neighboring blocks of the target block. For example, inter-predictor 124 is configured to generate a list of merge candidates by obtaining a predetermined number of merge candidates from neighboring samples L, BL, A, AR, and AL of the target block illustrated in FIG. 4. Each of the merge candidates included in the list corresponds to a pair of motion vectors from two or three control points.

[121] First, the inter-predictor 124 is configured to obtain a fusion candidate from the motion vectors of the affine-mode-predicted neighboring blocks' control points among the neighboring blocks. In some embodiments, the number of merge candidates obtained from adjacent blocks predicted in affine mode may be limited. For example, inter-predictor 124 may be configured to obtain two merge candidates from adjacent blocks predicted in affine mode: one from L and BL and one from A, AR and AL. Priority can be assigned in the order L and BL and in the order A, AR and AL. When the total number of merge candidates is greater than or equal to 3, the inter-predictor 124 can be configured to obtain the required number of merge candidates from the translation vectors of neighboring blocks.

[122] FIG. 10 is an exemplary diagram illustrating a method for obtaining fusion candidates for affine motion prediction from translational motion vectors of neighboring blocks. The inter-predictor 124 is configured to obtain reference point motion vectors CPMV1, CPMV2, CPMV3 from the neighboring block group {B2, B3, A2}, the neighboring block group {B1, B0} and the neighboring block group {A1, A0}, respectively. For example, the priorities in each group of neighboring blocks may be assigned in the order B2, B3, A2, the order B1 and B0, and the order A1 and A0. In addition, another reference point motion vector CPMV4 is obtained from a co-located block T in the reference image. The inter-predictor 124 is configured to generate as many merge candidates as needed through combinations of two or three control point motion vectors among the four control point motion vectors. Combination priorities are assigned as shown below. The elements in each group are listed in order of the motion vectors of the top left corner, top right corner and bottom left corner control points.

{CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}

[123] The inter-predictor 124 is configured to select a merge candidate from the list of merge candidates and perform affine motion prediction for the target block. When the selected candidate consists of two control point motion vectors, affine motion prediction is performed using a four-parameter model. On the other hand, when the selected candidate consists of three control point motion vectors, affine motion prediction is performed using a six-parameter model. The entropy encoder 155 of the video encoding device is configured to encode index information indicating a merge candidate selected from the merge candidates in the merge candidate list and signal it to the video decoding device.

[124] The entropy decoder 510 of the video decoding apparatus is configured to decode index information signaled from the video encoding apparatus. The video decoding device inter-predictor 544 lists fusion candidates in the same manner as the video encoding device, and performs affine motion prediction using motion vectors of reference points corresponding to the fusion candidate indicated by the index information.

[125] As another example, in AMVP mode, the video encoding device inter-predictor 124 is configured to determine the type of affine model and reference point motion vectors for a target block. Next, the inter-predictor 124 is configured to calculate motion vector differences, which are differences between the actual motion vectors of the target block's reference points and the predicted motion vectors of the corresponding reference points, and transmit the motion vector differences, which respectively correspond to the reference points.

[126] Accordingly, the inter-predictor 124 of the video encoding device configures a list of a predetermined number of affine AMVPs. When the target block is a 4-parameter type, each of the candidates included in the list consists of a pair of motion vectors of two control points. On the other hand, when the target block is a 6-parameter type, each candidate included in the list is composed of a pair of motion vectors of three control points. The list of affine AMVPs can be obtained using the motion vectors of control points or the translational motion vectors of neighboring blocks in a manner similar to the method for generating the list of merge candidates described above.

[127] However, to obtain candidates for inclusion in the affine AMVP list, there may be a limitation of considering only neighboring blocks that rely on the same reference image as the target block among the neighboring blocks in FIG. 4.

[128] In addition, in AMVP mode, the type of affine model of the target block should be taken into account. When the affine model type of the target block is a 4-parameter type, the video encoding device is configured to obtain two reference point motion vectors (the top left corner reference point motion vector and the top right corner reference point motion vector of the target block) using the affine model of the adjacent block. When the affine model type of the target block is a 6-parameter type, the device obtains three control point motion vectors (upper left corner, upper right corner, and lower left control point motion vector of the target block) using the affine model of the neighboring block.

[129] When the adjacent block is a 4-parameter type, two or three control point motion vectors are predicted according to the affine model type of the target block using the two control point motion vectors of the adjacent block. For example, an affine neighbor block model expressed by Equation 10 may be used. In Equation 10, (mv _0x , mv _0y ) and (mv _1x , mv _1y ) are replaced by the motion vectors of the control point at the upper left and upper right corner of the adjacent block, respectively. In particular, the horizontal length of the adjacent block is substituted into W. The predicted motion vector for each target block's control point can be obtained by entering into (x, y) the difference between the position of the corresponding target block's control point and the position of the upper left corner of the adjacent block.

[130] When the adjacent block is a 6-parameter type, two or three control point motion vectors are predicted according to the affine model type of the target block using the three control point motion vectors of the adjacent block. For example, the affine neighbor block model expressed by Equation 11 can be used. In Equation 11, (mv _0x , mv _0y ), (mv _1x , mv _1y ) and (mv _2x , mv _2y ) are replaced by the motion vectors of the control point of the upper left corner, the upper right corner and lower left corner of the adjacent block, respectively. Specifically, W and H are replaced by the horizontal length and vertical length of the adjacent block, respectively. The predicted motion vector for each target block's control point can be obtained by entering into (x, y) the difference between the position of the corresponding target block's control point and the position of the upper left corner of the adjacent block.

[131] The inter-predictor 124 of the video encoding device is configured to select one candidate from the list of affine AMVPs and generate a motion vector difference between the motion vector of each actual reference point and the predicted motion vector of the corresponding reference point of the selected candidate. The entropy encoder 155 of the video encoding device is configured to encode type information indicating the type of the affine model of the target block, index information indicating a candidate selected among the candidates in the AMVP affine list, and a motion vector difference corresponding to each reference point, and to signal it to video decoding device.

[132] The video decoder inter-predictor 544 is configured to determine the affine model type using information signaled from the video encoder and generate a motion vector difference for each reference point. The inter-predictor is then configured to generate an AMVP affine list in the same manner as the video encoder, and select a candidate indicated by the index information transmitted in the AMVP affine list. The inter-predictor 544 of the video decoding device is configured to calculate the motion vector of each reference point by adding the predicted motion vector of each reference point of the selected candidate and the corresponding motion vector difference.

[133] III. Block-by-block adjustment of affine motion prediction samples

[134] Sub-block affine motion prediction for a target block was described above. Another aspect of the present disclosure relates to updating, on a sample-by-sample basis, sample values of predicted samples generated by sub-block affine motion prediction. The motion based on the position of each sample is further compensated in each sub-block, which forms the basis of affine motion prediction.

[135] When the sample values in any sub-block generated as a result of the sub-block affine motion prediction for the target block are equal to I(x, y), the video encoding device calculates the horizontal and vertical gradients g _x (i, j) and g _y (i, j) at each reading position. Equation 12 can be used to calculate gradients.

Equation 12

[136] The sample offset ΔI(i, j) for correcting the prediction sample is calculated by the following equation.

Equation 13

[137] where Δmv(i,j) denotes the motion displacement, that is, the difference between the affine motion vector at sample (i, j) and the affine motion vector at the center position of the subblock, and can be calculated by Equation 10 or 11 depending on the type affine model of the target block. In other words, Δmv(i, j) can be calculated from Equation 10 or 11 by subtracting the motion vector specified when the subblock center position is substituted into (x, y) from the motion vector specified when (i, j) is substituted into ( x, y). In addition, Δmv(i, j) can be calculated from the equation, which is obtained by replacing (x, y) in Equations 11 and 12 with the horizontal and vertical displacement from the sub-block center position to the reference position (i,j) and removing the last term " + mv _0x " and " + mv _0y ". The center position may be the actual center point of the subblock or may be the lower right position of the center point reference.

[138] The motion vector of each target block reference point used to calculate ∆mv(i, j) and the difference between the reference position (i, j) and the subblock center position are the same for all subblocks. Accordingly, the values of Δmv(i, j) can be calculated for only one sub-block, for example, the first sub-block, and can be reused for other sub-blocks.

[139] The technique of the present disclosure is based on the assumption that the object is moving at a constant speed and the change in reference values is smooth. Therefore, the sample variation in the horizontal direction and the magnitude of the sample change in the vertical direction are obtained by multiplying the x component (Δmv _x ) and the y component (Δmv _y ) Δmv(i, j) by the horizontal and vertical sample gradient values, respectively. The offset ΔI(i,j) of the reference is calculated by adding the two reference options. The final value of the predicted count is calculated as follows.

Equation 14

[140] When sample-by-sample correction is applied to the affine motion prediction samples, the inter-predictor 124 of the video encoding device and the inter-predictor 544 of the video decoding device perform the above-described process to correct the sample values of the predicted samples generated by the affine motion prediction. The gradient values are obtained from the predicted samples generated through affine motion prediction, and Δmv(i, j) is obtained from the motion vectors of the target block reference points. Therefore, the video encoding apparatus is not required to signal additional information for the process of the present technique to the video decoding apparatus.

[141] IV. Local illumination compensation for affine motion prediction samples

[142] The local illumination compensation technique is a coding method that compensates for the illumination variation between a target block and a predicted block using a linear model. The video encoding device inter-predictor 124 is configured to determine a reference block in the reference image using a motion vector (translation vector) of the target block and obtain linear model parameters for illumination compensation using pre-reconstructed samples around (on the top and left sides) of the reference block and pre-reconstructed samples around (on the top and left sides) of the target block.

[143] When the pre-recovered samples around the reference block are denoted as x and the corresponding pre-recovered samples around the target block are denoted as y, the parameters " A " and " b " are derived as in Equation 15 such that the sum of the squared differences between y and ( Ax + b ) is minimized.

Equation 15

[144] The final predicted samples are generated by applying a weight A and a bias b to the samples in the prediction block (reference block) generated by the motion vector of the target block, as shown in Equation 16. In Equation 16, pred[x][y] is the predicted sample at position (x, y), generated by the motion vector of the target block, and predLIC[x][y] is the final predicted sample after illumination compensation.

Equation 16

[145] Another aspect of the present disclosure relates to a technique for combining a lighting compensation method with affine motion prediction.

[146] As described above, when sub-block affine motion prediction is applied to a target block, a motion vector is generated for each of the sub-blocks. Light compensation parameters can be obtained using a corresponding motion vector for each sub-block, and then light compensation can be performed in the sub-block module using the corresponding motion vector. This, however, not only increases the computational complexity but also causes a latency problem. Since the reconstructed samples in each sub-block are needed to compensate for the illumination of the next sub-block, the illumination compensation process for the sub-block must be suspended until the adjacent sub-block is reconstructed (that is, until both the prediction block and the residual block for subblock). The present disclosure addresses such problems.

[147] FIG. 11A to 11C show various examples of determining the position of the reference block to obtain the illumination compensation parameter according to an embodiment of the present disclosure. In this embodiment, one set of light compensation parameters (A, b) for the target block is output, and the same parameter is applied to all sub-blocks in the target block. That is, the entire target block is modified by one set of light compensation parameters.

[148] As shown in FIG. 11A, the inter-predictor 124 of the video encoder may be configured to determine the position of a reference block in the reference image using an affine motion vector of a subblock located in the upper left corner of the target block or a motion vector of a reference point in the upper left corner of the target block. Pre-reconstructed samples around a specific reference block are used to obtain parameters. Alternatively, as shown in FIG. 11B, the position of the reference block can be determined using the affine motion vector of the central subblock in the target block. Once the position of the reference block is determined, the illumination compensation parameter is derived using the pre-reconstructed samples adjacent to the top and left sides of the reference block and the corresponding pre-reconstructed samples adjacent to the top and left sides of the target block.

[149] As another example, multiple sub-blocks in a target block can be used. As shown in FIG. 11C, inter-predictor 124 determines a reference sub-block corresponding to each boundary sub-block using the affine motion vectors of the sub-blocks (border sub-blocks) located on the boundary in the target block. The samples for obtaining the illumination compensation parameter are extracted from the pre-reconstructed samples adjacent to the boundary subblocks in the target block and the corresponding reference subblocks, respectively. For subblocks located at the top edge in the target block and the corresponding reference subblocks, samples are extracted from the previously reconstructed samples adjacent to the top side. For subblocks located on the left edge in the target block and the corresponding reference subblocks, samples are extracted from the previously reconstructed samples adjacent to the left side.

[150] In the present embodiment, since the illumination compensation parameter is derived using pre-constructed samples around the target block and pre-constructed samples around the reference block, the above-mentioned delay problem can be solved. However, applying only one illumination compensation parameter to a target block to which sub-block affine motion prediction is applied could potentially result in lower coding efficiency. Another embodiment described below results in increased coding efficiency in exchange for a slight increase in complexity.

[151] According to another embodiment of the present disclosure, the inter-predictor 124 of the video encoding device divides the target block into a plurality of regions. Each region includes at least one sub-block, which is the unit in which affine motion prediction is performed. Inter-predictor 124 is configured to calculate light compensation parameters for a plurality of regions, and performs light compensation in units of regions or sub-blocks in a target block based on the calculated light compensation parameters.

1. Splitting the target block

[152] The inter-predictor 124 of the video encoding device is configured to divide a target block into a plurality of regions, each of which includes one or more sub-blocks, which are units in which affine motion prediction is performed.

[153] The target block can be divided using motion vectors of the target block's control points. Fig. 12A-12B are examples of dividing a target block using motion vectors of reference points. Referring to FIG. 12A, when the angle between the top left corner reference point motion vector MV0 and the top right corner reference point motion vector MV1 of the target block is an obtuse angle (more than 90 degrees), the target block is divided vertically. For example, the target block can be divided into two areas. Otherwise, the target block is not split vertically. Referring to FIG. 12B, when the angle between the upper left corner reference point motion vector MV0 and the lower left corner reference point motion vector MV2 is an obtuse angle, the target block is divided horizontally into two halves, for example. Otherwise, the target block is not split horizontally. When the affine model type of the target block corresponds to 4 parameters, the motion vector MV2 of the lower left corner reference point can be obtained by inputting the reference coordinates of the lower left corner of the target block into (x, y) in Equation 1.

[154] Based on the angle formed by the reference point motion vectors, inter-predictor 124 either skips the target block division or divides the target block into two regions (a left region and a right region in the case of a vertical division, and an upper region and a lower region in the case of a horizontal division ) or 4 areas. When the dot product of two vectors is less than 0, the angle is obtuse. Accordingly, inter-predictor 124 can determine whether the angle between the motion vectors of the reference point is an obtuse angle based on the dot product of the two vectors.

[155] Although the above example shows that the target block is divided when the angle between the motion vectors of the two reference points is an obtuse angle, the block can be divided when the angle is an acute angle (an angle less than 90 degrees). In addition, the size of the target block may be taken into account when dividing the target block. For example, when the horizontal length exceeds a set threshold, the target block is split vertically. When the vertical length exceeds a preset threshold, the target block is split horizontally. To split a target block, you can use both the size of the target block and the angle between the control point motion vectors.

[156] For example, a video encoding device may be configured to determine whether to divide a block based on the angle between the motion vectors of the reference points only when the horizontal and/or vertical length of the target block is greater than a predetermined threshold. As another example, when the horizontal length exceeds a predetermined threshold, the video encoding device may be configured to determine whether to vertically divide a block based on the angle between the motion vectors of the upper left corner reference point and the upper right corner of the target block. In addition, when the vertical length exceeds a predetermined threshold, the device may be configured to determine whether to split a block vertically based on the angle between the motion vectors of the upper left corner reference point and the lower left corner of the target block.

[157] Although it has been described that a target block is divided into a plurality of regions based on motion vectors of reference points of the target block, embodiments are not limited to this. For example, the affine motion vectors of the top left sub-block, the top right sub-block and the bottom left sub-block in the target block may be used instead of the top left corner reference point motion vector MV0, the top right corner reference point motion vector MV1 and the bottom left corner reference point motion vector MV2.

[158] While FIG. 12A-12B show that the target block is divided in half in the horizontal or vertical direction, embodiments are not limited to this. For example, the target block may be divided into a plurality of blocks in the horizontal direction and/or a plurality of blocks in the vertical direction. Alternatively, the target block may be divided into subblocks in which affine motion prediction has been performed. In this case, one subblock constitutes one area.

[159] Each partition region from the target block must be greater than or equal to the size of the subblock on which the affine motion prediction was performed. In other words, each area must consist of one or more subblocks. For example, if the subblock size is 4x4, the area size is ^2m x ²ⁿ (where m and n are integers greater than or equal to 2).

[160] When the target block is not divided, the light compensation parameters can be obtained according to the first embodiment described above. Next, in the case where the target block is divided, a method for obtaining light compensation parameters will be described.

2. Obtaining light compensation parameters and light compensation

[161] The video encoding apparatus is configured to calculate illumination compensation parameters corresponding to a plurality of areas. Light compensation is performed in units of regions or sub-blocks within a target block using a variety of light compensation parameters. The video encoding apparatus is configured to determine, for each of a plurality of regions, pre-constructed samples (reference samples) in the current picture and the reference picture to be used to obtain compensation parameters. The reference samples in the current image are selected from pre-constructed samples adjacent to the target block. For the area distant from the left border of the target block, pre-constructed samples adjacent to the left side of the target block are selected as reference. For the area distant from the upper boundary of the target block, pre-constructed samples adjacent to the upper side of the target block are selected as reference ones.

[162] Reference samples in the reference image may be selected using motion vectors of the reference points of the target block or one or more motion vectors derived from the motion vectors of the reference points. Next, a method for obtaining illumination compensation parameters and performing illumination compensation in accordance with various types of division of the target block will be described in detail.

[163] 2-1. When the target block is divided into 4 areas

[164] First, as an example, a case will be described where a target block having a horizontal length and a vertical length equal to W and H is divided in the horizontal and vertical directions and is thus composed of four regions. The video encoding apparatus is configured to determine pre-constructed samples (reference samples) in the current image and the reference image to be used to obtain compensation parameters for each region.

[165] FIG. 13 is an exemplary diagram illustrating a method for determining reference samples to be used to obtain compensation parameters. For the top left region in the target block, the video encoder selects T ₀ to T _w/2-1 next to the top side of the top left region and L ₀ to L _H/2-1 next to the left side of the top left region in the current image in as reference samples that will be used to obtain compensation parameters. Then, a reference region corresponding to the upper left region of the reference image is determined using a representative motion vector of the upper left region, and then pre-constructed samples adjacent to the upper and left sides of the reference region are selected as reference samples. In particular, the representative motion vector of the upper left region may be the motion vector MV0 of the control point of the upper left corner of the target block.

[166] For the upper right region, the video encoding apparatus is configured to select T _w/2 to T _w-1 adjacent the upper side of the upper right region as at least some reference samples to be used to obtain compensation parameters. In particular, L ₀ - L _{H /2-1} adjacent to the left side of the target block can be further selected as reference samples. Since there are no pre-constructed samples adjacent to the left side of the upper right region, L ₀ to L _H/2-1 on the left side of the target block can be filled in as reference samples on the left side of the upper right region.

[167] The video encoding apparatus is then configured to determine a reference region corresponding to the upper right region in the reference image using a representative motion vector of the upper right region, and selects pre-constructed samples adjacent to the upper side of the reference region as reference samples used to obtain compensation parameters. When additionally selected from L₀before L_H/2-1adjacent to the left side of the target block, the video encoding device may further select pre-constructed samples adjacent to the left side of the reference region as reference samples. Specifically, the representative motion vector of the upper right region may be the motion vector MV1 of the upper right corner reference point of the target block.

[168] For the lower left region, the video encoding device selects L _H/2 to L _H-1 adjacent the left side of the lower left region as at least some reference samples. Additionally, T ₀ to T _w/2-1 adjacent to the top side of the target block can be selected. The video encoding device is then configured to determine a reference region corresponding to the lower left region in the reference image using a representative motion vector of the lower left region, and select pre-constructed samples adjacent to the left side of the reference region as reference samples. When T ₀ to T _w/2-1 adjacent to the top side of the target block are further selected, the video encoding apparatus may be configured to further select pre-constructed samples adjacent to the top side of the reference region. Here, the representative motion vector of the upper left region may be the motion vector MV1 of the control point of the lower left corner of the target block.

[169] Once the reference samples for each of the upper left, upper right and lower left regions in the target block are determined, the video encoding device is configured to obtain a light compensation parameter for each region by correspondingly inputting the reference samples selected from the reference image and the reference samples. selected from the current image, in x and y of Equation 15. The illumination compensation parameter for the lower right region can be obtained from the illumination compensation parameters of the upper left, upper right and lower left regions. For example, the light compensation parameter of the lower right region may be set to the average of the light compensation parameters of the upper right and lower left regions.

[170] Alternatively, the video encoding apparatus may be configured to determine a representative motion vector for the lower right region and obtain a light compensation parameter for the lower right region using the representative motion vector. The representative motion vector for the bottom right region may be the motion vector of a reference point in the bottom right corner of the target block. The video encoding apparatus may be configured to obtain a reference point motion vector of the lower right corner of the target block by replacing the coordinate (W-1, H-1) of the lower right corner sample (lower right corner sample of the lower right region) in the target block for (x, y) in Equation 10 or 11.

[171] There are no pre-constructed samples adjacent to the bottom right of the target block. Therefore, from T_w/2 before T_w-1 and from L_H/2 to L_H-1, adjacent to the top and left sides of the target block, are used as reference samples to obtain the compensation parameters. Then, a reference region in the reference image is determined based on the motion vector of the reference point of the lower right corner of the target block, and pre-constructed samples adjacent to the top and left sides of the reference region are selected as reference samples. The video encoder obtains the illumination compensation parameter for the lower right region using Equation 15 with the selected reference samples.

[172] In the above example, a representative motion vector of each region is used to determine reference samples in the reference image to obtain a compensation parameter. The reference point motion vectors (i.e. parameters for affine motion prediction) are used as an example. However, the embodiments are not limited to this. The representative motion vector of each region may be a motion vector obtained from the motion vectors of the target block's control points.

[173] In an exemplary embodiment, the representative motion vector of each region may be a predetermined position motion vector obtained from the motion vectors of target block reference points. For example, the 4x4 affine motion vectors of the top left, top right, bottom left, and bottom right subblocks in a target block previously generated by affine motion prediction can be used as representative motion vectors of the top left, top right, bottom left, and bottom right regions, respectively. As another example, the affine motion vector at the center point of each of the top left, top right, bottom left, and bottom right regions can be used as a representative motion vector of each region. An affine motion vector at the center point of each region can be generated by replacing (x, y) with the position of the center of each region in Equation 10 or 11.

[174] In another exemplary embodiment, the average of the affine motion vectors of the subblocks in each region may be used as a representative motion vector of the corresponding region. As another way to determine the reference samples to be used in the reference image to obtain the compensation parameters, affine motion vectors of the subblocks in each region can be used instead of the representative motion vector of each region. In other words, 4x4 reference blocks at positions indicated by the affine motion vectors of 4x4 sub-blocks in each region can be determined in the reference image, and pre-constructed pixels adjacent to the top and left sides of the 4x4 reference blocks can be selected as reference pixels.

[175] FIG. 14 is an exemplary diagram for describing a method for performing illumination compensation using illumination compensation parameters for four areas of a target block. Each region includes one or more subblocks, which are the units in which the affine transformation is performed. In the example in FIG. 14, the upper left area TL, upper right area TR, lower left area BL and lower right area BR of the target block each include four sub-blocks (4×4 sub-block). The video encoding apparatus is configured to perform illumination compensation for predicted samples in a corresponding region (predicted samples generated by affine motion prediction) using the illumination compensation parameter of each region. Each subblock can be compensated using the light compensation parameter of the area to which the subblock belongs. For example, four subblocks included in the upper left region are compensated using the light compensation parameters ( _A0 , _b0 ) of the upper left region.

[176] In some other embodiments, the video encoding device may be configured to interpolate the light compensation parameters that respectively correspond to the sub-blocks in each region using the light compensation parameters of the four regions TL, TR, BL and BR, and then perform light compensation on each sub-block using an interpolated light compensation parameter. As described above, the illumination compensation parameter of each region can be obtained from a representative motion vector of the region. However, the representative motion vector may not reflect the motion of all sub-blocks in the corresponding region.

[177] For example, the light compensation parameters of the corresponding regions can be obtained from the motion vectors of the control points of the top left, top right, bottom left and bottom right corners of the target block or the affine motion vectors from the 4x4 top left, top right, bottom left and bottom right subblocks target block. In particular, the movement of the top left, top right, bottom left, and bottom right 4x4 subblocks can have a greater impact on obtaining the light compensation parameters. Accordingly, the video encoding apparatus may be configured to obtain the light compensation parameters of the corresponding sub-blocks by interpolation using the light compensation parameters of the corresponding regions.

[178] FIG. 15 is an exemplary diagram illustrating a method for interpolating the illumination compensation parameters of sub-blocks using the illumination compensation parameters of the corresponding areas. (A ₀ , b ₀ ) - (A ₃ , b ₃ ) - illumination compensation parameters of the corresponding areas, obtained from the motion vectors of the control points of the upper left corner, upper right corner, lower left corner and lower right corner of the target block, respectively, or from affine 4x4 motion vectors of the top left, top right, bottom left and bottom right subblocks of the target block. In addition, it is assumed that (A ₀ , b ₀ ) is the illumination compensation parameter of the upper left 4x4 subblock in the upper left region of the target block. Similarly, it is assumed that (A ₁ , b ₁ ) to (A3, b ₃ ) are the illumination compensation parameters of the upper right 4x4 subblock in the upper right region, the illumination compensation parameter of the lower left 4x4 subblock in the lower left region, and the illumination compensation parameter of the lower right 4x4 subblock in the lower right area. In other words, it is assumed that (A ₀ , b ₀ ) - (A ₃ , b ₃ ) are the compensation parameters of the four corner subblocks of the target block.

[179] The video encoding device interpolates the compensation parameters of the other sub-blocks using the compensation parameters of the four corner sub-blocks. Bilinear interpolation using four corner subblocks or linear interpolation can be performed. For example, subblocks in the first row of a target block can be interpolated using (A ₀ , b ₀ ) and (A ₂ , b ₂ ). Subblocks in the first column of the target block can be interpolated using (A ₀ , b ₀ ) and (A ₂ , b ₂ ). In this way, the compensation parameters of all subblocks are obtained.

[180] In particular, the filter coefficient or weight coefficient used for interpolation can be set according to the difference between the positions of the corner sub-blocks and the positions of the target sub-block (or the distance between the corner sub-blocks and the target sub-block). Smaller differences in position are given more weight.

[181] Alternatively, the filter coefficient or weight used for interpolation can be set according to the difference between the motion vectors used to obtain the corner subblock compensation parameters (top-left, top-right, bottom-left, and bottom-right reference point motion vectors corners, or affine motion vectors of the top left, top right, bottom left, and bottom right 4x4 subblocks of the target block) and the affine motion vector of the target subblock. Smaller differences in motion vectors are assigned more weight.

[182] 2.2 When the target block is divided into two areas

[183] Even when the target block is divided into two areas, the illumination compensation parameter of each area can be obtained in the same manner as when the target block is divided into four areas. For example, referring to FIG. 16, the video encoding apparatus selects as reference samples in the current image to obtain the upper region compensation parameter the pre-built samples T ₀ to T _w-1 adjacent to the top side of the upper region, and the pre-built samples L ₀ to L _{H/2 -1} adjacent to the left side of the top region. The video encoding device is then configured to determine a reference region in the reference image using a representative motion vector of the upper region, and select pre-constructed samples adjacent to the top and left sides of the reference region as reference samples.

[184] The representative motion vector of the upper region may be a motion vector of the upper left corner reference point of the target block. Alternatively, it could be a motion vector derived from the motion vectors of the target block's control points. For example, an affine motion vector of the top left 4x4 subblock in the top region may be used, or an affine motion vector corresponding to the center point of the top region may be used. In another example, the average of the affine motion vectors of the subblocks in each region may be used as a representative motion vector of the corresponding region.

[185] The video encoding apparatus is configured to select, as reference samples in the current image to obtain a lower region compensation parameter, the pre-built samples T ₀ - T _w-1 adjacent to the top side of the target block, and the pre-built samples L _H/2 - L _H-1 adjacent to the left side of the lower area. The video encoding device is then configured to determine a reference region in the reference image using a representative motion vector of the bottom region, and select pre-constructed samples adjacent to the top and left sides of the reference region as reference samples.

[186] The representative motion vector of the lower region may be a motion vector of the control point of the lower left corner of the target block. Alternatively, it could be a motion vector derived from the motion vectors of the target block's control points. For example, an affine motion vector of the lower left 4x4 subblock in the bottom region may be used, or an affine motion vector corresponding to the center point of the bottom region may be used. As another example, the average of the affine motion vectors of the subblocks in the bottom region can be used as a representative motion vector of the bottom region.

[187] In the above example, a representative motion vector of each region is used as a method for determining reference samples in the reference image used to obtain the compensation parameter. However, the affine motion vectors of the subblocks in each region can be used instead of the representative motion vector of each region. That is, for each region, 4x4 reference blocks at positions indicated by the affine motion vectors of the 4x4 sub-blocks can be determined in the reference image, and pre-constructed pixels adjacent to the top and left sides of the 4x4 reference blocks can be selected as reference pixels.

[188] For each region, the video encoding apparatus is configured to input reference samples selected in the reference image and reference samples selected in the current image into x and y in Equation 15, respectively, to obtain compensation parameters for the respective regions. In addition, the video encoding device performs illumination compensation for the prediction samples in the corresponding region (prediction samples generated by affine motion prediction) using the illumination compensation parameter of each region. Each subblock can be compensated using the light compensation parameter of the area to which the subblock belongs.

[189] Alternatively, the video encoding apparatus may be configured to interpolate the light compensation parameters that correspond to the sub-blocks using the light compensation parameters of the corresponding regions, and to perform light compensation for each sub-block using the interpolated parameters, as in the case where the target block is divided into four areas.

[190] Interpolation can be used when the representative motion vector of each region used to obtain the compensation parameter is associated with a specific position in each region. For example, the control point motion vectors of the top left and bottom left corners of the target block or the affine motion vectors of the top left and bottom left 4x4 subblocks of the target block can be used as representative motion vectors. Specifically, the compensation parameters of the two areas are considered as the illumination compensation parameters of the upper left and lower left corner subblocks TL and BL in the target block, as shown in FIG. 17.

[191] The video encoding apparatus may be configured to interpolate a compensation parameter of a target subblock by applying a weighting factor to the compensation parameters (A ₀ , b ₀ ) of the upper region and the compensation parameters (A ₂ , b ₂ ) of the lower region. In particular, the weight may be assigned according to the distance between the upper left and lower left corner sub-blocks in the target block and the target sub-block. Alternatively, it can be set according to the difference between the motion vectors used to obtain the compensation parameters of the two regions (the control point motion vectors of the top left and bottom left corners of the target block, or the affine motion vectors of the top left and bottom left 4x4 subblock of the target block ) and the affine motion vector of the target subblock. As the difference between motion vectors decreases, the weight increases.

[192] Even when the target block is divided vertically into two areas, the same method as when the target block is divided horizontally into two areas can be used. Specifically, the representative motion vectors of the two regions (left region and right region) may be, for example, the control point motion vectors of the top left corner and top right corner of the target block, or the affine motion vectors of the top left and top right 4x4 subblocks of the target block . Alternatively, the affine motion vector of the center point of each region or the average of the affine motion vectors of the subblocks of each region may be used as a representative motion vector of each region. Other operations are the same as in the case of horizontal division described above or can be explicitly derived from it, so further detailed description thereof will be omitted.

[193] 2.3 When a target block is split into more than 4 subblocks

[194] As described above, the target block can be divided into ^2m and ²ⁿ blocks (where m and n are integers greater than or equal to 2) in the horizontal and vertical directions. For example, a target block can be divided into subblocks, which are units in which affine motion prediction is performed. This case is illustrated in FIG. 18. Referring to FIG. 18, the video encoding device outputs light compensation parameters for the upper left area TL, upper right area TR, lower left area BL and lower right area BR of the target block using the method described in Section 2.1.

[195] For areas other than these four areas, the illumination compensation parameters for the remaining areas can be obtained by applying the interpolation described in section 2.1. Alternatively, for the remaining areas, the illumination compensation parameters can be obtained in the same way as in the case of the upper left area TL, upper right area TR, lower left area BL and lower right area BR. For example, for region A, the video encoding device selects pre-constructed samples located on the top side of region A and adjacent to the target block, and pre-constructed samples located on the left side of region A and adjacent to the target block, as reference pixels in the current image. In addition, reference samples in the reference image are selected using the motion vector of a reference point in the upper left corner of area A, the affine motion vector of a 4x4 subblock located in the upper left corner of area A, or the affine motion vector of the center point of area A. Area A Light Compensation Parameter output using the selected reference samples.

[196] The illumination compensation technique described above is a technique that uses a motion vector that can be obtained using pre-constructed samples and affine motion prediction. Accordingly, the video decoding apparatus may be configured to perform illumination compensation in the same manner as the video encoding apparatus. The video encoding apparatus is not required to signal additional information for the illumination compensation process to the video decoding apparatus.

[197] The illumination compensation technique described above can be used in conjunction with other encoding tools to compensate for the motion of a target block in units of subblocks in addition to affine motion prediction. For example, this can be applied to bidirectional optical flow. The video encoding apparatus and the video decoding apparatus may be configured to generate prediction samples for each sub-block by performing bidirectional optical flow on a sub-block basis for the target block, and may be further configured to perform the above illumination compensation on the prediction samples in the target block. The illumination compensation technique applied to bidirectional optical flow is the same as the illumination compensation technique described above, except for the following.

[198] In the illumination compensation technique described above, target block reference point motion vectors or sub-block affine motion vectors are used to partition the target block and calculate a representative motion vector of each region separated from the target block. However, in the case of bidirectional optical flow, there is neither a reference point motion vector nor an affine motion vector. Accordingly, when the illumination compensation technique is applied to bidirectional optical flow, the optical flows (v _x , v _y ) of the subblocks are used instead of the motion vectors of the control points or the affine motion vectors of the subblocks.

[199] V. Syntax signaling

[200] Another aspect of the present disclosure relates to a method for signaling high-level syntaxes for controlling enable/disable the various features described above. The above-described affine motion prediction, affine motion prediction sample-based adjustment, adaptive motion vector resolution, and illumination compensation are coding tools used to improve video coding efficiency. However, for specific content, such as screen content, the aforementioned various encoding tools may not improve compression performance. Accordingly, signaling on a coding unit basis whether to apply each coding tool or deciding on a coding unit basis whether to apply each coding tool is likely to degrade coding efficiency or increase computational complexity. The present disclosure discloses a signaling technique for effectively controlling the above-described encoding tools.

[201] To encode a sequence of images, the video encoding apparatus is configured to encode a first high-level syntax element of the sequence indicating whether the plurality of encoding tools is disabled. The first element of the high-level syntax can be encoded in the sequence header, that is, SPS. In particular, the plurality of encoding tools includes affine motion prediction and also includes at least one of the following: sample-by-sample adjustment for affine motion prediction samples, adaptive motion vector resolution, and illumination compensation.

[202] When the first high-level syntax element indicates that the entire set of encoding tools is disabled, no encoding tools are applied to the encoding of the sequence. In other words, the encoding tools described above do not apply to any images in the sequence. When the first high-level syntax element indicates that the plurality of encoding tools is not disabled, the plurality of encoding tools may be used to encode the corresponding sequence. For example, affine motion prediction can be applied to all images in a sequence. Accordingly, the video encoding apparatus may encode a coding unit flag indicating whether affine motion prediction coding has been applied to each of the encoding units and transmit it to the video decoding apparatus.

[203] When the encoding unit flag indicates that affine motion prediction has been applied to the corresponding encoding unit, the video decoding apparatus may be configured to perform the above-described affine motion prediction on the encoding unit. When the first high-level syntax element indicates that the plurality of encoding tools is not disabled, the video encoding device may be configured to encode one or more second high-level syntax elements indicating whether each encoding tool is valid in at least a portion of the plurality of sequence-level encoding tools or at the image level.

[204] In some embodiments, to encode one or more second high-level syntax elements, the video encoding device encodes, for each encoding tool, a sequence level flag indicating whether the corresponding encoding tool is enabled in the SPS. When the sequence level flag indicates that the corresponding encoding tool is not enabled, the corresponding encoding tool is not used to encode images belonging to the sequence. In addition, the video encoding device is configured to encode a picture level flag indicating whether a corresponding encoding tool is enabled in a picture header (ie, PPS) of each of the pictures belonging to the sequence depending on the value of the sequence level flag.

[205] For example, when a sequence level flag indicates that the corresponding encoding tool is enabled at the sequence level, a picture level flag is encoded to indicate whether the corresponding encoding tool is enabled at the picture level. When the first high-level syntax element indicates that the entire set of encoding tools is disabled, or the sequence level flag indicates that the corresponding encoding tool is not enabled at the sequence level, the image level flag is not separately encoded. In particular, the video decoding apparatus is configured to automatically set the picture level flag to a value indicating that the corresponding encoding tool is not enabled.

[206] Image level flags for encoding means other than affine motion prediction may be transmitted depending on the image level flag for affine motion prediction. When the image level flag for affine motion prediction indicates that affine motion prediction is not enabled at the image level, image level flags for other encoding tools are not encoded, and other encoding tools are not enabled at the image level. When picture level flags for other encoding tools are not included in the bitstream, the video decoding apparatus is configured to set each picture level flag to a value indicating that the corresponding encoding tools are not enabled at the picture level.

[207] The video decoding device may be configured to set at the picture level whether to enable a corresponding encoding tool from a picture level flag signaled from the video encoding device or a picture level flag whose value is set automatically. The second high-level syntax element for affine motion prediction may be omitted. In particular, whether affine motion prediction is enabled at the sequence level is controlled by the first high-level syntax. The video decoding apparatus is configured to decode the first high-level syntax from the video encoding apparatus, and then determine whether to enable affine motion prediction at the sequence level.

[208] According to the above-described syntax structure, a video encoding apparatus may be configured to adaptively determine whether to enable the use of encoding tools at the sequence level and the picture level. In addition, the video decoding apparatus may be configured to adaptively determine whether to enable sequence-level and picture-level encoding tools by decoding syntaxes transmitted from the video encoding apparatus.

[209] In some other embodiments, to encode one or more second high-level syntax elements, the image level flag of the above-described embodiment may be replaced by a slice level flag. In other words, when the sequence level flag indicates that the corresponding encoding tool is enabled at the sequence level, the video encoding device encodes in the slice header a slice level flag to indicate whether the corresponding encoding tool is enabled at the slice level. When the sequence level flag indicates that the corresponding encoding tool is not enabled, the slice level flag is not encoded. The video encoding apparatus may be configured to adaptively determine whether to enable encoding tools at the sequence level and the slice level. The video decoder may be configured to determine whether to allow each tool to be used at the slice level using a slice level flag.

[210] In some other embodiments, the sequence level flag may be omitted to encode one or more second high-level syntax elements. The image-level flag or slice-level flag is signaled depending on the first high-level syntax element. When the first high-level syntax element indicates that multiple encoding tools are enabled, the image-level flag or slice-level flag is encoded in the image header or slice header.

[211] It should be understood that the exemplary embodiments described above may be implemented in a variety of different ways. The functions described in one or more examples may be implemented in hardware, software, firmware, or any combination thereof. It should be understood that the functional components described here have been labeled "module" to further emphasize their implementation independence.

[212] Various methods or functions described herein may be implemented as instructions stored on a non-volatile storage medium that can be read and executed by one or more processors. Non-transitory recording medium includes, for example, all types of recording devices in which data is stored in a form readable by a computer system. For example, non-transitory recording media includes media such as erasable programmable read-only memory (EPROM), flash drives, optical drives, magnetic hard disks, and solid-state drives (SSDs).

[213] Although exemplary embodiments have been described for illustrative purposes, those skilled in the art will appreciate that various modifications and changes are possible without departing from the spirit and scope of the embodiments. Exemplary embodiments have been described for brevity and clarity. Accordingly, one of ordinary skill in the art will appreciate that the scope of the embodiments is not limited to the embodiments expressly described above, but includes the claims and their equivalents.

CROSS REFERENCE TO RELATED APPLICATION

This application claims the benefit of Korean Patent Application Nos. 10-2019-0070633, 10-2019-0082010, and 10-2020-0072332, filed June 14, 2019, July 8, 2019, and June 15, 2020, respectively, all contents of which are incorporated herein by reference.

Claims (44)

1. A video decoding device for decoding a sequence of encoded images on a block-by-block basis, comprising: decoder configured: determine, at the sequence level, whether affine motion prediction is enabled by decoding the first high-level syntax element from the bitstream; And determine, at the image level, whether each of the at least one encoding tool is enabled by decoding one or more second high-level syntax elements for each of the at least one encoding tool from the bitstream depending on the first high-level syntax element, wherein the at least one encoding tool includes sample-by-sample adjustment of the affine motion prediction samples; And a predictor configured, when the sequence level assertion indicates that affine motion prediction is enabled, to determine two or three motion vectors of target block reference points encoded using affine motion prediction, and to generate affine prediction samples for each of the subblocks in the target block by executing the affine predicting motion in a unit of subblocks using motion vectors of control points, wherein, when it is determined based on the image level determination that sample-by-sample adjustment of affine motion prediction samples is allowed for the current image containing the target block, the predictor is configured to modify sample values in the affine prediction samples by motion compensation in accordance with the position of the sample in each of the sub-blocks based on motion vectors of control points of the target block. 2. The apparatus of claim 1, wherein the first high-level syntax element is decoded from a sequence header. 3. The apparatus of claim 2, wherein the decoder is configured to decode one or more second high-level syntax elements for sample-by-sample adjustment of the affine motion prediction samples by: when the first high-level syntax element indicates that affine motion prediction is enabled, decoding a sequence level flag indicating whether sample-by-sample adjustment of affine motion prediction samples is enabled from the sequence header; And depending on the sequence level flag, decoding a picture level flag indicating whether sample-by-sample adjustment of affine motion prediction samples is enabled from the image header of each of the images belonging to the sequence. 4. The apparatus of claim 3, wherein when the first high-level syntax element indicates that affine motion prediction is not enabled or the sequence level flag indicates that sample-by-sample adjustment is not enabled, the decoder is configured to set the picture level flag to a value indicating that sample-by-sample adjustment is not allowed, without decoding from the image header. 5. The apparatus of claim 1, wherein when the first high-level syntax element indicates that affine motion prediction is not enabled, any of the affine motion prediction and the at least one encoder is not enabled for all pictures in the sequence. 6. The device according to claim 1, in which the predictor is configured to modify the values of the affine prediction samples by: generating a motion offset resulting from a difference between a reference position in the target sub-block to be modified and the center of the target sub-block using motion vectors of the reference points of the target block; generating horizontal and vertical gradients relative to a reference position in the target subblock; generating a sample offset for an affine prediction sample corresponding to a sample position in the target sub-block using horizontal and vertical gradients and motion offset; And adding a sample offset to the affine prediction sample value corresponding to the sample position. 7. The apparatus of claim 6, wherein the motion offset generated for a reference position in the target subblock is reused as a motion offset for the same reference position in another subblock of the target block. 8. A video encoding apparatus for encoding a sequence of images on a block-by-block basis, comprising: encoder configured to implement: encoding, in a sequence header, a first high-level syntax element in the sequence level to indicate whether both the affine motion prediction and at least one encoding tool are disabled, wherein the at least one encoding tool includes a sample-by-sample adjustment of the affine prediction samples movements; depending on the first high-level syntax element, encoding one or more second high-level syntax elements indicating the establishment of a sequence level or an image level with respect to the resolution of each of the at least one encoding tool; And predictor configured to implement: when the first high-level syntax element indicates that both the affine motion prediction and at least one encoding tool are not disabled, determining two or three reference point motion vectors for the target block to be encoded, and generating affine prediction samples for each of the subblocks by performing sub-block affine motion prediction using control point motion vectors; And wherein when sample-by-sample adjustment of affine motion prediction samples is enabled for the current image containing the target block, the predictor is configured to modify sample values in the affine prediction samples by motion compensation in accordance with the position of the sample in each of the sub-blocks based on the motion vectors of the target block's reference points. 9. The apparatus of claim 8, wherein the encoder is configured to encode the at least one second high-level syntax element for sample-by-sample adjustment of the affine motion prediction samples as follows: when the first high-level syntax element indicates that affine motion prediction is enabled, encoding a sequence level flag indicating whether sample-by-sample adjustment of affine motion prediction samples is enabled in the sequence header; And depending on the sequence level flag, encoding a picture level flag indicating whether per-sample adjustment is enabled in a picture header of each of the pictures belonging to the sequence. 10. The apparatus of claim 9, wherein when the first high-level syntax element indicates that affine motion prediction is not enabled or the sequence level flag indicates that sample-by-sample adjustment is not enabled, the encoder is configured not to encode the picture level flag. 11. The apparatus of claim 8, wherein the predictor is configured to modify sample values for the affine prediction samples as follows: generating a motion offset resulting from a difference between a reference position in the target sub-block to be modified and the center of the target sub-block using motion vectors of the reference points of the target block; generating horizontal and vertical gradients relative to a reference position in the target subblock; generating a sample offset for an affine prediction sample corresponding to a sample position in the target sub-block using horizontal and vertical gradients and motion offset; And adding a sample offset to the affine prediction sample value corresponding to the sample position. 12. The apparatus of claim 11, wherein the motion offset generated for a reference position in the target subblock is reused as a motion offset for the same reference position in another subblock of the target block. 13. A method for storing a bitstream associated with video data, the method comprising: generating a bit stream by encoding video data for a sequence of images; And storing the bitstream in a non-transitory machine-readable medium, wherein the bitstream generation comprises: encoding, in a sequence header, a first high-level syntax element in the sequence level to indicate whether both the affine motion prediction and the at least one encoding tool are disabled, wherein the at least one encoding tool includes a sample-by-sample adjustment of the affine prediction samples movements; depending on the first high-level syntax element, encoding one or more second high-level syntax elements indicating the establishment of a sequence level or an image level with respect to the resolution of each of the at least one encoding tool; And when the first high-level syntax element indicates that both the affine motion prediction and at least one encoding tool are not disabled, determining two or three reference point motion vectors for the target block to be encoded, and generating affine prediction samples for each of the subblocks by performing sub-block affine motion prediction using control point motion vectors; when sample-by-sample adjustment of the affine motion prediction samples is enabled for the current image containing the target block, modifying the sample values in the affine prediction samples by motion compensation according to the position of the sample in each of the sub-blocks based on the motion vectors of the reference points of the target block; And generating residual signals for the target block based on the modified sample values and encoding the residual signals.