KR20210134266A - Method and Apparatus for Estimating Optical Flow for Motion Compensation - Google Patents

Abstract

Disclosed is a method for estimating adaptive bidirectional optical flow (BIO) for intra prediction correction in an image encoding process. An object of the present invention is to reduce complexity and/or cost of a bidirectional optical flow at a pixel level or a sub-block level. The method for encoding image data comprises: a process of determining a first motion vector and determining a second motion vector; a process of generating a prediction block; and a process of encoding a current block.

Description

Optical flow estimation method and apparatus for motion compensation {Method and Apparatus for Estimating Optical Flow for Motion Compensation}

The present invention relates to video encoding or decoding. More particularly, the present invention relates to an adaptive bidirectional optical flow estimation method for inter prediction correction in a video encoding process.

The content described in this section merely provides background information for the present embodiment and does not constitute the prior art.

When it comes to video coding, compression takes advantage of data redundancy in both spatial and temporal dimensions. Spatial redundancy reduction is largely achieved by transform coding. Temporal redundancy is reduced through predictive coding. Observing that the temporal correlation is maximized along the motion trajectory, motion compensated prediction is used for this purpose. The main purpose of motion estimation in this context is not to look for 'real' motion in the scene, but to maximize compression efficiency. That is, the motion vector should provide an accurate prediction of the signal. It should also enable a compressed representation since the motion information has to be transmitted as overhead in the compressed bit stream. Efficient motion estimation is important to achieve high compression in video coding.

Motion in a video sequence is an important source of information. Motion is caused not only by the object but also by the camera movement. The apparent motion, also known as optical flow, captures spatio-temporal variations in pixel intensity in an image sequence.

Bi-directional Optical Flow (BIO) is a motion estimation/compensation technique disclosed in JCTVC-C204 and VCEG-AZ05 BIO, based on the assumption of optical flow and steady motion (steady motion) at the sample level. motion refinement). The bi-directional optical flow estimation method currently under discussion has the advantage of being able to fine-tune motion vector information, but has a disadvantage in that it requires much higher computational complexity than traditional bi-directional prediction for fine motion vector information correction.

Non-Patent Document 1: JCTVC-C204 (E. Alshina, et al., Bi-directional optical flow, Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 3rd Meeting: Guangzhou, CN, 7-15 October, 2010)

Non-Patent Document 2: VCEG-AZ05 (E. Alshina, et al., Known tools performance investigation for next generation video coding, ITU-T SG 16 Question 6, Video Coding Experts Group (VCEG), 52nd Meeting: 19-26 June 2015, Warsaw, Poland)

An object of the present invention is to reduce the complexity and/or cost of a bi-directional optical flow (BIO).

According to an aspect of the present invention, a first motion vector indicating a first correspondence region most similar to a current block in a first reference picture is determined, and a second motion vector indicating a second correspondence region most similar to the current block in a second reference picture is determined. the process of determining a motion vector; generating a prediction block for the current block by applying a sub-block-based BIO process; and reconstructing the current block by using the generated prediction block. Here, the generating of the prediction block may include: determining a BIO motion vector for each sub-block constituting the current block; and generating prediction values of pixels constituting the corresponding sub-block based on the determined BIO motion vector.

According to another aspect of the present invention, there is provided an apparatus for decoding image data including a memory and one or more processors, wherein the one or more processors include a first motion vector indicating a first corresponding region most similar to a current block in a first reference picture. determining , and determining a second motion vector indicating a second corresponding region most similar to the current block in a second reference picture; generating a prediction block for the current block by applying a sub-block-based BIO process; and performing a process of reconstructing pixels of the current block by using the generated prediction block. Here, the generating of the prediction block may include: determining a BIO motion vector for each sub-block constituting the current block; and generating prediction values of pixels constituting the given sub-block based on the determined BIO motion vector.

The BIO motion vector ( v _x , v _y ) is a vector such that the sum of squares of flow differences for each pixel located in the search area defined by a predetermined masking window centered on each pixel in the sub-block is minimized. can Alternatively, the BIO motion vector ( v _x , v _y ) is the square of the flow differences for all pixels located within the search area defined by a predetermined masking window centered on some pixels in the sub-block. It can be determined as a vector such that the sum is minimized. For example, positions of pixels to which the masking window is applied and pixels to which the masking window is not applied may form a check pattern, a horizontal stripes pattern, or a vertical stripes pattern.

In some examples, instead of repeatedly calculating the flow differences, a weight may be given to the overlapping difference value according to the number of overlapping differences. In some examples, in determining the BIO motion vector for a sub-block located at the edge of the current block, flow differences for pixels located in an area outside the current block may not be considered.

In some examples, a masking window may not be used. For example, the BIO motion vectors ( v _x , v _y ) may be determined as vectors such that the sum of squares of flow differences for each pixel in the sub-block is minimized.

According to another aspect of the present invention, a first motion vector indicating a first correspondence region most similar to a current block in a first reference picture is determined, and a second motion vector indicating a second correspondence region most similar to the current block in a second reference picture is determined. 2 The process of determining a motion vector; generating a prediction block for the current block by applying a BIO process in units of pixels; and reconstructing pixels of the current block by using the generated prediction block. Here, the process of generating the prediction block is a process of determining a BIO motion vector for each pixel constituting the current block, and the BIO motion vector is located within a plus-shaped or diamond-shaped masking window centered on the corresponding pixel. determined as a vector such that the sum of squares of flow differences obtained for all masking pixels that and generating a predicted value of the corresponding pixel based on the determined BIO motion vector.

According to another aspect of the present invention, there is provided an image decoding apparatus including a memory and one or more processors, wherein the one or more processors determine a first motion vector indicating a first corresponding region most similar to a current block in a first reference picture. and determining a second motion vector indicating a second corresponding region most similar to the current block in the second reference picture; generating a prediction block for the current block by applying a BIO process in units of pixels; and a process of reconstructing pixels of the current block by using the generated prediction block. Here, the process of generating the prediction block is a process of determining a BIO motion vector for each pixel constituting the current block, and the BIO motion vector is located within a plus-shaped or diamond-shaped masking window centered on the corresponding pixel. determined as a vector such that the sum of squares of flow differences obtained for all masking pixels that and generating a predicted value of the corresponding pixel based on the determined BIO motion vector.

1 is an exemplary block diagram of an image encoding apparatus that can implement techniques of the present disclosure.
2 is an exemplary diagram of a neighboring block of a current block.
3 is an exemplary block diagram of an image decoding apparatus capable of implementing the techniques of the present disclosure.
4 is a reference diagram for explaining the basic concept of BIO.
5A is a flowchart illustrating a bidirectional motion compensation method performed based on pixel-level BIO according to an embodiment of the present invention.
5B is a flowchart illustrating a bidirectional motion compensation method performed based on sub-block level BIO according to an embodiment of the present invention.
6 is a diagram illustrating a 1×1 block of a current block and a 5×5 masking window used for BIO-based motion compensation according to the first embodiment.
7 is a diagram illustrating non-rectangular masking windows that can be used to determine a pixel-level BIO motion vector according to the second embodiment.
8 is a diagram illustrating a 1×1 block of a current block and a diamond-shaped masking window used to determine a pixel-level BIO motion vector according to the second embodiment.
9 is a diagram illustrating a 5×5 masking window and a 4×4 sub-block used for determining a sub-block level BIO motion vector according to the third embodiment.
FIG. 10A is a diagram for explaining that difference values used to determine a BIO motion vector of a sub-block level are repeatedly calculated.
FIG. 10B is a diagram exemplarily illustrating weights of difference values for each pixel position used to determine a BIO motion vector at a sub-block level.
11 is a diagram illustrating a diamond-shaped masking window and 4×4 sub-blocks used to determine a BIO motion vector at a sub-block level according to the fourth embodiment.
12 is a diagram illustrating three types of positions of pixels to which a masking window is applied in a sub-block according to the fifth embodiment.
13 is a diagram illustrating a 5×5 masking window used to determine a sub-block level BIO motion vector and a 4×4 sub-block in which pixels to which the masking window is applied are sampled in a check pattern according to the fifth embodiment.
14 is a diagram illustrating a diamond-type masking window used for BIO-based motion compensation and prediction pixels in a 4×4 sub-block according to the sixth embodiment.
15 is a diagram illustrating an example of weighting each pixel of a sub-block according to the seventh embodiment.
16A is a diagram illustrating sub-blocks positioned at the edge of a current block in a 16×16 current block including 16 4×4 sub-blocks.
FIG. 16B is a diagram exemplarily showing weights for each pixel position of difference values used to determine a BIO motion vector for a 4×4 sub-block located at an upper-left corner in a 16×16 current block.

Hereinafter, some embodiments of the present invention will be described in detail with reference to exemplary drawings. It should be noted that in adding identification codes to the components of each drawing, the same components are to have the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted.

The techniques of this disclosure generally relate to reducing the complexity and/or cost of bi-directional optical flow (BIO) techniques. BIO can be applied during motion compensation. In general, the BIO calculates a motion vector for each pixel in the current block through an optical flow, and is used to update a prediction value located in the corresponding pixel based on the calculated motion vector value for each pixel.

1 is an exemplary block diagram of an image encoding apparatus that can implement techniques of the present disclosure.

The image encoding apparatus includes a block divider 110 , a predictor 120 , a subtractor 130 , a transform unit 140 , a quantizer 145 , an encoder 150 , an inverse quantizer 160 , and an inverse transform unit ( 165 ), an adder 170 , a filter unit 180 , and a memory 190 . Each component of the image encoding apparatus may be implemented as a hardware chip, or may be implemented as software and a microprocessor may be implemented to execute a function of software corresponding to each component.

The block divider 110 divides each picture constituting an image into a plurality of coding tree units (CTUs) and then recursively divides the CTUs using a tree structure. A leaf node in the tree structure becomes a coding unit (CU), which is a basic unit of encoding. As a tree structure, a quadtree (QT) in which a parent node (or parent node) is divided into four child nodes (or child nodes) of the same size, or a QT structure and a parent node divided into two child nodes A QTBT (QuadTree plus BinaryTree) structure mixed with a binary tree (BinaryTree, BT) structure may be used. That is, QTBT may be used to divide a CTU into a plurality of CUs.

In a QTBT (QuadTree plus BinaryTree) structure, a CTU may first be divided into a QT structure. The quadtree splitting may be repeated until the size of a splitting block reaches the minimum block size of a leaf node (MinQTSize) allowed in QT. If the leaf node of the quadtree is not larger than the maximum block size (MaxBTSize) of the root node allowed in the BT, it may be further partitioned into the BT structure. In BT, a plurality of partition types may exist. For example, in some examples, there may be two types of horizontally splitting a block of a corresponding node into two blocks of the same size (ie, symmetric horizontal splitting) and a vertical splitting type (ie, symmetric vertical splitting). In addition, a type for dividing the block of the corresponding node into two blocks having an asymmetric shape may further exist. The asymmetric form may include a form of dividing the block of the corresponding node into two rectangular blocks having a size ratio of 1:3, or a form of dividing the block of the corresponding node in a diagonal direction.

The division information generated by the block division unit 110 by dividing the CTU according to the QTBT structure is encoded by the encoding unit 150 and transmitted to the image decoding apparatus.

Hereinafter, a block corresponding to a CU to be encoded or decoded (ie, a leaf node of QTBT) is referred to as a 'current block'.

The prediction unit 120 generates a prediction block by predicting the current block. The prediction unit 120 includes an intra prediction unit 122 and an inter prediction unit 124 .

In general, each of the current blocks in a picture may be predictively coded. Prediction of the current block is performed using either intra prediction techniques (using data from the picture containing the current block) or inter prediction techniques (using data from pictures previously coded for the picture containing the current block). In general, this can be done. Inter prediction includes both uni-prediction and bi-prediction.

For each inter-predicted block, a set of motion information may be available. One set of motion information may include motion information for forward and backward prediction directions. Here, the forward and backward prediction directions are two prediction directions of a bi-directional prediction mode, and the terms “forward” and “reverse” do not necessarily have a geometric meaning. Instead, they generally correspond to whether the reference picture will be displayed before (“reverse”) or after (“forward”) the current picture. In some examples, “forward” and “reverse” prediction directions may correspond to reference picture list 0 (RefPicList0) and reference picture list 1 (RefPicList1) of the current picture.

For each prediction direction, the motion information includes a reference index and a motion vector. The reference index can be used to identify a reference picture in the current reference picture list (RefPicList0 or RefPicList1). A motion vector has horizontal (x) and vertical (y) components. In general, the horizontal component represents the horizontal displacement in the reference picture relative to the position of the current block in the current picture, which is necessary to locate the x-coordinate of the reference block. The vertical component represents a vertical displacement in the reference picture relative to the position of the current block, which is required to locate the y-coordinate of the reference block.

The inter prediction unit 124 searches for a block most similar to the current block in the reference picture encoded and decoded before the current picture, and generates a prediction block for the current block using the searched block. Then, a motion vector corresponding to the displacement between the current block in the current picture and the prediction block in the reference picture is generated. In general, motion estimation is performed for a luma component, and a motion vector calculated based on the luma component is used for both the luma component and the chroma component. Motion information including information on a reference picture and information on a motion vector used to predict the current block is encoded by the encoder 150 and transmitted to the image decoding apparatus.

Examples of this disclosure relate generally to bi-directional optical flow (BIO) techniques. As such, certain techniques of the present disclosure may be performed by the inter prediction unit 124 . That is, for example, the inter prediction unit 124 may perform the techniques of the present disclosure described with reference to FIGS. 4 to 13 below. In other words, after determining the bidirectional motion vector for the current block, the inter prediction unit 124 may generate a prediction block for the current block by using motion compensation according to the BIO technique in units of image pixels or sub-blocks. . In other examples, one or more other units of the encoding apparatus may be additionally involved in performing the techniques of this disclosure. In addition, since there is an explicit equation for calculating a motion vector, a search process for acquiring motion information and signaling for transmitting motion information are not required.

Various methods may be used to minimize the amount of bits required to encode motion information.

For example, when the reference picture and motion vector of the current block are the same as the reference picture and motion vector of the neighboring block, the motion information of the current block may be transmitted to the decoding apparatus by encoding information for identifying the neighboring block. This method is called 'merge mode'.

In the merge mode, the inter prediction unit 124 selects a predetermined number of merge candidate blocks (hereinafter referred to as 'merge candidates') from neighboring blocks of the current block.

As the neighboring blocks for inducing the merge candidate, as shown in FIG. 2 , the left block (L), the upper block (A), the upper right block (AR), and the lower left block (BL) adjacent to the current block in the current picture. ), all or part of the upper left block AL may be used. Also, a block located in a reference picture (which may be the same as or different from a reference picture used to predict the current block) other than the current picture in which the current block is located may be used as a merge candidate. For example, a block co-located with the current block in the reference picture or blocks adjacent to the co-located block may be further used as merge candidates.

The inter prediction unit 124 constructs a merge list including a predetermined number of merge candidates by using these neighboring blocks. A merge candidate to be used as motion information of the current block is selected from among the merge candidates included in the merge list, and merge index information for identifying the selected candidate is generated. The generated merge index information is encoded by the encoder 150 and transmitted to the decoding apparatus.

Another method of encoding motion information is to encode a differential motion vector.

In this method, the inter prediction unit 124 derives motion vector prediction candidates for the motion vector of the current block using neighboring blocks of the current block. As neighboring blocks used to derive prediction motion vector candidates, a left block (L), an upper block (A), an upper right block (AR), a lower left block (L) adjacent to the current block in the current picture shown in FIG. BL), all or part of the upper left block AL may be used. In addition, a block located in a reference picture (which may be the same as or different from the reference picture used to predict the current block) other than the current picture in which the current block is located is used as a neighboring block used to derive prediction motion vector candidates. may be For example, a block co-located with the current block in the reference picture or blocks adjacent to the co-located block may be used.

The inter prediction unit 124 derives prediction motion vector candidates by using the motion vectors of the neighboring blocks, and determines a predicted motion vector with respect to the motion vector of the current block using the prediction motion vector candidates. Then, a differential motion vector is calculated by subtracting the predicted motion vector from the motion vector of the current block.

The prediction motion vector may be obtained by applying a predefined function (eg, a median value, an average value operation, etc.) to the prediction motion vector candidates. In this case, the image decoding apparatus also knows the predefined function. In addition, since the neighboring block used to derive the prediction motion vector candidate is a block that has already been encoded and decoded, the image decoding apparatus already knows the motion vector of the neighboring block. Therefore, the image encoding apparatus does not need to encode information for identifying the prediction motion vector candidate. Accordingly, in this case, information on a differential motion vector and information on a reference picture used to predict a current block are encoded.

Meanwhile, the prediction motion vector may be determined by selecting any one of the prediction motion vector candidates. In this case, information for identifying the selected prediction motion vector candidate is additionally encoded together with information on the differential motion vector and information on the reference picture used to predict the current block.

The intra prediction unit 122 predicts pixels in the current block by using pixels (reference pixels) located around the current block in the current picture including the current block. A plurality of intra prediction modes exist according to a prediction direction, and neighboring pixels to be used and an operation expression are defined differently according to each prediction mode. In particular, the intra prediction unit 122 may determine an intra prediction mode to be used for encoding the current block. In some examples, the intra prediction unit 122 may encode the current block using several intra prediction modes and select an appropriate intra prediction mode to use from the tested modes. For example, the intra prediction unit 122 calculates rate-distortion values using rate-distortion analysis for several tested intra prediction modes, and has the best rate-distortion characteristics among the tested modes. An intra prediction mode may be selected.

The intra prediction unit 122 selects one intra prediction mode from among a plurality of intra prediction modes, and predicts the current block by using a neighboring pixel (reference pixel) determined according to the selected intra prediction mode and an equation. Information on the selected intra prediction mode is encoded by the encoder 150 and transmitted to the image decoding apparatus.

The subtractor 130 generates a residual block by subtracting the prediction block generated by the intra prediction unit 122 or the inter prediction unit 124 from the current block.

The transform unit 140 transforms a residual signal in a residual block having pixel values in the spatial domain into transform coefficients in the frequency domain. The transform unit 140 may transform the residual signals in the residual block using the size of the current block as a transform unit, or divide the residual block into a plurality of smaller sub-blocks and convert the residual signals in a transform unit of the sub-block size. You can also convert There may be various methods for dividing the residual block into smaller subblocks. For example, it may be divided into sub-blocks of the same predefined size, or a quadtree (QT) type partitioning using a residual block as a root node may be used.

The quantization unit 145 quantizes the transform coefficients output from the transform unit 140 , and outputs the quantized transform coefficients to the encoder 150 .

The encoder 150 generates a bitstream by encoding the quantized transform coefficients using an encoding method such as CABAC. In addition, the encoder 150 encodes information such as CTU size, MinQTSize, MaxBTSize, MaxBTDepth, MinBTSize, QT split flag, BT split flag, and split type related to block splitting so that the video decoding apparatus is identical to the video encoding apparatus. Allows the block to be partitioned.

The encoder 150 encodes information on a prediction type indicating whether the current block is encoded by intra prediction or inter prediction, and encodes intra prediction information or inter prediction information according to the prediction type.

When the current block is intra-predicted, a syntax element for the intra-prediction mode is encoded as intra-prediction information. When the current block is inter-predicted, the encoder 150 encodes a syntax element for inter prediction information. The syntax element for inter prediction information includes the following.

(1) Mode information indicating whether motion information of the current block is encoded in a merge mode or a differential motion vector encoding mode

(2) Syntax elements for motion information

When motion information is encoded by the merge mode, the encoder 150 uses merge index information indicating which of the merge candidates is selected as a candidate for extracting the motion information of the current block as a syntax element for the motion information. encode

On the other hand, when the motion information is encoded according to the differential motion vector encoding mode, information on the differential motion vector and information on the reference picture are encoded as syntax elements for the motion information. If the prediction motion vector is determined by selecting any one candidate from among a plurality of prediction motion vector candidates, the syntax element for the motion information further includes prediction motion vector identification information for identifying the selected candidate. include

The inverse quantization unit 160 inverse quantizes the quantized transform coefficients output from the quantization unit 145 to generate transform coefficients. The inverse transform unit 165 restores the residual block by transforming the transform coefficients output from the inverse quantization unit 160 from the frequency domain to the spatial domain.

The addition unit 170 reconstructs the current block by adding the reconstructed residual block to the prediction block generated by the prediction unit 120 . Pixels in the reconstructed current block are used as reference pixels when intra-predicting the next block.

The filter unit 180 deblocks and filters the boundary between the reconstructed blocks in order to remove a blocking artifact generated due to block-by-block encoding/decoding, and stores the deblocking filter in the memory 190 . When all blocks in one picture are reconstructed, the reconstructed picture is used as a reference picture for inter prediction of blocks in a picture to be encoded later.

Hereinafter, an image decoding apparatus will be described.

3 is an exemplary block diagram of an image decoding apparatus capable of implementing the techniques of the present disclosure.

The image decoding apparatus includes a decoding unit 310 , an inverse quantization unit 320 , an inverse transform unit 330 , a prediction unit 340 , an adder 350 , a filter unit 360 , and a memory 370 . Like the image encoding apparatus of FIG. 2 , each component of the image decoding apparatus may be implemented as a hardware chip, or may be implemented as software and a microprocessor may be implemented to execute a function of software corresponding to each component.

The decoder 310 decodes the bitstream received from the image encoding apparatus, extracts information related to block division, determines a current block to be decoded, and predicts information necessary for reconstructing the current block, information on the residual signal, etc. to extract

The decoder 310 extracts information about the CTU size from a sequence parameter set (SPS) or a picture parameter set (PPS), determines the size of the CTU, and divides the picture into CTUs of the determined size. Then, the CTU is determined as the top layer of the tree structure, that is, the root node, and the CTU is divided using the tree structure by extracting the division information on the CTU. For example, when a CTU is split using a QTBT structure, a first flag (QT_split_flag) related to QT splitting is first extracted and each node is split into four nodes of a lower layer. And, for a node corresponding to a leaf node of QT, a second flag (BT_split_flag) and split type information related to BT splitting are extracted and the corresponding leaf node is split into a BT structure.

Meanwhile, when the decoding unit 310 determines a current block to be decoded through division of the tree structure, information on a prediction type indicating whether the current block is intra-predicted or inter-predicted is extracted.

When the prediction type information indicates intra prediction, the decoder 310 extracts a syntax element for intra prediction information (intra prediction mode) of the current block.

When the prediction type information indicates inter prediction, the decoder 310 extracts a syntax element for the inter prediction information. First, mode information indicating whether motion information of the current block is encoded by which mode among a plurality of encoding modes is extracted. Here, the plurality of encoding modes include a merge mode including a skip mode and a differential motion vector encoding mode. When the mode information indicates the merge mode, the decoder 310 extracts merge index information indicating whether to derive the motion vector of the current block from any of the merge candidates as a syntax element for the motion information. On the other hand, when the mode information indicates a differential motion vector encoding mode, the decoder 310 extracts information on the differential motion vector and information on the reference picture referenced by the motion vector of the current block as syntax elements for the motion vector. do. On the other hand, when the video encoding apparatus uses any one of the plurality of prediction motion vector candidates as the prediction motion vector of the current block, the prediction motion vector identification information is included in the bitstream. Therefore, in this case, information on the differential motion vector and information on the reference picture as well as the prediction motion vector identification information are extracted as syntax elements for the motion vector.

Meanwhile, the decoder 310 extracts information on the quantized transform coefficients of the current block as information on the residual signal.

The inverse quantization unit 320 inverse quantizes the quantized transform coefficients, and the inverse transform unit 330 inversely transforms the inverse quantized transform coefficients from the frequency domain to the spatial domain to reconstruct residual signals to generate a residual block for the current block.

The prediction unit 340 includes an intra prediction unit 342 and an inter prediction unit 344 . The intra prediction unit 342 is activated when the prediction type of the current block is intra prediction, and the inter prediction unit 344 is activated when the prediction type of the current block is intra prediction.

The intra prediction unit 342 determines the intra prediction mode of the current block from among the plurality of intra prediction modes from the syntax element for the intra prediction mode extracted from the decoder 310, and a reference pixel around the current block according to the intra prediction mode. Predict the current block using

The inter prediction unit 344 determines motion information of the current block by using the syntax element for the intra prediction mode extracted from the decoder 310 , and predicts the current block using the determined motion information.

First, the inter prediction unit 344 checks mode information in inter prediction extracted from the decoder 310 . When the mode information indicates a merge mode, the inter prediction unit 344 constructs a merge list including a predetermined number of merge candidates using neighboring blocks of the current block. A method for the inter prediction unit 344 to construct the merge list is the same as that of the inter prediction unit 124 of the image encoding apparatus. Then, one merge candidate is selected from among the merge candidates in the merge list by using the merge index information transmitted from the decoder 310 . Then, motion information of the selected merge candidate, that is, the motion vector and reference picture of the merge candidate, is set as the motion vector and reference picture of the current block.

On the other hand, when the mode information indicates the differential motion vector encoding mode, the inter prediction unit 344 derives prediction motion vector candidates using motion vectors of neighboring blocks of the current block, and uses the prediction motion vector candidates to obtain the current block. A predicted motion vector is determined for the motion vector of . The method of the inter prediction unit 344 inducing prediction motion vector candidates is the same as that of the inter prediction unit 124 of the image encoding apparatus. If the video encoding apparatus uses any one of the plurality of prediction motion vector candidates as the prediction motion vector of the current block, the syntax element for the motion information includes the prediction motion vector identification information. Accordingly, in this case, the inter prediction unit 344 may select a candidate indicated by the prediction motion vector identification information from among the prediction motion vector candidates as the prediction motion vector. However, when the image encoding apparatus determines the predicted motion vector by using a function predefined for the plurality of prediction motion vector candidates, the inter prediction unit may determine the predicted motion vector by applying the same function as that of the image encoding apparatus. When the predicted motion vector of the current block is determined, the inter prediction unit 344 determines the motion vector of the current block by adding the predicted motion vector and the differential motion vector transmitted from the decoder 310 . Then, the reference picture referred to by the motion vector of the current block is determined by using the information on the reference picture transmitted from the decoder 310 .

When the motion vector of the current block and the reference picture are determined in the merge mode or the differential motion vector encoding mode, the inter prediction unit 342 generates a prediction block of the current block by using the block at the position indicated by the motion vector in the reference picture. do.

Examples of this disclosure relate generally to bi-directional optical flow (BIO) techniques. As such, certain techniques of the present disclosure may be performed by the inter prediction unit 344 . That is, for example, the inter prediction unit 344 may perform the techniques of the present disclosure described with reference to FIGS. 4 to 13 below. In other words, after determining the bidirectional motion vectors for the current block, the inter prediction unit 344 may generate a prediction block for the current block by using motion compensation according to the BIO technique in units of image pixels or sub-blocks. . In other examples, one or more other units of the decoding apparatus may be additionally involved in performing the techniques of the present disclosure.

The adder 350 reconstructs the current block by adding the residual block output from the inverse transform unit and the prediction block output from the inter prediction unit or the intra prediction unit. Pixels in the reconstructed current block are used as reference pixels when intra-predicting a block to be decoded later.

The filter unit 360 deblocks and filters the boundary between the reconstructed blocks in order to remove a blocking artifact caused by block-by-block decoding and stores the deblocking filter in the memory 370 . When all blocks in one picture are reconstructed, the reconstructed picture is used as a reference picture for inter prediction of blocks in a picture to be decoded later.

The present disclosure relates to using a bi-directional optical flow (BIO) estimation technique to correct motion vector information obtained through inter prediction. The encoding apparatus performs motion estimation and compensation in units of coding blocks (CUs) in an inter prediction process, and then transmits a resultant motion vector (MV) value to the decoding apparatus, and the encoding apparatus and the decoding apparatus may additionally correct the MV value in units of pixels or sub-blocks smaller than CUs (ie, sub-CUs) using BIO based on the MV values in units of CUs. That is, the BIO can precisely compensate for the motion of the coding block (CU) from an n×n block to a 1×1 block (ie, pixel) unit based on the size of each block. In addition, since there is an explicit equation for calculating a motion vector, a search process for acquiring motion information and signaling for transmitting motion information are not required.

4 is a reference diagram for explaining the basic concept of BIO.

BIO used for video encoding and decoding is based on the assumption that motion vector information should be bi-prediction information, and that motion is a steady motion that moves sequentially based on the time axis. 4 shows a current picture (B-picture) referring to two reference pictures (Ref ₀ and Ref _{1 ).}

First, a bidirectional motion vector indicating regions (ie, reference blocks) most similar to the coded current block of the current picture _{in the reference pictures (Ref 0} and Ref ₁ ) by (normal) bidirectional motion prediction for the current block. Assume that (MV ₀ , MV ₁ ) is determined. The two bidirectional motion vectors are values representing the motion of the current block. That is, it is a value obtained by setting the current block as a unit and estimating and compensating for the motion of the unit as a whole.

In the example of FIG. 4 , the pixel in the reference picture Ref _{0 indicated by the bidirectional motion vector MV 0} as corresponding to the pixel P in the current block is P _{0 , and the bidirectional motion vector (} A pixel in the reference picture Ref _{1 pointed to by MV 1 is P 1} . Also, it is assumed that the motion of the pixel P in FIG. 4 is slightly different from the overall motion of the current block. For example, if _{an object located at pixel A in Ref 0} of FIG. 4 moves past pixel P in the current block of the current picture to pixel B in Ref1, pixel A and pixel B have significantly similar values. In addition, in this case, the point in Ref0 that is most similar to the pixel P in the current block is not P0 indicated by the bidirectional motion vector MV0, but A that has moved P0 by a predetermined displacement vector ( v _x τ ₀ , v _y τ _{0 ).} , and the point in Ref1 that is most similar to pixel P in the current block is not P1 indicated by the bidirectional motion vector (MV1), but B, which has shifted P1 by a predetermined displacement vector (- v _x τ ₁ , -v _y τ ₁ ). . Hereinafter, for convenience, ( v _x , v _y ) is referred to as a “BIO motion vector”.

Therefore, in predicting the pixel (P) value of the current block in the current picture, if the values of the two reference pixels (A, B) are used, the reference pixels (P0, P1) indicated by the bidirectional motion vectors (MV0, MV1) are It is possible to make a more accurate prediction than using it. As described above, the concept of changing the reference pixel used to predict one pixel of the current block in consideration of the pixel-wise motion in the current block specified by the BIO motion vector ( v _x , v _{y ) is a sub-block within the current block.} can be extended to

Hereinafter, a theoretical method of generating a prediction value for a pixel in the current block according to the BIO technique will be described. For convenience of description, it is assumed that BIO-based bidirectional motion compensation is performed in units of pixels.

A bidirectional motion vector indicating regions (ie, reference blocks) most similar to the current block to be coded _{in the reference pictures (Ref 0} and Ref ₁ ) by (normal) block-by-block bidirectional motion prediction for the current block. It is assumed that _{MV 0} and MV _{1 are determined.} The decoding apparatus may determine the _{bidirectional motion vectors MV 0} and MV ₁ from motion vector information included in the bitstream. In addition, the luminance value of the pixel in the reference picture Ref ₀ indicated by the motion vectors MV ₀ as corresponding to the pixel ( i,j ) in the current block is called I ⁽⁰⁾ ( i,j ) is defined, and the brightness value of the pixel in the reference picture Ref ₁ indicated by the motion vectors MV ₁ as corresponding to the pixel (i,j ) in the current block is defined as I ⁽¹⁾ ( i,j ).

로 정의될 수 있으며, 참조 픽처 Ref₁ 내의 픽셀 B의 밝기값은

로 정의될 수 있다. 따라서, 픽셀 A와 픽셀 B 사이의 플로우 차이값(flow difference) Δ는 일반적으로 <수학식 1>와 같이 정의된다.The brightness value of pixel A in the _{reference picture Ref 0 indicated by} the BIO motion vector ( v _x , v _y ) as corresponding to the pixel in the current block is

It can be defined as , and the brightness value of pixel B in the _{reference picture Ref 1 is}

can be defined as Accordingly, the flow difference Δ between the pixel A and the pixel B is generally defined as in Equation 1 above.

와

는 각각 I ^(k) 그래디언트(gradient)의 수평 및 수직 성분을 나타낸다. τ ₀와 τ ₁은 현재 픽처와 2개의 참조 픽처(Ref₀과 Ref₁) 간의 시간적 거리를 나타낸다. τ ₀와 τ ₁은 POC(picture order count)를 기반으로 계산될 수 있으며, 예컨대, τ ₀=POC(current)-POC(Ref₀), τ ₁= POC(Ref₁)-POC(current) 이다. 여기서, POC(current), POC(Ref₀), 및 POC(Ref₁)은 각각 현재 픽처, 참조 픽처 Ref₀, 및 참조 픽처 Ref₁의 POC를 의미한다.Here, I ^(k) (k = 0, 1) corresponds to a pixel to be predicted in the current block, and the brightness of a pixel in the reference pictures Ref ₀ and Ref _{1 indicated by the motion vectors MV 0} and MV _{1 .} (luminance), and ( v _x , v _y ) is the BIO motion vector to obtain. For brevity, positions ( i , j _{) of pixels in reference pictures (Ref 0} and Ref ₁ ) in each term of Equation 1 above are omitted.

Wow

denotes the horizontal and vertical components of the I ^{(k) gradient, respectively.} τ ₀ and τ ₁ indicate a temporal distance between the current picture and two reference pictures (Ref ₀ and Ref _{1 ).} τ ₀ and τ ₁ may be calculated based on a picture order count (POC), for example, τ ₀ =POC(current)-POC(Ref ₀ ), τ ₁ = POC(Ref ₁ )-POC(current). . Here, POC(current), POC(Ref ₀ ), and POC(Ref ₁ ) refer to POCs of the current picture, the reference picture Ref ₀ , and the reference picture Ref ₁ , respectively.

Movement within the geographically close based on the assumption that the pixels and consistency, the pixel constant region Ω centered on the pixel (i, j) of BIO motion vectors to the current prediction for the (i, j) to the current prediction Consider the difference values Δ in <Equation 1> for all pixels ( i' , j') present. That is, the BIO motion vector for the current pixel ( i , j ) minimizes the sum of squares of the difference value Δ[i',j' ] obtained for each pixel within a certain region Ω as shown in Equation 2 below. It can be determined by a vector that

Here, ( i' , j' ) means all pixels located within the search area Ω. Therefore, as shown in <Equation 2>, the BIO motion vector ( v _x , v _y ) of the current pixel can be determined by calculating an explicit equation that minimizes the ^{objective function (the sum of Δ 2 ) at the current pixel position.} There is no need for a search process for acquiring the specific motion information and signaling for transmitting the motion information.

In general, the search area Ω may be defined as a masking window having a size of (2M+1)×(2N+1) centered on the current pixel (i , j). The structure and size of the masking window have a significant impact on the complexity of the algorithm for determining the BIO motion vector (v _x , v _{y ) and its precision.} Therefore, the selection of the masking window is very important for the algorithm for determining the BIO motion vector ( v _x , v _{y ).}

When the BIO motion vector ( v _x , v _y ) of the current pixel is determined, a bidirectional prediction value ( pred _BIO ) based on the BIO motion vector for the current pixel ( i , j ) may be calculated by <Equation 3>.

In <Equation 3>, ( I ⁽⁰⁾ + I ⁽¹⁾ )/2 is a typical block-wise bidirectional prediction compensation, so the remaining terms may be referred to as BIO offsets.

Hereinafter, a BIO-based bidirectional motion compensation method will be described with reference to FIGS. 5A and 5B . The following method is common to the video encoding apparatus and the video decoding apparatus. Although not shown in FIG. 5 , it is assumed that the encoding apparatus encodes pictures for use as reference pictures, then decodes them, and stores them in a memory. Also, it is assumed that the decoding apparatus decodes pictures to be used as reference pictures and stores them in a memory.

5A is a flowchart illustrating a bidirectional motion compensation method performed based on pixel-level BIO according to an embodiment of the present invention.

First, the encoding apparatus and the decoding apparatus determine a first motion vector indicating a first correspondence region most similar to a current block in a first reference picture, and a second motion vector indicating a second correspondence region most similar to the current block in a second reference picture. A motion vector is determined (S510).

The encoding apparatus and the decoding apparatus determine the BIO motion vectors v _x , v _y corresponding to each pixel in the current block by applying the BIO process in units of pixels ( S520 ).

BIO motion vectors ( v _x , v _y ) are flow differences expressed by <Equation 1> for each pixel located in a search area defined by a predetermined masking window centered on the corresponding pixel. It can be determined as a vector such that the sum of squares is minimized.

In some examples, in determining the BIO motion vector for a pixel located at the edge of the current block, flow differences for pixels located outside the current block may not be considered.

In some examples, a rectangular-shaped masking window having a size of (2M + 1) × (2N + 1) may be used. Preferably, a masking window in the shape of a square, eg 5×5 in size, may be used. In some other examples, a non-rectangular masking window may be used, such as a plus shape or a diamond shape.

The encoding apparatus and the decoding apparatus generate a prediction block for the current block by using bi-directional prediction based on the BIO motion vectors (v _x , v _{y ) calculated in units of pixels ( S530 ).} That is, the encoding apparatus and the decoding apparatus generate a bidirectional prediction value of a corresponding pixel based on Equation 3 by using each BIO motion vector.

Finally, the encoding apparatus and the decoding apparatus encode or decode pixels of the current block by using the generated prediction block ( S540 ).

5B is a flowchart illustrating a bidirectional motion compensation method performed based on sub-block level BIO according to an embodiment of the present invention.

First, the encoding apparatus and the decoding apparatus determine a first motion vector indicating a first correspondence region most similar to a current block in a first reference picture, and a second motion vector indicating a second correspondence region most similar to the current block in a second reference picture. A motion vector is determined (S560).

The encoding apparatus and the decoding apparatus determine the BIO motion vectors v _x , v _y corresponding to each sub-block by applying the BIO process to each sub-block in the current block ( S570 ).

The BIO motion vector ( v _x , v _y ) is a vector such that the sum of squares of flow differences for each pixel located in the search area defined by a predetermined masking window centered on each pixel in the sub-block is minimized. can Alternatively, the BIO motion vector ( v _x , v _y ) is the square of the flow differences for all pixels located within the search area defined by a predetermined masking window centered on some pixels in the sub-block. It can be determined as a vector such that the sum is minimized. For example, positions of pixels to which the masking window is applied and pixels to which the masking window is not applied may form a check pattern, a horizontal stripes pattern, or a vertical stripes pattern.

In some examples, instead of repeatedly calculating the flow differences, a weight may be given to the overlapping difference value according to the number of overlapping differences. In some examples, in determining the BIO motion vector for a sub-block located at the edge of the current block, flow differences for pixels located in an area outside the current block may not be considered.

In some examples, a rectangular-shaped masking window having a size of (2M + 1) × (2N + 1) may be used. In some examples, the masking window may be square in shape (eg, 5x5 in size). In some other examples, non-rectangular masking windows may be used, such as, for example, plus shapes and diamond shapes. In some examples, a masking window may not be used. For example, the BIO motion vectors ( v _x , v _y ) may be determined as vectors such that the sum of squares of flow differences for each pixel in the sub-block is minimized.

The encoding apparatus and the decoding apparatus generate a prediction block for the current block using bi-directional prediction based on the BIO motion vectors ( v _x , v _{y ) calculated in units of sub-blocks ( S580 ).} All pixels in a sub-block share the BIO motion vector ( v _x , v _{y ) calculated for each sub-block.} That is, BIO-based prediction values for all pixels in a sub-block are calculated by <Equation 3> using one BIO motion vector (v _x , v _{y) determined for the sub-block.}

Finally, the encoding apparatus and the decoding apparatus encode or decode pixels of the current block by using the generated prediction block (S590).

In some embodiments of the present disclosure, BIO is applied on a pixel-level basis, and in some other embodiments, BIO is applied on a block-level basis. Hereinafter, embodiments related to a pixel-level BIO process will be first described, and then embodiments related to a block-level BIO process will be described.

In the first and second embodiments described below, BIO is applied on a pixel-level basis. The masking window used in the BIO process may have a size of (2M + 1) × (2N + 1) centered at the current pixel ( i , j ), but for simplicity of explanation, hereinafter, horizontal and vertical sizes are Assume that they have the same value (ie, M=N). The pixel-level BIO obtains a BIO motion vector at the pixel-level when generating a prediction block of the current block, and obtains a bidirectional prediction value at the pixel-level based on the obtained BIO motion vector.

first embodiment

In this embodiment, a rectangular masking window is used in calculating the BIO motion vector at the pixel level. In the present embodiment, the total number of difference values Δ required to determine the BIO motion vector of a pixel to be predicted will be described with reference to FIG. 6 . 6 illustrates a 5×5 masking window 610 and a pixel 621 to be predicted in the current block. One pixel 621 to be predicted in the current block becomes the center of the masking window 810 indicated by hatching in FIG. 6, and the number of pixels located in the masking window 810 includes the pixel 821 to be predicted. This makes a total of 25. Therefore, the number of difference values (Δ) required to determine the BIO motion vector ( v _x , v _y ) for the pixel 621 to be predicted in the current block is 25. Finally, the BIO motion vector (v _x , v _y ) for the pixel to be predicted is obtained by substituting the corresponding 25 difference values (Δ) into <Equation 2>. When the BIO motion vector ( v _x , v _y ) is determined based on the optical flow, the bidirectional prediction value for the corresponding pixel of the current block is calculated by <Equation 3>. This process is iteratively applied to each pixel in the current block to generate prediction values of all pixels constituting the prediction block of the current block.

However, in determining the BIO motion vector for the pixel located at the edge of the current block, even if it is included in the masking window, flow differences with respect to pixels located outside the current block may not be considered.

second embodiment

7 is a diagram illustrating non-rectangular masking windows used for BIO-based motion compensation according to the second embodiment.

Unlike the first embodiment in which a square-shaped masking window is used, this embodiment uses masking windows of various shapes. Although FIG. 7 shows two types of masking windows (ie, a plus shape and a diamond shape), the present invention does not exclude the use of any masking window of various shapes other than a rectangular masking window. . The use of such a masking window reduces the complexity of processing all pixels in the square-shaped masking window used in the first embodiment. As illustrated in FIG. 7 , the sizes of the plus-shaped and diamond-shaped masking windows may be scaled depending on the parameter M value.

In the present embodiment, the total number of difference values Δ required to determine the BIO motion vector of the sub-block will be described with reference to FIG. 8 .

8 illustrates a diamond-shaped masking window 810 with M=2 and a pixel 821 to be predicted in the current block. One pixel 821 to be predicted in the current block becomes the center of the masking window 810 indicated by hatching in FIG. There are a total of 13 including Therefore, the number of difference values (Δ) required to determine the BIO motion vector ( v _x , v _y ) for the pixel 821 to be predicted in the current block is 13. Finally, by substituting the corresponding 13 difference values (Δ) into <Equation 2>, the BIO motion vector ( v _x , v _y ) for the pixel to be predicted is obtained. In this embodiment, these processes are performed for all pixels in the current block, and a BIO motion vector corresponding to each pixel is calculated.

However, in determining the BIO motion vector for the pixel located at the edge of the current block, even if included in the masking window, flow differences with respect to pixels located outside the current block may not be considered.

In the third to eighth embodiments described below, BIO-based motion compensation is applied at the block-level. In the sub-block-level BIO motion compensation process, the sub-block size may be M × N (M and N are integers). All pixels in an M × N sub-block share a BIO motion vector ( v _x , v _{y ) calculated for each sub-block.} That is, the bidirectional prediction based on the optical flow for all pixels in the M × N sub-block is calculated by Equation 3 using the calculated BIO motion vectors (v _x , v _{y ).} Note that although the methods of the present disclosure do not limit the size of a sub-block, the BIO process is described based on a 4×4 sub-block for convenience of description in the following embodiments.

third embodiment

In this embodiment, in order to determine one BIO motion vector for a sub-block, a square-shaped masking window centered on the pixel is applied to each pixel in the sub-block, and all pixels located in the masking window are each Find the difference value (Δ) of <Equation 1> for Finally, by substituting these difference values into <Equation 2>, a BIO motion vector corresponding to the corresponding sub-block is obtained.

9 shows an example of a 5×5 masking window 910 and a 4×4 sub-block 920 according to an example of the method proposed in this embodiment. The masking window 910 illustrated in FIG. 9 has a square shape with M=2. The current pixel ( i , j ) 921 in the sub-block 920 becomes the center of the masking window 910 corresponding to the hatched portion of FIG. 9 . The number of pixels in the masking window 910 for one pixel ( i , j ) of the sub-block is a total of 25 (= (2M + 1) × (2M + 1) = 5 × 5 including ( i , j ) pixels) ) becomes a dog. Therefore, the total number of difference values required to determine the BIO motion vector for the 4×4 sub-block reaches 400 (= 16×25) in total based on the size of the sub-block and the size of the masking window. The BIO motion vector for the sub-block is determined as a vector that minimizes the sum of squares of these difference values.

Note that, among the above 400 difference values, the remaining difference values except for the 64 distinct difference values are in the form of overlapping the 64 difference values. For example, as shown in FIG. 10A , most of the pixels located in the masking window 1010a centered on the pixel at the (0,0) position of the sub-block 1020 are at the (1,0) position of the sub-block 1020 , as shown in FIG. 10A . It is also located within the masking window 1010b centered on the pixel of . Accordingly, the calculation of <Equation 2> can be simplified by assigning a weight to the overlapping difference value according to the number of overlapping times, instead of repeatedly calculating the overlapping difference values. For example, when a 5×5 masking window is applied to a 4×4 sub-block, after calculating a total of 64 distinct difference values, a weight is assigned to each difference value, and then a weighted difference value is applied. The BIO motion vector ( v _x , v _y ) can be determined so that the sum of squares of the values is minimized. In FIG. 10B , a number marked on each pixel is a weight value according to the number of overlapping times. Here, the highlighted 4×4 block indicates the position of the sub-block.

4th embodiment

Unlike the third embodiment, which uses a square-shaped masking window, this embodiment uses masking windows of various patterns (as illustrated in FIG. 7 ). The use of such a masking window reduces the complexity of processing every pixel in a rectangular masking window.

In FIG. 11 , a diamond-type masking window 1110 and a 4×4 sub-block 1120 are illustrated. As illustrated in FIG. 11 , when the diamond-shaped masking window 1110 with M=2 is used, the number of pixels in the masking window 1110 becomes 13 in total. Therefore, the total number of difference values (Δ) required to determine the BIO motion vector ( v _x , v _{y ) of the sub-block is 208 (= 16 × 13).} Finally, by substituting the 208 difference values into <Equation 2>, a BIO motion vector corresponding to the corresponding 4×4 block is obtained. As in the third embodiment, a BIO motion vector for the corresponding 4×4 sub-block may be obtained by substituting a weight according to the number of overlaps to the difference value and substituting it in <Equation 2>.

5th embodiment

In the third and fourth embodiments, a masking window was applied to all pixels in the sub-block. In contrast, the present embodiment applies a masking window to some pixels in a sub-block.

12 exemplifies three types of positions of pixels to which a masking window is applied in a sub-block. In one type, positions of pixels to which a masking window is applied and pixels to which a masking window is not applied form a check pattern (see FIG. 12(a) ), and in the other two types, horizontal stripes It forms a pattern and a vertical stripes pattern (see FIGS. 12 (b) and (c)). The present invention does not exclude the use of any type of sampling and processing only some pixels in a sub-block, other than the types illustrated in FIG. 12 . Through this, in the above-described embodiments, it is possible to reduce the computational complexity required for calculating the number of difference values corresponding to the masking windows for all pixels in the sub-block.

In the present embodiment, the total number of difference values Δ required to determine the BIO motion vector of the sub-block will be described with reference to FIG. 13 . 13 illustrates a 5×5 square-shaped masking window 1310 and pixels in a 4×4 sub-block 1320 sampled in a check pattern. The number of pixels in the 5×5 square-shaped masking window 1310 is 25 in total. For 8 pixels indicated by hatching in the sub-block, a masking window is applied to each pixel to obtain a difference value Δ of 25 <Equation 1>. Therefore, the total number of difference values (Δ) required to determine the BIO motion vectors ( v _x , v _y ) of the 4×4 sub-block amounts to a total of 200 (= 8×25). Finally, by substituting the 200 difference values into <Equation 2>, a BIO motion vector corresponding to the corresponding 4×4 block is obtained. As in the third embodiment, a BIO motion vector for the corresponding 4×4 sub-block may be obtained by substituting a weight according to the number of overlaps to the difference value and substituting it in <Equation 2>.

6th embodiment

This embodiment is a combined form of the methods presented in the fourth and fifth embodiments. That is, the present embodiment employs a masking window of various patterns rather than a rectangular shape (similar to the fourth embodiment), and employs a method of applying the masking window only to some sample pixels in the sub-block (the fifth embodiment) similar to). Accordingly, the technique of the present embodiment has lower computational complexity than the fourth and fifth embodiments.

In FIG. 14 , sample pixels to which a BIO process is applied in a diamond-type masking window 1410 and a 4×4 sub-block 1420 are illustrated according to an example of the method proposed in this embodiment. In the case of FIG. 14 , the total number of difference values Δ required to determine the BIO motion vector ( v _x , v _{y ) of the sub-block is 104 (= 8×13).} Finally, by substituting the 104 difference values into <Equation 2>, the BIO motion vector ( v _x , v _y ) corresponding to the corresponding 4×4 sub-block is obtained. As in the third embodiment, a BIO motion vector for the corresponding 4×4 sub-block may be obtained by substituting a weight according to the number of overlaps to the difference value and substituting it in <Equation 2>.

7th embodiment

In the previous embodiments, a number of difference values Δ corresponding to the size of the window mask were calculated for (all or some) pixels of the sub-block. For example, in the third embodiment, using a 5×5 masking window, the number of difference values required to determine a BIO motion vector for a 4×4 sub-block reached a total of 400 (= 16×25). In contrast, this embodiment does not use a masking window. Alternatively, it can be seen that a 1×1 masking window is used. That is, for each of the pixels in the sub-block, only one difference value Δ of <Equation 1> is calculated. For example, the number of difference values Δ considered to obtain a BIO motion vector for a 4×4 subblock is only 16 in total. Finally, only 16 difference values (Δ) are substituted into <Equation 2> to obtain a BIO motion vector for a 4×4 subblock. That is, the BIO motion vector is calculated to minimize the sum of the squares of 16 difference values.

Alternatively, a BIO motion vector corresponding to the corresponding 4×4 sub-block may be obtained by giving different weights to the 16 difference values and substituting them in <Equation 2>. Here, a higher weight may be given to the inside of the sub-block, and a lower weight may be given to the sub-block edge region. 15 shows an example of weighting each pixel of a sub-block.

eighth embodiment

In the present embodiment, when determining the BIO motion vectors for sub-blocks located at the edge of the current block, there is a restriction that the difference value Δ is not calculated for an area outside the current block. For example, as illustrated in FIG. 16A , it is assumed that the size of the current block is 16×16 and the BIO motion vector is calculated in units of 4×4 sub-blocks. When determining the BIO motion vectors of 12 4x4 subblocks located at the edge of the current block among a total of 16 4x4 subblocks, the difference value Δ for the masking pixels located outside the current block is do not consider Here, the masking pixel located in the area outside the current block may vary according to the size of the sub-block and the size and position of the masking window. Accordingly, in the present embodiment, the number of difference values Δ to be calculated to determine the BIO motion vector of a sub-block may vary depending on the position of the sub-block in the current block.

When this method is combined with the method of assigning weights to overlapping difference values described in the third embodiment, the weights for each masking pixel are as shown in FIG. 16B . That is, pixels marked with 0 in FIG. 16B are pixels located outside the current block, and a difference value is not calculated therebetween. According to this method, compared to the third embodiment, since the number of difference values to be calculated is small, the amount of calculation is reduced, and since the pixel value located outside the current block is not referenced, the amount of memory usage can be saved.

Note that this method is not limited to the case of using a square masking window, and can be applied to a case where masking windows of various shapes including diamonds and pluses are used.

The above description is merely illustrative of the technical idea of this embodiment, and various modifications and variations will be possible by those skilled in the art to which this embodiment belongs without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain rather than limit the technical spirit of the present embodiment, and the scope of the technical spirit of the present embodiment is not limited by these embodiments. The protection scope of this embodiment should be interpreted by the following claims, and all technical ideas within the equivalent range should be interpreted as being included in the scope of the present embodiment.

Claims (12)

A method for encoding image data, the method comprising:
determining a first motion vector indicating a first region for predicting a current block in a first reference picture and determining a second motion vector indicating a second region for predicting the current block in a second reference picture;
generating a prediction block for the current block by applying a sub-block-based BIO process;
A process of encoding the current block using the generated prediction block
including,
The process of generating the prediction block is
determining a BIO motion vector for each sub-block constituting the current block; and
A process of generating prediction values of pixels constituting a corresponding sub-block based on the determined BIO motion vector
including,
The size of the sub-block is 4x4, the encoding method. According to claim 1,
The BIO motion vector is determined based on flow differences obtained for pixels in a square block surrounding the corresponding sub-block,
and a flow difference for a given pixel in the square block is calculated between a first point on the first reference picture corresponding to the given pixel and a second point on the second reference picture. 3. The method of claim 2,
The BIO motion vector is,
The encoding method, characterized in that it is determined as a vector such that the sum of squares of flow differences obtained for each of the pixels in the square block surrounding the corresponding sub-block or the weighted sum of squares is minimized. 4. The method of claim 3,
An encoding method, characterized in that a higher weight is given to a flow difference obtained with respect to pixels located inside the square block surrounding the corresponding sub-block. A method of decoding image data, the method comprising:
determining a first motion vector indicating a first region for predicting a current block in a first reference picture and determining a second motion vector indicating a second region for predicting the current block in a second reference picture;
generating a prediction block for the current block by applying a sub-block-based BIO process; and
Reconstructing the current block using the prediction block,
The process of generating the prediction block is
determining a BIO motion vector for each sub-block constituting the current block; and
A process of generating prediction values of pixels constituting a corresponding sub-block based on the determined BIO motion vector
including,
The size of the sub-block is 4x4, the decoding method. 6. The method of claim 5,
The BIO motion vector is determined based on flow differences obtained for pixels in a square block surrounding the corresponding sub-block,
and a flow difference for a given pixel in the square block is calculated between a first point on the first reference picture and a second point on the second reference picture corresponding to the given pixel. 7. The method of claim 6,
The BIO motion vector is,
The decoding method, characterized in that it is determined as a vector such that the sum of squares of flow differences obtained for each of the pixels in the square block surrounding the corresponding sub-block or the weighted sum of squares is minimized. 8. The method of claim 7,
The decoding method, characterized in that a higher weight is given to a flow difference obtained with respect to pixels located inside the square block surrounding the corresponding sub-block. A computer-readable non-transitory recording medium storing a bitstream including encoded data for image data, the bitstream comprising:
determining a first motion vector indicating a first region for predicting a current block in a first reference picture and determining a second motion vector indicating a second region for predicting the current block in a second reference picture;
generating a prediction block for the current block by applying a sub-block-based BIO process;
A process of encoding the current block using the prediction block
It is generated by an encoding process comprising
The process of generating the prediction block is
determining a BIO motion vector for each sub-block constituting the current block; and
A process of generating prediction values of pixels constituting a corresponding sub-block based on the determined BIO motion vector
including,
The size of the sub-block is 4x4, a computer-readable non-transitory recording medium. 10. The method of claim 9,
The BIO motion vector is determined based on flow differences obtained for pixels in a square block surrounding the corresponding sub-block,
wherein a flow difference for a given pixel within the square block is calculated between a first point on the first reference picture corresponding to the given pixel and a second point on the second reference picture. media. 11. The method of claim 10,
The BIO motion vector is,
A computer-readable non-transitory recording medium, characterized in that it is determined as a vector such that the sum of squared or weighted sum of squares of flow differences obtained for each of the pixels in the square block surrounding the corresponding sub-block is minimized. 12. The method of claim 11,
A computer-readable non-transitory recording medium, characterized in that a higher weight is given as the flow difference obtained with respect to the pixels located inside the square block surrounding the corresponding sub-block is increased.

Images