KR20210038289A - Method and apparatus for constructing reference sample - Google Patents

Abstract

Disclosed are a method for constructing a reference sample and image decoder to improve the encoding and decoding efficiency by reflecting a correlation between reference samples in the configuration of reference samples. According to one embodiment of the present invention, the method for constructing reference samples used for the prediction of a current block comprises the steps of: determining the availability of a current reference sample to be constructed; determining the availability of other reference samples located around the current reference sample when the current reference sample is not available; and padding the current reference sample based on the correlation between the other reference samples when the other reference samples are available.

Description

Reference sample composition method and video decoding device {METHOD AND APPARATUS FOR CONSTRUCTING REFERENCE SAMPLE}

The present invention relates to encoding and decoding of an image, and more particularly, to a reference sample construction method and an image decoding apparatus in which the efficiency of encoding and decoding is improved by generating a reference sample more accurately.

Since moving picture data has a larger amount of data than audio data or still image data, it requires a lot of hardware resources including memory in order to store or transmit itself without processing for compression.

Accordingly, when moving or transmitting moving picture data, the moving picture data is compressed and stored or transmitted using an encoder, and the decoder receives the compressed moving picture data, decompresses and reproduces the compressed moving picture data. As such video compression techniques, there are H.264/AVC and HEVC (High Efficiency Video Coding), which improves coding efficiency by about 40% compared to H.264/AVC.

However, the size, resolution, and frame rate of an image are gradually increasing, and accordingly, the amount of data to be encoded is also increasing. Accordingly, a new compression technique having higher encoding efficiency and higher quality improvement effect than the existing compression technique is required.

In order to meet these needs, an object of the present invention is to provide an improved video encoding and decoding technology. In particular, an aspect of the present invention reflects the correlation between reference samples in the configuration of a reference sample. It relates to technology that improves efficiency.

An aspect of the present invention is a method of constructing reference samples used for prediction of a current block, the method comprising: determining whether or not a current reference sample to be constructed is available; If the current reference sample is not available, determining whether other reference samples located around the current reference sample are available; And if the other reference samples are available, padding the current reference sample based on a correlation between the other reference samples.

Another aspect of the present invention is an image decoding apparatus that configures reference samples used for prediction of a current block, and determines whether a current reference sample as a configuration target is available, and when the current reference sample is not available, the A determination unit that determines whether other reference samples located in the vicinity of the current reference sample can be used; And a padding unit for padding the current reference sample based on a correlation between the other reference samples when the other reference samples are available.

As described above, according to an embodiment of the present invention, the accuracy of the process of constructing the reference sample may be improved by padding the reference sample by reflecting the correlation between the reference samples.

In addition, according to another embodiment of the present invention, since a reference sample can be more accurately configured, it is possible to improve not only prediction accuracy but also compression performance.

1 is an exemplary block diagram of an image encoding apparatus capable of implementing the techniques of the present disclosure.
2 is a diagram for explaining a method of dividing a block using a QTBTTT structure.
3 is a diagram for describing a plurality of intra prediction modes.
4 is an exemplary block diagram of an image decoding apparatus capable of implementing the techniques of the present disclosure.
5 is a diagram for explaining reference samples used for intra prediction and a conventional method of configuring the reference samples.
6 is a diagram for describing a method of configuring reference samples used for inter prediction.
7 is an exemplary block diagram of an image decoding apparatus capable of implementing a method of constructing a reference sample.
8 is a flowchart illustrating an example of a method of configuring a reference sample.
9 is a flowchart illustrating an example of a method of constructing a reference sample based on a linear relationship.
10 is a flowchart illustrating an example of a method of configuring a reference sample based on a gradient.
11 is a flowchart illustrating an example of a method of configuring a reference sample based on a linear relationship and a gradient.
12 is a flowchart illustrating a syntax structure for a method of constructing a reference sample.
13 is a flowchart illustrating a method of constructing a reference sample when all reference samples are not available.

Hereinafter, some embodiments of the present invention will be described in detail through exemplary drawings. In adding identification symbols to elements of each drawing, it should be noted that the same elements are to have the same symbols as possible even if they are indicated on different drawings. In addition, in describing the present invention, when it is determined that a detailed description of a related known configuration or function may obscure the subject matter of the present invention, a detailed description thereof will be omitted.

1 is an exemplary block diagram of an image encoding apparatus capable of implementing the techniques of the present disclosure. Hereinafter, an image encoding apparatus and sub-elements of the apparatus will be described with reference to FIG. 1.

The image encoding apparatus includes a block division unit 110, a prediction unit 120, a subtractor 130, a transform unit 140, a quantization unit 145, an encoding unit 150, an inverse quantization unit 160, and an inverse transform unit ( 165, an adder 170, a filter unit 180, and a memory 190 may be included.

Each component of the image encoding apparatus may be implemented as hardware or software, or may be implemented as a combination of hardware and software. In addition, functions of each component may be implemented as software, and a microprocessor may be implemented to execute a function of software corresponding to each component.

One image (video) is composed of a plurality of pictures. Each picture is divided into a plurality of regions, and encoding is performed for each region. For example, one picture is divided into one or more tiles or/and slices. Here, one or more tiles may be defined as a tile group. Each tile or/slice is divided into one or more Coding Tree Units (CTUs). And each CTU is divided into one or more CUs (Coding Units) by a tree structure. Information applied to each CU is encoded as the syntax of the CU, and information commonly applied to CUs included in one CTU is encoded as the syntax of the CTU. In addition, information commonly applied to all blocks in one tile is encoded as the syntax of a tile or is encoded as a syntax of a tile group in which a plurality of tiles are collected, and information applied to all blocks constituting one picture is It is encoded in a picture parameter set (PPS) or a picture header. Furthermore, information commonly referred to by a plurality of pictures is encoded in a sequence parameter set (SPS). In addition, information commonly referred to by one or more SPSs is encoded in a video parameter set (VPS).

The block dividing unit 110 determines the size of a coding tree unit (CTU). Information on the size of the CTU (CTU size) is encoded as the syntax of the SPS or PPS and transmitted to the video decoding apparatus.

The block dividing unit 110 divides each picture constituting an image into a plurality of coding tree units (CTUs) having a predetermined size, and then repeatedly divides the CTU using a tree structure. Split (recursively). A leaf node in a tree structure becomes a coding unit (CU), which is a basic unit of coding.

In a tree structure, a quadtree (QuadTree, QT) in which an upper node (or parent node) is divided into four lower nodes (or child nodes) of the same size, or a binary tree (BinaryTree) in which an upper node is divided into two lower nodes. , BT), or a ternary tree (TT) in which an upper node is divided into three lower nodes in a 1:2:1 ratio, or a structure in which two or more of these QT structures, BT structures, and TT structures are mixed. have. For example, a QTBT (QuadTree plus BinaryTree) structure may be used, or a QTBTTT (QuadTree plus BinaryTree TernaryTree) structure may be used. Here, BTTT may be collectively referred to as MTT (Multiple-Type Tree).

2 shows a QTBTTT split tree structure. As shown in FIG. 2, the CTU may be first divided into a QT structure. The quadtree division may be repeated until the size of a splitting block reaches the minimum block size (MinQTSize) of a leaf node allowed in QT. A first flag (QT_split_flag) indicating whether each node of the QT structure is divided into four nodes of a lower layer is encoded by the encoder 150 and signaled to the image decoding apparatus. If the leaf node of the QT is not larger than the maximum block size (MaxBTSize) of the root node allowed in BT, it may be further divided into one or more of a BT structure or a TT structure. In the BT structure and/or the TT structure, a plurality of division directions may exist. For example, there may be two directions in which a block of a corresponding node is divided horizontally and a direction vertically divided. As shown in FIG. 2, when MTT splitting starts, a second flag (mtt_split_flag) indicating whether nodes are split, and if split, a flag indicating a split direction (vertical or horizontal) and/or a split type (Binary or Ternary). A flag indicating) is encoded by the encoder 150 and signaled to the image decoding apparatus.

As another example of the tree structure, when a block is divided using a QTBTTT structure, information about a CU split flag (split_cu_flag) indicating that the block has been split first and a QT split flag (split_qt_flag) indicating whether the split type is QT splitting is information from the encoding unit 150 ) And signaled to the video decoding apparatus. When it is indicated that the value of the CU split flag (split_cu_flag) is not split, the block of the corresponding node becomes a leaf node in the split tree structure and becomes a coding unit (CU), which is a basic unit of encoding. When indicating that the value of the CU split flag (split_cu_flag) is split, whether the split type is QT or MTT is distinguished through the value of the QT split flag (split_qt_flag). If the split type is QT, there is no additional information, and if the split type is MTT, additionally, a flag indicating the MTT split direction (vertical or horizontal) (mtt_split_cu_vertical_flag) and/or a flag indicating the MTT split type (Binary or Ternary) (mtt_split_cu_binary_flag) is encoded by the encoder 150 and signaled to the video decoding apparatus.

When QTBT is used as another example of the tree structure, there are two types of horizontally splitting a block of a corresponding node into two blocks of the same size (i.e., symmetric horizontal splitting) and a type splitting vertically (i.e., symmetric vertical splitting). Branches can exist. A split flag indicating whether each node of the BT structure is divided into blocks of a lower layer and split type information indicating a type to be divided are encoded by the encoder 150 and transmitted to the image decoding apparatus. Meanwhile, there may be additional types of dividing the block of the corresponding node into two blocks of an asymmetric form. The asymmetric form may include a form of dividing a block of a corresponding node into two rectangular blocks having a size ratio of 1:3, or a form of dividing a block of a corresponding node in a diagonal direction.

The CU may have various sizes according to the QTBT or QTBTTT split from the CTU. Hereinafter, a block corresponding to a CU to be encoded or decoded (ie, a leaf node of QTBTTT) is referred to as a'current block'.

The prediction unit 120 predicts the current block and generates a prediction block. The prediction unit 120 includes an intra prediction unit 122 and an inter prediction unit 124.

In general, each of the current blocks in a picture can be predictively coded. In general, prediction of the current block is performed using an intra prediction technique (using data from a picture containing the current block) or an inter prediction technique (using data from a picture coded before a picture containing the current block). Can be done. Inter prediction includes both one-way prediction and two-way prediction.

The intra prediction unit 122 predicts pixels in the current block by using pixels (reference pixels) located around the current block in the current picture including the current block. There are a plurality of intra prediction modes according to the prediction direction. For example, as shown in FIG. 3, the plurality of intra prediction modes may include a non-directional mode including a planar mode and a DC mode, and 65 directional modes. Depending on each prediction mode, the surrounding pixels to be used and the calculation expression are defined differently.

The intra prediction unit 122 may determine an intra prediction mode to be used to encode the current block. In some examples, the intra prediction unit 122 may encode the current block using several intra prediction modes, and select an appropriate intra prediction mode to use from the tested modes. For example, the intra prediction unit 122 calculates rate distortion values using rate-distortion analysis for several tested intra prediction modes, and has the best rate distortion characteristics among the tested modes. It is also possible to select an intra prediction mode.

The intra prediction unit 122 selects one intra prediction mode from among a plurality of intra prediction modes, and predicts the current block by using a neighboring pixel (reference pixel) and an equation determined according to the selected intra prediction mode. Information on the selected intra prediction mode is encoded by the encoder 150 and transmitted to the image decoding apparatus.

The inter prediction unit 124 generates a prediction block for the current block through a motion compensation process. A block that is most similar to a current block is searched for in a reference picture that is coded and decoded before the current picture, and a prediction block for the current block is generated using the searched block. Then, a motion vector corresponding to a displacement between the current block in the current picture and the prediction block in the reference picture is generated. In general, motion estimation is performed on a luma component, and a motion vector calculated based on the luma component is used for both the luma component and the chroma component. Motion information including information on a reference picture used to predict a current block and information on a motion vector is encoded by the encoder 150 and transmitted to an image decoding apparatus.

The subtractor 130 generates a residual block by subtracting the prediction block generated by the intra prediction unit 122 or the inter prediction unit 124 from the current block.

The transform unit 140 converts the residual signal in the residual block having pixel values in the spatial domain into transform coefficients in the frequency domain. The transform unit 140 may transform residual signals in the residual block using the total size of the residual block as a transform unit, or divide the residual block into two sub-blocks, which are transform regions and non-transform regions, Residual signals can be transformed using only a block as a transform unit. Here, the transform region subblock may be one of two rectangular blocks having a size ratio of 1:1 based on the horizontal axis (or vertical axis). In this case, a flag indicating that only the subblock has been transformed (cu_sbt_flag), directional (vertical/horizontal) information (cu_sbt_horizontal_flag), and/or location information (cu_sbt_pos_flag) are encoded by the encoder 150 and signaled to the video decoding apparatus. . In addition, the size of the transform region subblock may have a size ratio of 1:3 based on the horizontal axis (or vertical axis), and in this case, a flag (cu_sbt_quad_flag) that separates the division is additionally encoded by the encoder 150 to decode the image. Signaled to the device.

The quantization unit 145 quantizes the transform coefficients output from the transform unit 140 and outputs the quantized transform coefficients to the encoding unit 150.

The encoding unit 150 generates a bitstream by encoding the quantized transform coefficients using an encoding method such as Context-based Adaptive Binary Arithmetic Code (CABAC). The encoder 150 encodes information such as a CTU size related to block division, a CU division flag, a QT division flag, an MTT division direction, and an MTT division type, so that the video decoding apparatus can divide the block in the same manner as the video encoding apparatus. To be there.

In addition, the encoder 150 encodes information on a prediction type indicating whether the current block is encoded by intra prediction or inter prediction, and intra prediction information (ie, intra prediction mode) according to the prediction type. Information) or inter prediction information (information on a reference picture and a motion vector) is encoded.

The inverse quantization unit 160 inverse quantizes the quantized transform coefficients output from the quantization unit 145 to generate transform coefficients. The inverse transform unit 165 converts transform coefficients output from the inverse quantization unit 160 from the frequency domain to the spatial domain to restore the residual block.

The addition unit 170 restores the current block by adding the reconstructed residual block and the prediction block generated by the prediction unit 120. The pixels in the reconstructed current block are used as reference pixels when intra-predicting the next block.

The filter unit 180 filters reconstructed pixels to reduce blocking artifacts, ringing artifacts, blurring artifacts, etc. that occur due to block-based prediction and transformation/quantization. Perform. The filter unit 180 may include a deblocking filter 182 and a sample adaptive offset (SAO) filter 184.

The deblocking filter 180 filters the boundary between reconstructed blocks to remove blocking artifacts caused by block-based encoding/decoding, and the SAO filter 184 is additionally applied to the deblocking filtered image. Perform filtering. The SAO filter 184 is a filter used to compensate for a difference between a reconstructed pixel and an original pixel caused by lossy coding.

The reconstructed block filtered through the deblocking filter 182 and the SAO filter 184 is stored in the memory 190. When all blocks in one picture are reconstructed, the reconstructed picture is used as a reference picture for inter prediction of a block in a picture to be encoded later.

4 is an exemplary block diagram of an image decoding apparatus capable of implementing the techniques of the present disclosure. Hereinafter, an image decoding apparatus and sub-elements of the apparatus will be described with reference to FIG. 4.

The image decoding apparatus may include a decoding unit 410, an inverse quantization unit 420, an inverse transform unit 430, a prediction unit 440, an adder 450, a filter unit 460, and a memory 470. have.

Like the image encoding apparatus of FIG. 1, each component of the image decoding apparatus may be implemented as hardware or software, or may be implemented as a combination of hardware and software. In addition, functions of each component may be implemented as software, and a microprocessor may be implemented to execute a function of software corresponding to each component.

The decoding unit 410 determines the current block to be decoded by decoding the bitstream received from the video encoding apparatus and extracting information related to block division, and information on prediction information and residual signals necessary to restore the current block, etc. Extract.

The decoder 410 determines the size of the CTU by extracting information on the CTU size from a sequence parameter set (SPS) or a picture parameter set (PPS), and divides the picture into CTUs of the determined size. Then, the CTU is determined as the highest layer of the tree structure, that is, the root node, and the CTU is divided using the tree structure by extracting the partition information for the CTU.

For example, in the case of splitting the CTU using the QTBTTT structure, a first flag (QT_split_flag) related to the splitting of the QT is first extracted, and each node is split into four nodes of a lower layer. And, for the node corresponding to the leaf node of QT, the second flag (MTT_split_flag) related to the splitting of the MTT and the splitting direction (vertical / horizontal) and/or split type (binary / ternary) information are extracted and the corresponding leaf node is MTT. Divide into structure. Through this, each node below the leaf node of the QT is recursively divided into a BT or TT structure.

As another example, when a CTU is split using a QTBTTT structure, first, a CU split flag indicating whether to split a CU (split_cu_flag) is extracted, and when the corresponding block is split, a QT split flag (split_qt_flag) is extracted. . If the split type is not QT but MTT, a flag indicating the MTT split direction (vertical or horizontal) (mtt_split_cu_vertical_flag) and/or a flag indicating the MTT split type (Binary or Ternary) (mtt_split_cu_binary_flag) is additionally extracted. In the partitioning process, each node may have zero or more repetitive MTT partitions after zero or more repetitive QT partitions. For example, in the CTU, MTT division may occur immediately, or, conversely, only multiple QT divisions may occur.

As another example, when the CTU is divided using the QTBT structure, each node is divided into four nodes of a lower layer by extracting the first flag (QT_split_flag) related to the division of the QT. In addition, a split flag indicating whether or not the node corresponding to the leaf node of the QT is further split into BT and split direction information are extracted.

On the other hand, when determining the current block to be decoded through division of the tree structure, the decoder 410 extracts information on a prediction type indicating whether the current block is intra predicted or inter predicted. When the prediction type information indicates intra prediction, the decoder 410 extracts a syntax element for intra prediction information (intra prediction mode) of the current block. When the prediction type information indicates inter prediction, the decoder 410 extracts a syntax element for the inter prediction information, that is, information indicating a motion vector and a reference picture referenced by the motion vector.

Meanwhile, the decoding unit 410 extracts information on quantized transform coefficients of the current block as information on the residual signal.

The inverse quantization unit 420 inverse quantizes the quantized transform coefficients, and the inverse transform unit 430 inversely transforms the inverse quantized transform coefficients from the frequency domain to the spatial domain to restore residual signals to generate a residual block for the current block. .

In addition, when the inverse transform unit 430 inverse transforms only a partial region (subblock) of the transform block, a flag indicating that only the subblock of the transform block has been transformed (cu_sbt_flag), and the direction (vertical/horizontal) information of the subblock (cu_sbt_horizontal_flag) ) And/or sub-block location information (cu_sbt_pos_flag), and inverse transform coefficients of the sub-block from the frequency domain to the spatial domain to restore residual signals. By filling in, the final residual block for the current block is created.

The prediction unit 440 may include an intra prediction unit 442 and an inter prediction unit 444. The intra prediction unit 442 is activated when the prediction type of the current block is intra prediction, and the inter prediction unit 444 is activated when the prediction type of the current block is inter prediction.

The intra prediction unit 442 determines an intra prediction mode of the current block among a plurality of intra prediction modes from a syntax element for the intra prediction mode extracted from the decoder 410, and a reference pixel around the current block according to the intra prediction mode. The current block is predicted by using them.

The inter prediction unit 444 determines a motion vector of the current block and a reference picture referenced by the motion vector using the syntax element for the intra prediction mode extracted from the decoding unit 410, and uses the motion vector and the reference picture. To predict the current block.

The adder 450 restores the current block by adding the residual block output from the inverse transform unit and the prediction block output from the inter prediction unit or the intra prediction unit. The pixels in the reconstructed current block are used as reference pixels for intra prediction of a block to be decoded later.

The filter unit 460 may include a deblocking filter 462 and an SAO filter 464. The deblocking filter 462 performs deblocking filtering on the boundary between reconstructed blocks in order to remove blocking artifacts occurring due to block-by-block decoding. The SAO filter 464 performs additional filtering on the reconstructed block after deblocking filtering in order to compensate for the difference between the reconstructed pixel and the original pixel caused by lossy coding. The reconstructed block filtered through the deblocking filter 462 and the SAO filter 464 is stored in the memory 470. When all blocks in one picture are reconstructed, the reconstructed picture is used as a reference picture for inter prediction of a block in a picture to be encoded later.

The method of padding a reference sample can be applied to various processes for encoding and decoding an image. For example, the reference sample padding method may be applied to a process of configuring reference samples for intra prediction of a current block, and may also be applied to a process of configuring reference samples for inter prediction of a current block.

In intra prediction, prediction is performed using samples that have already been reconstructed around a current block, and a surrounding sample used for intra prediction is called a reference sample.

A PU (Prediction Unit) used for intra prediction of HEVC (High Efficiency Video Coding) is composed of a square having a size of NxN as shown in FIG. 5A. Reference samples located on the left side of the PU (left reference samples), reference samples located above the PU (top reference samples), and a reference sample located on the upper left side of the PU (top left reference sample) are currently These are reference samples required for intra prediction of a block.

In Fig. 5A, a dotted block represents reference samples, and R0,1, R0,2, ... , R0,2N denote left reference samples, R0,0 denote upper left reference samples, R1,0, R2,0, ... , R2N,0 denotes upper reference samples. When the reference samples are configured in this way, the values of prediction samples or prediction pixels are set in the lower left region of the PU using the left reference samples, and the upper right region of the PU is predicted using the upper reference samples. The values of the samples are set. Here, the upper left reference sample may or may not be used as a reference sample for both regions (lower left region and upper right region).

In HEVC, when the reference samples are not available (or do not exist), intra prediction is performed by generating or configuring a non-usable reference sample through padding. When the reference samples are not available, a case in which some of the reference samples are not available (a case in which some of the reference samples are available) and a case in which all of the reference samples are not available may be included.

When not all of the reference samples are available (when the boundary of the current block is the upper leftmost boundary of the picture), the reference samples are filled with 1<<(BitDepth _{Y ), which is the intermediate value of the expressible bit depth (BitDepth Y ).} When some of the reference samples are not usable, the unusable reference sample is padded by using the value of the nearest reference sample (usable) from the position of the unusable reference sample.

An example of a method of padding the values of some unusable reference samples is shown in FIG. 5B. As shown in (a) of FIG. 5B, when unusable reference samples (blocks not indicated by shaded lines) are located between A and B, which are usable reference samples (blocks indicated by shaded lines), the value of A that is closest to Unusable reference samples may be constructed by padding in the direction of the arrow. As shown in (b) of FIG. 5B, by padding the value of A in the direction of the arrow from the lower left block, unusable reference samples located at the left and lower left of A can be constructed, and the value of B is moved to the right of B. By padding, an unusable reference sample located on the right side of B may be constructed.

Meanwhile, even when the current block is predicted using a bi-directional optical flow (BDOF) corresponding to an inter prediction mode, reference sample padding may be performed. A diagram for explaining a reference sample padding method for BDOF is shown in FIG. 6.

In order to predict the current block with BDOF, a gradient value between reference samples (or prediction samples) in a reference picture for prediction of the current block is required. In FIG. 6, samples indicated by solid lines represent reference samples in a reference picture for prediction of a current block, and a block composed of the reference samples represents a reference block (or prediction block) corresponding to the current block. Here, the boundary of the reference block may be treated like the boundary of the current block in the current picture.

In order to calculate a gradient value between reference samples, a value of a reference sample located outside the boundary of the reference block (located outside the boundary of the current block) may be required. In FIG. 6, samples indicated by dotted lines represent reference samples (border-external reference samples) positioned outside the boundary of a reference block, and an area composed of border-external reference samples may be treated as an extended area.

Boundary external reference samples that are not usable among the boundary external reference samples are obtained through a method (copying method) of obtaining a sample value from an integer location (reference samples located inside the boundary or available reference samples). Can be padded. In FIG. 6, an arrow indicates a direction (padding direction) of copying a sample value from an integer position around the boundary external reference samples for padding.

As described above, in the conventional intra prediction method, padding is performed by simply filling the value of the nearest available reference sample into the unusable reference sample, and the BDOF simply refers to the surrounding sample value outside the boundary. Since padding is performed by filling the sample, the accuracy of the configuration of the reference sample may be degraded.

In the present specification, various methods that can solve the problems of the conventional method are proposed by more accurately constructing a reference sample by using a correlation between the available reference samples.

As shown in FIG. 7, an image decoding apparatus capable of implementing a reference sample construction method may include a decoding unit 410, a determination unit 710, and a padding unit 720. The determination unit 710 and the padding unit 720 may constitute a prediction unit 440. 7 shows only an example in which the determination unit 710 and the padding unit 720 are included in the prediction unit 440 of the image decoding apparatus, but the determination unit 710 and the padding unit 720 are the prediction units of the image encoding apparatus. It may also be included in (120). That is, the following processes performed by the determination unit 710 and the padding unit 720 may be performed by both the prediction unit 440 of the image decoding apparatus and the prediction unit 120 of the image encoding apparatus.

The determination unit 710 may determine whether or not the current reference sample is available (S810).

The current reference sample may correspond to any one of reference samples used for intra prediction of the current block, or may correspond to a boundary external reference sample of the BDOF. In addition, the current reference sample may correspond to a reference sample that is an object of determination (to be a configuration object) of availability among reference samples used for intra prediction or BDOF of the current block. Accordingly, the current reference sample is the left reference samples R0,1, R0,2, …, R0,2N, the upper left reference sample R0,0, and the upper reference samples R1,0, R2 of FIG. 5A. ,0, ..., R2N, 0), and may correspond to any one of the border external reference samples of FIG. 6.

When the current reference sample is not available, the determination unit 710 may determine whether other reference samples are available (S820).

Other reference samples are reference samples for intra prediction of the current block or gradient calculation of the BDOF, and may be located around the current reference sample. For example, other reference samples may be located in the same row or column as the current reference sample. Other reference samples may be continuously located from the current reference sample or may be located non-contiguous. Other reference samples may or may not be located adjacent to each other.

When the currently unavailable reference sample is included in the left reference samples, other reference samples may be reference samples included in the left reference samples. When the currently unavailable reference sample is included in the upper reference samples, other reference samples may be reference samples included in the upper reference samples. The number of other reference samples to be determined whether or not the availability is available may be a plurality.

When other reference samples are available, the padding unit 720 may pad the current reference sample based on a correlation between the other reference samples (S830).

Adjacent pixels (samples) may have a certain amount of change, and the correlation may mean a certain amount of change between usable reference samples (other reference samples). The correlation may include a linear relationship between usable reference samples and a gradient between usable reference samples.

If other reference samples are not available, the current reference sample may be padded with the value of the nearest available reference sample.

An example of a method of configuring reference samples is shown in FIG. 9.

In FIG. 9, among upper reference samples R1,0, R2,0, ..., R2N,0, R1,0 to RN,0 are available reference samples, and RN+1,0 to R2N,0 Is assumed to be unavailable (Unavailable reference samples). In addition, it is assumed that among the left reference samples R0,1, R0,2, …, R0,2N, R0,1 to R0,N can be used, and R0,N+1 to R0,2N cannot be used.

In the case of configuring the upper reference samples, since the current reference sample (RN+1,0) is not available, it is determined whether or not other reference samples (a plurality of R1,0 to RN,0) can be used. Since other reference samples can be used, the current reference sample RN+1,0 may be padded based on the correlation between the other reference samples. If such a process is repeatedly performed, unusable upper reference samples RN+1,0 to R2N,0 may be generated or configured.

When configuring the left reference samples, since the current reference samples R0 and N+1 are not available, it is determined whether or not other reference samples (a plurality of R0,1 to R0,N) can be used. Since other reference samples can be used, the current reference samples R0 and N+1 may be padded based on the correlation between the other reference samples. When such a process is repeatedly performed, unusable left reference samples R0,N+1 to R0,2N may be generated or configured.

Embodiments proposed in the present invention are largely 1) a padding method based on a linear relationship, 2) a padding method based on a gradient, 3) a method of constructing syntax for transmitting information on selection of a padding method, and 4) reference samples. It can be divided into padding methods for cases where all of them are unavailable. Hereinafter, embodiments for each method will be described in detail.

Example 1

Embodiment 1 is a method of constructing a current reference sample based on a correlation between reference samples. Embodiment 1 may be classified into the following sub-embodiments according to the type of parameter used as a correlation.

Example 1-1

In Example 1-1, a linear relationship between different reference samples is used as a correlation. The number of other reference samples used in Example 1-1 may be at least three.

The determination unit 710 may check the current reference sample corresponding to the configuration target (S1010) and determine whether the current reference sample is available (S1020).

If the current reference sample is available, after moving to the next reference sample (S1030), the next reference sample is used as a new current reference sample to perform a configuration process. If the current reference sample is not available, a padding process based on a linear relationship or a padding process using the nearest reference sample value is performed according to availability of other reference samples.

Specifically, when the current reference sample is not available, the determination unit 710 may determine whether other reference samples are available (S1040). The padding unit 720 pads the current reference sample with the value of the nearest reference sample that can be used when other reference samples are not available (copying the value of the available nearest reference sample to the current reference sample) to configure the current reference sample. It can be done (S1060). In contrast, when other reference samples are available, the padding unit 720 may pad the current reference sample using a value (linear relationship value) derived from a linear relationship between other reference samples (S1050).

The linear relationship between different reference samples can be expressed by a linear equation such as Equation 1.

In Equation 1, each of X and Y represents a reference sample, and each of a and b represents a slope and an intercept.

In the case of upper reference samples, the result of applying the _{N-th other reference sample (R N,0} ) and the N-1-th other reference sample (R _{N-1,0) to Equation 1 among other reference samples is shown in Equation 2} It is the same, and the result of applying the N-1th other reference sample and the _N-2th other reference sample (R N-2,0) to Equation 1 is as Equation 3.

If b is derived using Equation 3, it is the same as Equation 4, and if Equation 4 is substituted into Equation 1, it is the same as Equation 5, and if a is derived through Equation 5, it is the same as Equation 6, and Substituting a in Equation 6 into Equation 2 to induce b again is equivalent to Equation 7.

_{Since the slope and intercept of the linear equation (linear relationship) are derived, the current reference sample (R N+1,0} ) is the value (Y) derived by substituting the Nth other reference sample into the linear equation (substituting it for X). It can be padded.

If such processes are repeatedly performed, all currently unavailable reference samples (R _N+1,0 to R _2N,0 or R _0,N+1 to R _0,2N ) may be generated or configured.

In the above, based on the method of padding the current reference sample using _{other reference samples (R N,0} to R _N-2,0 or R _0,N to R _0,N-2 ) that are located in succession with each other. Although Example 1-1 has been described, other reference samples may be positioned not to be continuous with each other.

Example 1-2

In Example 1-2, gradients between different reference samples are used as correlations. The number of other reference samples used in Example 1-2 may be at least two.

The determination unit 710 may check the current reference sample corresponding to the configuration target (S1110) and determine whether the current reference sample is available (S1120).

If the current reference sample is available, after moving to the next reference sample (S1130), the next reference sample is used as a new current reference sample to perform a configuration process. If the current reference sample is not available, a padding process based on a gradient or a padding process using the nearest reference sample value is performed depending on whether other reference samples are available.

Specifically, when the current reference sample is not available, the determination unit 710 may determine whether other reference samples are available (S1140). The padding unit 720 pads the current reference sample with the value of the nearest reference sample that can be used when other reference samples are not available (copying the value of the available nearest reference sample to the current reference sample) to configure the current reference sample. It can be done (S1160). In contrast, when other reference samples are available, the padding unit 720 is a value obtained by summing the value of the average of the gradient values between the other reference samples and the values of other reference samples adjacent to the current reference sample (gradient relationship value). The current reference sample may be padded by using (S1150).

The S1150 process may be performed by Equation 8.

은 다른 참조샘플들 간의 그래디언트 값을 평균한 값을 나타낸다. 또한, W는 현재 참조샘플의 구성을 위해 이용되는 다른 참조샘플들의 수를 나타낸다. In Equation 8, R _N+1,0 represents a current reference sample, R _N,0 represents another reference sample neighboring the current reference sample,

Represents the average of the gradient values between different reference samples. In addition, W represents the number of other reference samples used to construct the current reference sample.

If such processes are repeatedly performed, all currently unavailable reference samples (R _N+1,0 to R _2N,0 or R _0,N+1 to R _0,2N ) may be generated or configured.

In the above, based on the method of padding the current reference sample using _{other reference samples (R N,0} and R _N-1,0 or R _0,N and R _0,N-1 ) located in succession with each other. Although Example 1-2 has been described, other reference samples may be positioned not to be continuous with each other.

Example 1-3

Example 1-3 is a method in which Example 1-1 and Example 1-2 are combined.

As described above, a method of padding a current reference sample based on a linear relationship (Example 1-1) uses at least three different reference samples, and a method of padding the current reference sample based on a gradient (Example 1-2) can use at least two different reference samples. Embodiment 1-3 may be configured to distinguish between Embodiment 1-1 and Embodiment 1-2 based on such differences.

The determination unit 710 may check the current reference sample (S1210) and determine whether the current reference sample is available (S1220). If the current reference sample is available, the next reference sample is moved to the next reference sample (S1230), and then the next reference sample is used as a new current reference sample to perform a configuration process.

When the current reference sample is not available, the determination unit 710 may determine whether at least three other reference samples are available (whether the number of available other reference samples is at least three) (S1240). When the number of available other reference samples is at least three, the padding unit 720 may pad the current reference sample with a value (linear relationship value) derived from a linear relationship between the other reference samples (S1250). In contrast, when the number of available other reference samples is less than three, it may be determined whether to apply a padding method based on a gradient as follows.

The determination unit 710 may determine whether the number of available other reference samples is two (S1260). When the number of available other reference samples is two, the padding unit 720 calculates the current reference sample as a value obtained by summing the value of the average of the gradient values between the other reference samples and the values of other reference samples adjacent to the current reference sample. It can be padded (S1270).

In contrast, when the number of available other reference samples is less than two, the current reference sample may be configured by padding the current reference sample with the value of the nearest available reference sample (S1280).

Example 1-4

Example 1-4 compares the padding method based on a linear relationship, a padding method based on a gradient, and the padding method based on the nearest reference sample value, and selects one of the three methods (the optimal method). This is an example.

When the current reference sample is not available, the video encoding/decoding apparatus performs padding based on a linear relationship, padding based on a gradient, and padding based on the value of the nearest reference sample, respectively, and compares values derived from each process. Thus, the current reference sample can be padded with any one of these values.

The comparison standard between the values derived based on the linear relationship, the values derived based on the gradient, and the values derived based on the value of the nearest reference sample is the SAD between the predicted pixel obtained using each value and the original pixel. to be. Specifically, the current reference sample may be padded based on a value having the smallest SAD among values derived from each of the processes.

Meanwhile, in the embodiments described above, whether to be enabled may be determined by an enable flag.

When only the padding method based on the value of the nearest reference sample is allowed, the video encoding apparatus may signal by setting the value of the enable flag to “0”. In addition, when a padding method based on a correlation is also allowed, the video encoding apparatus may signal by setting the value of the enable flag to “1”.

The enable flag may be defined and signaled at one of a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, and a coding tree unit (CTU). Table 1 shows an example of the enable flag (multi_padding_enabled_flag) defined and signaled in the Slice header.

The video decoding apparatus may decode multi_padding_enabled_flag from the bitstream and determine whether to enable the padding method according to the indication of the multi_padding_enabled_flag.

For example, when multi_padding_enabled_flag=0, only the padding method based on the value of the nearest reference sample is enabled, and when multi_padding_enabled_flag=1, the padding method based on the value of the nearest reference sample, correlation (linear relationship and gradient )-Based padding methods can all be enabled.

Example 2

Embodiment 2 is a method of constructing syntaxes for signaling information on selection of a padding method.

The image encoding apparatus may signal to the image decoding apparatus by setting information on the padding method used to configure the current reference sample as a value of a syntax (padding mode syntax).

Table 2 shows an example of the padding mode syntax (padding_mode_flag).

The padding_mode_flag may be divided into a first flag (first) and a second flag (second). The first flag can indicate whether to apply the copy the nearest available reference sample (or whether to apply the padding method based on correlation), and the second flag is a linear relationship. Whether to apply any one of a padding method based on the linear equation and a padding method based on gradients may be indicated.

The first flag = 0 indicates that the padding method based on the value of the nearest reference sample is applied, and the first flag = 1 indicates that the padding method based on the value of the nearest reference sample is not applied. Applicable). The second flag = 0 may indicate that the padding method based on the linear relationship is applied, and the second flag = 1 may indicate that the padding method based on the gradient is applied.

When the padding method based on the value of the nearest reference sample is applied, a primary flag set to a value of “0” may be signaled. When a padding method based on a linear relationship is applied, a primary flag set to a value of “1” and a secondary flag set to a value of “0” may be signaled. When a padding method based on a gradient is applied, a primary flag set to a value of “1” and a secondary flag set to a value of “1” may be signaled.

The image decoding apparatus (decoder) may determine whether to apply the padding method based on the correlation by decoding the primary flag from the bitstream (S1310) (S1320). When the first flag = 0, a padding method based on the value of the nearest reference sample may be performed (S1330).

When the first flag = 1, the video decoding apparatus may decode the second flag from the bitstream (S1340) and determine whether to apply any one of a padding method based on a linear relationship and a padding method based on a gradient (S1350). . When the second flag = 0, a padding method based on a linear relationship may be performed (S1370), and if the second flag = 1, a padding method based on a gradient may be performed (S1350).

Example 3

Embodiment 3 is a method of padding a current reference sample when there are no available reference samples.

In HEVC, reference samples are filled with _{1<<(BitDepth Y} ), which is an intermediate value of a bit depth that can be expressed when there are no reference samples. Such a conventional method may generate inaccurate results according to the brightness value of a corresponding image or block. In the present invention, encoding/decoding performance is performed by selectively signaling any one of 1/2 brightness value (first padding value), 1/4 brightness value (second padding value), and 3/4 brightness value (third padding value). Suggest a way to improve.

When all the reference samples do not exist (if not available), the video encoding apparatus may signal to the video decoding apparatus using any one of a first padding value, a second padding value, and a third padding value as a padding value. . The image decoding apparatus may decode the padding value and pad the current reference sample based on the decoded padding value.

A value included in the signaled padding value among the first padding value, the second padding value, and the third padding value may be indicated by the padding information. The padding information may be defined and signaled at one or more positions of a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, and a coding tree unit (CTU). Table 3 shows an example of padding information (padding_value_flag).

The padding_value_flag may be divided into a third flag and a fourth flag. The third flag indicates whether the first padding value is included, and the fourth flag may indicate a value included in the padding value among the second padding value and the third padding value.

The third flag = 0 indicates that the first padding value is included, and the third flag = 1 indicates that the first padding value is not included (any one of the second padding value and the third padding value is included). have. The fourth flag = 0 may indicate that the second padding value is included, and the fourth flag = 1 may indicate that the third padding value is included.

When the first padding value is included, a third flag set to a value of “0” may be signaled. When the second padding value is included, a third flag set to a value of “1” and a fourth flag set to a value of “0” may be signaled. When the third padding value is included, a third flag set to a value of "1" and a fourth flag set to a value of "1" may be signaled.

The image decoding apparatus may determine whether the first padding value is included by decoding the third flag from the bitstream (S1410) (S1420). When the third flag = 0, the first padding value may be decoded (S1430).

When the third flag = 1, the image decoding apparatus may decode the fourth flag from the bitstream (S1440) to determine whether any one of the second padding value and the third padding value is included (S1450). If the fourth flag = 0, the second padding value may be decoded (S1470), and if the fourth flag = 1, the third padding value may be decoded (S1460).

The above description is merely illustrative of the technical idea of the present embodiment, and those of ordinary skill in the technical field to which the present embodiment pertains will be able to make various modifications and variations without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are not intended to limit the technical idea of the present embodiment, but to explain the technical idea, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The scope of protection of this embodiment should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present embodiment.

120, 440: prediction unit 130: subtractor
170, 450: adder 180, 460: filter unit

Claims (14)

As a method of constructing reference samples used for prediction of a current block,
Determining whether or not a current reference sample to be configured is available;
If the current reference sample is not available, determining whether other reference samples located around the current reference sample are available; And
And if the other reference samples are available, padding the current reference sample based on a correlation between the other reference samples. The method of claim 1,
The correlation is,
Is a linear relationship between the other reference samples,
The padding step,
The reference sample construction method of padding the current reference sample by using a linear relationship value derived from a linear relationship between the other reference samples. The method of claim 2,
When the first flag indicating whether padding based on the correlation is applied or not is decoded from the bitstream, and when the first flag indicates that padding based on the correlation is applied, whether or not padding based on the linear relationship is applied is determined. Further comprising the step of decoding the indicated second flag from the bitstream,
The padding step,
When the second flag indicates that the padding based on the linear relationship is applied, the current reference sample is padded by using the linear relationship value. The method of claim 1,
The correlation is,
It is a gradient between the different reference samples,
The padding step,
Padding the current reference sample by using a gradient relationship value obtained by summing a value obtained by averaging gradient values between the other reference samples and a value of another reference sample neighboring the current reference sample among the other reference samples, How to construct a reference sample. The method of claim 4,
When the first flag indicating whether padding based on the correlation is applied or not is decoded from the bitstream, and when the first flag indicates that the padding based on the correlation is applied, indicating whether to apply the padding based on the gradient Further comprising the step of decoding the second flag to be performed from the bitstream,
The padding step,
When the second flag indicates that padding based on the gradient is applied, the current reference sample is padded by using the gradient relationship value. The method of claim 1,
If reference samples used for prediction of the current block are not available, decoding a padding value from the bitstream,
The padding step,
When the other reference samples are not available, the current reference sample is padded based on the decoded padding value. The method of claim 6,
The padding value is
Including any one of a first padding value that is 1/2 of the representable bit depth, a second padding value that is 1/4 of the representable bit depth, and a third padding value that is 3/4 of the representable bit depth, and ,
The decoding step,
Decoding a third flag indicating whether the first padding value is included;
When the third flag indicates that the first padding value is not included, decoding a fourth flag indicating whether any one of the second padding value and the third padding value is included; And
And decoding any one of the second padding value and the third padding value as indicated by the fourth flag. As an image decoding apparatus configuring reference samples used for prediction of a current block,
A determination unit that determines whether or not a current reference sample to be configured is available, and when the current reference sample is not available, determines whether or not other reference samples located in the vicinity of the current reference sample are available; And
And a padding unit for padding the current reference sample based on a correlation between the other reference samples when the other reference samples are available. The method of claim 8,
The correlation is,
Is a linear relationship between the other reference samples,
The padding part,
An image decoding apparatus for padding the current reference sample by using a linear relationship value derived from a linear relationship between the other reference samples. The method of claim 9,
When the first flag indicating whether padding based on the correlation is applied or not is decoded from the bitstream, and when the first flag indicates that padding based on the correlation is applied, whether or not padding based on the linear relationship is applied is determined. Further comprising a decoding unit for decoding the indicated second flag from the bitstream,
The padding part,
When the second flag indicates that padding based on the linear relationship is applied, the current reference sample is padded by using the linear relationship value. The method of claim 8,
The correlation is,
It is a gradient between the different reference samples,
The padding part,
Padding the current reference sample by using a gradient relationship value obtained by summing a value obtained by averaging gradient values between the other reference samples and a value of another reference sample neighboring the current reference sample among the other reference samples, Video decoding device. The method of claim 11,
When the first flag indicating whether padding based on the correlation is applied or not is decoded from the bitstream, and when the first flag indicates that the padding based on the correlation is applied, indicating whether to apply the padding based on the gradient Further comprising a decoding unit for decoding the second flag to be decoded from the bitstream,
The padding part,
When the second flag indicates that padding based on the gradient is applied, the current reference sample is padded by using the gradient relationship value. The method of claim 8,
If the reference samples used for prediction of the current block are not available, further comprising a decoding unit for decoding a padding value from the bitstream,
The padding part,
When the other reference samples are not available, the current reference sample is padded based on the decoded padding value. The method of claim 13,
The padding value is
Including any one of a first padding value that is 1/2 of the representable bit depth, a second padding value that is 1/4 of the representable bit depth, and a third padding value that is 3/4 of the representable bit depth, and ,
The decryption unit,
Decodes a third flag indicating whether the first padding value is included, and when the third flag indicates that the first padding value is not included, any one of the second padding value and the third padding value A video decoding apparatus for decoding a fourth flag indicating whether or not one is included, and decoding any one of the second padding value and the third padding value according to the indication of the fourth flag.

Images