KR20210134555A - Apparatus and method for intra-prediction based video encoding or decoding - Google Patents

Abstract

The present invention provides a video decoding method based on intra prediction. The video decoding method comprises the steps of: decoding transform coefficients for a current block to be decoded from a bit-stream; constructing input data using a reference region decoded before the current block; generating prediction pixels of the current block by applying a 2-dimensional or 3-dimensional filter coefficient set to the input data; generating residual signals for the current block by inversely transforming the transform coefficients; and reconstructing the current block using the prediction pixels and the residual signals.

Description

Intra prediction-based video encoding or decoding apparatus and method

The present invention relates to an apparatus for encoding or decoding an image, and more particularly, to an apparatus and method for encoding or decoding an image based on intra prediction.

The content described in this section merely provides background information for the present embodiment and does not constitute the prior art.

Since moving image data has a larger amount of data than audio data or still image data, hardware resources such as memory are consumed a lot when the data source is stored or transmitted as it is. Therefore, in general, moving picture data is compressed using an encoder and then stored or transmitted, and the compressed moving picture data is decompressed using a decoder and then reproduced.

Meanwhile, as the demand for video content such as high-capacity games and 360-degree images is rapidly increasing today, the size, resolution, and frame rate of images are increasing. Accordingly, there is a problem in that the amount of data to be decoded increases and the complexity of the decoder also increases. In order to solve this problem, in the next-generation video codec, a technology capable of efficiently extracting data from a compressed bitstream without degrading coding efficiency is required.

According to recent experimental results, the Bjonteggrad-delta bit rate (BDBR) of about 3.57% is achieved by replacing the in-loop filter of the existing video encoding or decoding device with a Convolutional Neural Network (CNN) filter, which is a kind of artificial neural network. It has been shown that benefits can be achieved. Accordingly, image encoding/decoding technology using artificial neural network technology is attracting attention as a solution to the above-mentioned problem.

An object of the present embodiment is to provide an apparatus and method for encoding or decoding an image capable of improving prediction accuracy while maintaining complexity of a decoder.

According to an aspect of the present embodiment, there is provided a video decoding method based on intra prediction, comprising: decoding transform coefficients for a current block to be decoded from a bitstream; constructing input data using a reference region decoded before the current block; generating prediction pixels of the current block by applying a 2-dimensional or 3-dimensional filter coefficient set to the input data; generating residual signals for the current block by inverse transforming the transform coefficients; and reconstructing the current block using the prediction pixels and the residual signals.

According to another aspect of the present embodiment, there is provided an intra prediction-based video encoding method, comprising: constructing input data from a reference region decoded before a current block to be encoded; generating a prediction block of the current block by applying a 2-dimensional or 3-dimensional filter coefficient set to the input data; generating a residual block by subtracting the prediction block from the current block; and encoding the residual block to generate encoded data.

The image encoding or decoding apparatus according to the present embodiment can improve image encoding or decoding efficiency by using intra prediction to which a set of filter coefficients configured in two or three dimensions is applied.

1 is an exemplary diagram of a structure of a CNN that can be applied to the techniques of the present disclosure.
2 is an exemplary diagram of an image encoding apparatus capable of implementing the techniques of the present disclosure.
3 is an exemplary diagram for a plurality of intra prediction modes.
4 is an exemplary diagram of an image decoding apparatus capable of implementing the techniques of the present disclosure.
5 is a block diagram showing the configuration of a CNN prediction unit on the side of the video encoding apparatus according to the present embodiment.
6 is an exemplary diagram of a surrounding area that can be used as input data of CNN.
7 is a diagram illustrating an example of configuring an input layer of a CNN from a plurality of neighboring blocks.
8 is an exemplary diagram of a prediction direction of a current block based on pixel value shapes of neighboring blocks.
9 is an exemplary diagram of a layer configuration of a CNN including hint information.
10 is a block diagram illustrating the configuration of a CNN prediction unit on the side of the image decoding apparatus according to the present embodiment.
11 is a flowchart illustrating the operation of the CNN prediction unit of the image encoding apparatus of FIG. 5 .
12 is a flowchart illustrating an operation of the CNN prediction unit of the image decoding apparatus of FIG. 10 .

Hereinafter, some embodiments of the present invention will be described in detail with reference to exemplary drawings. It should be noted that in adding identification codes to the components of each drawing, the same components are to have the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted.

In addition, in describing the components of the present invention, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the components from other components, and the essence, order, or order of the components are not limited by the terms. Throughout the specification, when a part 'includes' or 'includes' a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated. . In addition, the '... Terms such as 'unit' and 'module' mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software.

Hereinafter, embodiments of the present invention will be described in more detail with reference to the accompanying drawings.

1 is an exemplary diagram of a structure of a CNN that can be applied to the techniques of the present disclosure.

CNN refers to a multi-layer neural network with a special connection structure designed for image processing. The CNN algorithm can be divided into a learning process and an inference process, and the learning process can be further divided into supervised learning, unsupervised learning, and reinforcement learning according to a learning method. Among them, supervised learning refers to a process of calculating coefficient values of a convolution kernel using an output label that is an explicit correct answer for input data. In addition, the coefficient values of the convolution kernel may be updated through an iterative learning process using an error backpropagation algorithm in order to minimize an error between the output data and the output label.

Thereafter, an inference process of generating output data from input data using the coefficient values of the convolution kernel calculated through the learning process is performed. For example, a U/V image may be generated from the Y image through an inference process using coefficient values of the convolution kernel.

Referring to FIG. 1 , a CNN may include an input layer 110 , a hidden layer 130 , and an output layer 150 . The hidden layer 130 is positioned between the input layer 110 and the output layer 150 and may include a plurality of convolutional layers 131 to 139 . In addition, the hidden layer 130 may further include an upsampling layer or a pooling layer in order to adjust the resolution of a feature map that is a result of a convolution operation.

All layers constituting the CNN each include a plurality of nodes, and each node is interconnected with nodes of other adjacent layers to transmit an output value to which a predetermined connection weight is applied as an input of other nodes.

The convolution layers 131 to 139 may generate a feature map by performing a convolution operation on image data input to each layer using a convolution kernel (ie, a filter) in the form of a 2D or 3D matrix. . Here, the feature map refers to image data in which various features of image data input to each layer are expressed. The number of convolutional layers 131 to 139, the size of the convolution kernel, etc. may be preset prior to the learning process.

The output layer 150 may be configured as a fully connected layer. Nodes of the output layer 150 may output image data by combining various features expressed in the feature map.

2 is an exemplary diagram of an image encoding apparatus capable of implementing the techniques of the present disclosure.

The image encoding apparatus includes a block divider 210 , a predictor 220 , a subtractor 230 , a transform unit 240 , a quantizer 245 , an encoder 250 , an inverse quantizer 260 , and an inverse transform unit ( 265 ), an adder 270 , a filter unit 280 , and a memory 290 . Each component of the image encoding apparatus may be implemented as hardware or software, or may be implemented as a combination of hardware and software. In addition, the plurality of “units” and “units” may be integrated into at least one module or chip and implemented as at least one processor, except when each needs to be implemented as individual specific hardware.

The block divider 210 divides each picture constituting an image into a plurality of coding tree units (CTUs) and then recursively divides the CTUs using a tree structure. In this case, in the tree structure, a leaf node becomes a coding unit (CU), which is a basic unit of encoding.

The tree structure is a QuadTree (QT) structure in which the parent node (or parent node) is divided into four child nodes (or child nodes) of the same size, and a binary tree (QT) structure in which the parent node is divided into two child nodes ( It may be a BinaryTree, BT) structure or a TernaryTree (TT) structure in which an upper node is divided into three lower nodes in a ratio of 1:2:1. In addition, the tree structure may be a structure in which at least one or more of these QT structures, BT structures, and TT structures are mixed, for example, a QuadTree plus BinaryTree (QTBT) structure or a QuadTree plus BinaryTree TernaryTree (QTBTTT) structure.

Hereinafter, a block corresponding to a CU to be encoded or decoded (ie, a leaf node of a tree structure used for CTU splitting) will be referred to as a 'current block'. A pixel constituting the current block may include a luma component and two chroma components. A luma block composed of a luma component and a chroma block composed of a chroma component may be individually predicted and encoded.

Hereinafter, each component will be classified using the term "channel". That is, the same channel means the same component, and different channels mean different components.

Information applied to each CU is encoded as a syntax of the CU, and information commonly applied to CUs included in one CTU is encoded as a syntax of the CTU. In addition, information commonly applied to all blocks in one slice is encoded as a syntax of the slice, and information applied to all blocks constituting one picture is encoded in a picture parameter set (PPS). . Furthermore, information commonly referenced by a plurality of pictures is encoded in a sequence parameter set (SPS). In addition, information commonly referenced by one or more SPSs is encoded in a video parameter set (VPS).

The prediction unit 220 generates a prediction block by predicting the current block. The prediction unit 220 includes an intra prediction unit 222 and an inter prediction unit 224 .

In general, each of the current blocks in a picture may be predictively coded. Prediction of the current block is generally performed using an intra prediction technique (using data from the picture containing the current block) or inter prediction technique (using data from a picture coded before the picture containing the current block). can

The intra prediction unit 222 predicts pixels in the current block by using pixels (reference pixels) located around the current block in the current picture including the current block. A plurality of intra prediction modes exist according to a prediction direction. For example, as shown in FIG. 3 , the plurality of intra prediction modes may include a non-directional mode including a planar mode and a DC mode and 65 directional modes. According to each prediction mode, neighboring pixels used for intra prediction and an arithmetic expression are defined differently.

Also, the intra prediction unit 222 may predict pixels in the current block using reference pixels through a CNN-based learning and inference process. In this case, the intra prediction unit 222 may parallelly operate a CNN-based intra prediction mode (hereinafter referred to as a 'CNN mode') together with the plurality of intra prediction modes described above with reference to FIG. 3 . Alternatively, the intra prediction unit 222 may independently operate only the CNN mode.

The intra prediction unit 222 may determine an intra prediction mode to be used for encoding the current block. In some examples, the intra prediction unit 222 may encode a current block using several intra prediction modes and select an appropriate intra prediction mode to use from the tested modes. For example, the intra prediction unit 222 calculates rate-distortion values using rate-distortion analysis for several tested intra prediction modes, and has the best rate-distortion characteristics among the tested modes. An intra prediction mode can be selected.

The intra prediction unit 222 predicts the current block using a neighboring pixel (or input data) determined according to the selected intra prediction mode and an arithmetic expression (or a coefficient value of a convolution kernel).

Intra prediction mode information indicating which of the plurality of intra prediction modes is used as the intra prediction mode of the current block is encoded by the encoder 250 and signaled to the image decoding apparatus.

Meanwhile, the intra prediction unit 222 is configured to efficiently encode intra prediction mode information indicating which of the plurality of intra prediction modes is used as the intra prediction mode of the current block, the current block among the plurality of intra prediction modes. As the intra prediction mode of , some modes with high probability may be determined as the most probable mode (MPM).

The MPM list may include intra prediction modes, planar mode, and DC mode of neighboring blocks of the current block. In addition, the MPM list may further include a CNN mode.

When the intra prediction mode of the current block is selected from among the MPMs, first intra identification information indicating which mode of the MPM is selected as the intra prediction mode of the current block is encoded by the encoder 250 and signaled to the image decoding apparatus do.

On the other hand, when the intra prediction mode of the current block is not selected from among the MPMs, second intra identification information indicating which of the remaining modes other than the MPM is selected as the intra prediction mode of the current block is transmitted to the encoder 250 . is encoded and signaled to an image decoding apparatus.

The inter prediction unit 224 generates a prediction block for the current block through motion estimation and motion compensation. That is, the inter prediction unit 224 searches for a block most similar to the current block in the reference picture encoded and decoded before the current picture, and generates a prediction block for the current block using the searched block. Then, the inter prediction unit 224 generates a motion vector corresponding to a displacement between the current block in the current picture and the prediction block in the reference picture.

In general, motion estimation is performed for a luma component, and a motion vector calculated based on the luma component is used for both the luma component and the chroma component. Motion information including information on a reference picture and information on a motion vector used to predict the current block is encoded by the encoder 250 and transmitted to the image decoding apparatus.

The subtractor 230 generates a residual block by subtracting the prediction block generated by the intra prediction unit 222 or the inter prediction unit 224 from the current block.

The transform unit 240 transforms the residual signal in the residual block having pixel values in the spatial domain into transform coefficients in the frequency domain. The transform unit 240 may transform the residual signals in the residual block by using the size of the current block as a transform unit, or divide the residual block into a plurality of smaller subblocks and convert the residual signals in the transform unit of the subblock size. You can also convert There may be various methods for dividing the residual block into smaller subblocks. For example, it may be divided into sub-blocks of the same size as predefined, or a quadtree (QT) type partitioning using a residual block as a root node may be used.

The quantization unit 245 quantizes the transform coefficients output from the transform unit 240 , and outputs the quantized transform coefficients to the encoder 250 .

The encoder 250 generates a bitstream by encoding the quantized transform coefficients using an encoding method such as CABAC. Also, the encoder 250 encodes information such as a CTU size, a QT split flag, a BT split flag, and a split type related to block splitting so that the video decoding apparatus can split the block in the same way as the video encoding apparatus.

The encoder 250 encodes information on a prediction type indicating whether the current block is encoded by intra prediction or inter prediction, and intra prediction information (ie, information about the intra prediction mode) according to the prediction type. information) or inter prediction information (information about reference pictures and motion vectors) is encoded.

The inverse quantization unit 260 generates transform coefficients by inverse quantizing the quantized transform coefficients output from the quantization unit 245 . The inverse transform unit 265 reconstructs a residual block by transforming the transform coefficients output from the inverse quantization unit 260 from the frequency domain to the spatial domain.

The adder 270 reconstructs the current block by adding the reconstructed residual block to the prediction block generated by the prediction unit 220 . Pixels in the reconstructed current block are used as reference pixels when intra-predicting the next block.

The filter unit 280 filters the reconstructed pixels to reduce blocking artifacts, ringing artifacts, blurring artifacts, etc. generated due to block-based prediction and transformation/quantization. carry out The filter unit 280 includes a deblocking filter 282 and an SAO filter 284 .

The deblocking filter 282 deblocks and filters the boundary between the reconstructed blocks in order to remove a blocking artifact caused by block-by-block encoding.

The SAO filter 284 performs additional filtering on the deblocking-filtered image to compensate for a difference between a reconstructed pixel and an original pixel caused by lossy coding.

The reconstruction block filtered using the deblocking filter 282 and the SAO filter 284 is stored in the memory 290 . When all blocks in one picture are reconstructed, the reconstructed picture is used as a reference picture for inter prediction of blocks within a picture to be encoded later.

4 is an exemplary diagram of an image decoding apparatus capable of implementing the techniques of the present disclosure.

The image decoding apparatus includes an image reconstructor 4000 including a decoder 410 , an inverse quantizer 420 , an inverse transform unit 430 , a predictor 440 , an adder 450 , and the like, and a filter unit 460 . and a memory 470 . Each component of the image decoding apparatus may be implemented as hardware or software, or a combination of hardware and software. In addition, the plurality of “units” and “units” may be integrated into at least one module or chip and implemented as at least one processor, except when each needs to be implemented as individual specific hardware.

The decoder 410 decodes the bitstream received from the image encoding apparatus, extracts information related to block division, determines a current block to be decoded, and includes prediction information (eg, hint information) necessary for reconstructing the current block. Information on the residual signal is extracted.

The decoder 410 extracts information about the CTU size from a sequence parameter set (SPS) or a picture parameter set (PPS), determines the size of the CTU, and divides the picture into CTUs of the determined size. Then, the decoder 410 determines the CTU as the highest layer of the tree structure, that is, the root node, and extracts the division information on the CTU to split the CTU using the tree structure.

For example, when a CTU is split using a QTBT structure, a first flag (QT_split_flag) related to QT splitting is first extracted and each node is split into four nodes of a lower layer. And, for a node corresponding to a leaf node of QT, a second flag (BT_split_flag) and split type (split direction) information related to BT splitting are extracted and the corresponding leaf node is split into a BT structure.

As another example, when a CTU is split using the QTBTTT structure, a first flag (QT_split_flag) related to QT splitting is first extracted and each node is split into four nodes of a lower layer. And, for a node corresponding to a leaf node of QT, a split flag (split_flag) indicating whether to be further split into BT or TT and split type (or split direction) information, additional information for distinguishing whether a BT structure or a TT structure to extract Through this, each node below the leaf node of QT is recursively divided into a BT or TT structure.

Meanwhile, when the decoding unit 410 determines a current block to be decoded through division of the tree structure, information on a prediction type indicating whether the current block is intra-predicted or inter-predicted is extracted.

When the prediction type information indicates intra prediction, the decoder 410 extracts a syntax element for intra prediction information (intra prediction mode) of the current block.

Next, when the prediction type information indicates inter prediction, the decoder 410 extracts a syntax element for the inter prediction information, that is, information indicating a motion vector and a reference picture referenced by the motion vector.

Meanwhile, the decoder 410 extracts information on the quantized transform coefficients of the current block as information on the residual signal.

The inverse quantization unit 420 inverse quantizes the quantized transform coefficients, and the inverse transform unit 430 inverse transforms the inverse quantized transform coefficients from the frequency domain to the spatial domain to restore residual signals to generate a residual block for the current block. .

The prediction unit 440 includes an intra prediction unit 442 and an inter prediction unit 444 . The intra prediction unit 442 is activated when the prediction type of the current block is intra prediction, and the inter prediction unit 444 is activated when the prediction type of the current block is inter prediction.

The intra prediction unit 442 determines the intra prediction mode of the current block from among the plurality of intra prediction modes from the syntax element for the intra prediction mode extracted from the decoder 410, and references the vicinity of the current block according to the determined intra prediction mode. Predict the current block using pixels.

When the intra prediction mode for the current block is determined to be the CNN mode, the intra prediction unit 442 performs an inference process of the CNN using the coefficients (ie, filter coefficients) of the convolution kernel determined by the image encoding apparatus to the current block. predict

The inter prediction unit 444 determines a motion vector of the current block and a reference picture to which the motion vector refers by using the syntax element for the inter prediction mode extracted from the decoder 410, and combines the determined motion vector and the reference picture. to predict the current block.

The adder 450 reconstructs the current block by adding the residual block output from the inverse transform unit 430 and the prediction block output from the inter prediction unit 444 or the intra prediction unit 442 . Pixels in the reconstructed current block are used as reference pixels when intra-predicting a block to be decoded later.

By sequentially reconstructing current blocks corresponding to CUs by the image reconstructor 4000 , a CTU composed of CUs and a picture composed of CTUs are reconstructed.

The filter unit 460 includes a deblocking filter 462 and an SAO filter 464 . The deblocking filter 462 deblocks and filters the boundary between the reconstructed blocks in order to remove a blocking artifact caused by block-by-block decoding. The SAO filter 464 performs additional filtering on the reconstructed block after deblocking filtering in order to compensate for a difference between the reconstructed pixel and the original pixel caused by lossy coding. The reconstruction block filtered through the deblocking filter 462 and the SAO filter 464 is stored in the memory 470 . When all blocks in one picture are reconstructed, the reconstructed picture is used as a reference picture for inter prediction of blocks within a picture to be encoded later.

Hereinafter, a CNN-based intra prediction unit according to the present embodiment will be described in detail with reference to the accompanying drawings.

5 is a block diagram showing the configuration of a CNN prediction unit on the side of the video encoding apparatus according to the present embodiment.

5, the CNN prediction unit 500 generates a prediction block by performing CNN-based intra prediction on the encoding target image (ie, the original image) transmitted from the block divider and the reconstructed image transmitted from the adder. can To this end, the CNN prediction unit 500 may include a CNN setting unit 510 and a CNN execution unit 530 .

The CNN setting unit 510 may calculate filter coefficients, ie, coefficients of a convolution kernel, by performing supervised learning using a CNN composed of a plurality of layers. Here, the structure of the CNN is as described above with reference to FIG. 1 , and the CNN may be configured to further include an upsampling layer or a pooling layer to adjust the size of the layer.

Image data input to the input layer (hereinafter, referred to as 'input data') may be configured as a reference region encoded before the current block.

The reference region is one of the neighboring regions adjacent to the current block and blocks of a component coded before a block of a component to be coded among luma blocks and chroma blocks constituting the current block (hereinafter, referred to as a 'current block of another channel'). It may include at least one block (or region). Here, the neighboring area may be an area of the same channel as the current block or an area of a different channel. In addition, the peripheral region may be configured in block units (ie, neighboring blocks) or may be configured in units of pixels (ie, peripheral pixels or peripheral lines). The reference region may further include a new region (ie, an average block, an average pixel, or an average line) generated by averaging pixel values of the surrounding region.

6 is an exemplary diagram of a surrounding area that can be used as input data of CNN. Specifically, FIG. 6A shows the peripheral area in units of blocks, and FIG. 6B shows the peripheral area in units of pixels.

Referring to (a) of FIG. 6 , the block unit reference area, that is, the neighboring blocks, is a left block (C), an upper block (B), an upper right block (D), and a lower left block (E) adjacent to the current block (X). ), the upper left block (A) may be included. In the present specification, an original block (ie, an uncoded block), a prediction block, and a reconstructed block of neighboring blocks are denoted differently. For example, with respect to the upper left block (A), the original block is denoted by 'Ao', the prediction block is denoted by 'Ap', and the reconstructed block is denoted by 'Ar'. In addition, the average block obtained by averaging the pixel values of the neighboring blocks A, B, C, D, E is denoted by 'F'.

Referring to FIG. 6B , the pixel unit reference region may include '1×1' pixels and '1×n' or 'n×1' lines adjacent to the current block X. . For reference, since the block-unit reference region has a wider application range of the convolution kernel than the pixel-unit reference region, it is possible to improve the accuracy of the CNN learning process and inference process. Hereinafter, for convenience of description, the present embodiment will be described on the assumption that the reference area is in block units.

On the other hand, in the YCbCr 4:2:0 or 4:2:2 format, the chroma block is used in its original size, or is up-scaled using an upsampling layer to have the same size as the luma block. can be used

When neighboring blocks of a channel different from the current block are input to the input layer, the right block of the current block (X) in addition to the neighboring blocks (Ar, Br, Cr, Dr, Er) shown in (a) of FIG. 6 ; One or more blocks (not shown) among the lower block and the lower right block may be further input to the input layer. For example, when the current block is a chroma block, the accuracy of intra prediction may be improved by adding one or more blocks among a right block, a lower block, and a lower right block of the current block of the luma channel that have already been encoded as input data.

7 is a diagram illustrating an example of configuring an input layer of a CNN from a plurality of neighboring blocks.

The input layer may be composed of a plurality of layers for each of the neighboring blocks (Ar, Br, Cr, Dr, Er) as shown in (a) of FIG. 7, and as shown in (b) of FIG. The neighboring blocks Ar and Br of may be integrated to form one layer.

Returning to FIG. 5 , image data output from the output layer (hereinafter referred to as 'output data') may be a prediction block of the current block. In this case, the output label may be composed of an original block (ie, an uncoded block) of the current block for supervised learning through comparison with output data.

Some configuration examples of the CNN layer can be summarized as shown in Table 1. However, it should be noted that this is illustrative and does not limit the present embodiment.

CNN Layer Example# input layer output layer data data label Example 1 Neighboring blocks of the same channel as the current block Prediction block of current block original block of current block Example 2 Current block on a different channel than the current block Prediction block of current block original block of current block Example 3 neighboring blocks of the same channel as the current block and their average blocks;
Current block on another channel Prediction block of current block original block of current block Example 4 neighboring blocks of the same channel as the current block and their average blocks;
current block on another channel,
Neighboring blocks of other channels and their average blocks Prediction block of current block original block of current block

Referring to Table 1, in the configuration example of the CNN layer, the data of the input layer may be configured in various combinations, the data of the output layer is the prediction block of the current block, and the label of the output layer is the original block of the current block. The input data and output data should be the same in the learning process and inference process of CNN, respectively.

Meanwhile, the CNN setting unit 510 may set hint information in order to minimize the error between the output data and the output label and to improve the accuracy of intra prediction. Here, the hint information includes at least one of directional information of intra prediction, a quantization parameter (QP) of a current block or reference region, and an absolute sum (ie, amount of residual) of transform coefficients or residual signals of a neighboring block. may include In addition, the hint information may be transmitted to the image decoding apparatus through a bitstream and used to decode the current block.

8 is an exemplary diagram of a prediction direction of a current block based on pixel value shapes of neighboring blocks.

In FIG. 8 , the neighboring blocks of the current block X are composed of an upper left block (A), an upper block (B), and a left block (C).

Referring to (a) of FIG. 8 , looking at the shape of the pixel values of the neighboring blocks A, B, and C, the upper left block (A) has about half of it white, and the left block (C) has mostly white color. However, most of the upper block (B) has a color other than white. Considering that the shape of the pixel value of the current block X is mostly white, it can be seen that performing the intra prediction in the horizontal direction (horizontal direction) can increase the prediction accuracy the most.

Referring to (b) of FIG. 8 , looking at the shape of the pixel values of the neighboring blocks A, B, and C, the upper left block (A) and the left block (C) are mostly white, but the upper block (B) Most of them have a color other than white. Considering that most pixel values of the current block X have a color other than white, it can be seen that performing the intra prediction in the vertical direction (vertical direction) can increase the prediction accuracy the most.

Accordingly, the CNN prediction unit 500 according to the present embodiment intends to improve the accuracy of the intra prediction by using the direction information of the intra prediction as hint information of the learning process and the inference process of the CNN.

The intra-prediction directional information may be an intra-prediction mode number indicating the 65 directional modes and the non-directional modes described above with reference to FIG. 3 . In addition, hint information including one or more prediction direction information may be encoded by the encoder 250 and transmitted to the image decoding apparatus.

In this case, various methods may be used to minimize the amount of bits required to encode hint information. For example, the CNN setting unit 510 selects some of the 65 prediction directions (eg, a horizontal direction, a vertical direction, a diagonal down-right direction, a diagonal up-right direction, etc.) as a representative direction, and selects any one of the selected representative directions. may be set as hint information for intra prediction of the current block. In addition, the CNN setting unit 510 may transmit hint information to the image decoding apparatus in a manner similar to the most probable mode (MPM).

The hint information may include a quantization parameter (QP) indicating the strength of quantization. Here, QP may be a QP value applied to the quantization process of the current block or reference region.

The hint information may include an amount of residual. Here, the amount of residual may be the sum of absolute values of transform coefficients or residual signals of neighboring blocks.

Hint information may be composed of one or more maps and concatenated to a layer of CNN. The map for hint information may be concatenated at various positions between the input layer and the output layer. For example, the map for hint information may be concatenated immediately after the input layer as shown in FIG. 9 , or may be concatenated immediately before the output layer.

Meanwhile, input data may be configured in various combinations according to the direction of intra prediction. For example, when the intra prediction direction is a horizontal direction (horizontal direction), the input data may be composed of one or more blocks selected from among the left neighboring blocks (Ar, Cr, Er) of the current block X and an average block thereof. can Conversely, when the direction of intra prediction is the vertical direction (vertical direction), the input data may be composed of one or more blocks selected from the upper neighboring blocks (Ar, Br, Dr) of the current block X and an average block thereof. can

The CNN setting unit 510 may calculate filter coefficients through an iterative learning process using an error backpropagation algorithm in order to minimize an error between the output data and the output label. Specifically, the error between the output data and the output label may be propagated in the reverse direction from the output layer of the CNN to the input layer through the hidden layer. In the error propagation process, connection weights between nodes may be updated in a direction to reduce the corresponding error. Then, the CNN setting unit 510 may calculate the filter coefficients by repeating the learning process of the CNN using the error backpropagation algorithm until the corresponding error is less than a predetermined threshold.

The above filter coefficient calculation process may be performed in a predetermined unit (eg, CU, CTU, slice, frame, or sequence (group of frames) unit). For example, the CNN setting unit 1010 may calculate a filter coefficient for each current block or may calculate a filter coefficient for each frame.

When the filter coefficients are calculated in units of frames, the corresponding filter coefficients may be commonly used for intra prediction of a plurality of current blocks included in the corresponding frame. In this case, there may be a plurality of prediction directionality information, which is one of the hint information. For example, when the intra prediction directionality information consists of one map, one map may include a plurality of directionality values.

Information on the calculated filter coefficients may be transmitted to an image decoding apparatus through a bitstream and used in an image decoding process.

Also, the CNN setting unit 510 may configure a filter coefficient set by pre-calculating a plurality of filter coefficients using predetermined sample images. In this case, the CNN setting unit 510 may set one filter coefficient selected according to a predetermined criterion in the corresponding set as the filter coefficient for the current block. For example, the CNN setting unit 510 may select one filter coefficient from the set based on the similarity of pixel values between the current block and sample images. Alternatively, the CNN setting unit 510 may select a filter coefficient most similar to the filter coefficient calculated through one learning process from the corresponding set. Filter coefficient selection information, eg, index information, may be transmitted to an image decoding apparatus through a bitstream and used in an image decoding process.

Meanwhile, although FIG. 5 shows that the CNN setting unit 510 is included in the CNN prediction unit 500, it should be noted that this is an example, and the present embodiment is not limited thereto. That is, the CNN setting unit 510 may be implemented as a unit separate from the CNN prediction unit 500 , or may be implemented as a single unit by being integrated with the CNN execution unit 530 .

The CNN execution unit 530 performs a CNN-based reasoning process on the input data using the filter coefficients set by the CNN setting unit 510, that is, the coefficient values of the convolution kernel, so that the output data, that is, the current block A prediction block can be generated. In this case, the generated prediction block may be transmitted to a subtractor of the image encoding apparatus and used to generate a residual block from the current block.

10 is a block diagram illustrating the configuration of a CNN prediction unit on the side of the image decoding apparatus according to the present embodiment. The CNN prediction unit of FIG. 10 differs from the CNN prediction unit of FIG. 5 only in the method of setting the input signal and filter coefficients, that is, the coefficient values of the convolution kernel, and the description of overlapping content will be omitted or simplified. do it with

Referring to FIG. 10 , the CNN prediction unit 1000 may generate a prediction block by performing CNN-based intra prediction based on a reconstructed image. To this end, the CNN prediction unit 1000 may include a CNN setting unit 1010 and a CNN execution unit 1030 .

The structure of the CNN is as described above with reference to FIG. 1 , and the CNN may be configured to further include an upsampling layer or a pooling layer to adjust the size of the layer.

Image data input to the input layer (hereinafter, referred to as 'input data') may be configured as a reference region decoded before the current block.

The reference region includes a neighboring region adjacent to the current block, and blocks of a component decoded earlier than a block of a component to be decoded among luma blocks and chroma blocks constituting the current block (hereinafter, referred to as a 'current block of another channel'). It may include at least one block (or region). Here, the neighboring area may be an area of the same channel as the current block or an area of a different channel. In addition, the peripheral region may be configured in block units (ie, neighboring blocks) or may be configured in units of pixels (ie, peripheral pixels or peripheral lines).

The reference region may further include a new region (ie, an average block, an average pixel, or an average line) generated by averaging pixel values of the surrounding region. For example, the input data may be composed of neighboring blocks of the same channel as the current block, and an average block thereof and a current block of a different channel.

Hereinafter, for convenience of description, the present embodiment will be described on the assumption that the reference area is in block units.

As described above with reference to FIG. 7 , the input layer may be composed of a plurality of layers for each neighboring block, or a plurality of neighboring blocks may be integrated to form one layer.

Image data output from the output layer (hereinafter, referred to as 'output data') may be a prediction block of the current block.

Some configuration examples of the CNN layer are as described above with reference to Table 1. However, it should be noted that this is illustrative and does not limit the present embodiment.

The CNN setting unit 1010 may configure one or more maps using hint information transmitted from the image encoding apparatus, and then concatenate them at various positions between the input layer and the output layer.

Hint information is information for improving the accuracy of intra prediction, and is an absolute sum (ie, residual) of prediction direction information, a quantization parameter (QP) of a current block or reference region, and transform coefficients or residual signals of a neighboring block. of) may include at least one of

The prediction directionality information included in the hint information may be an intra prediction mode number indicating 65 directional modes and non-directional modes, or index information indicating any one of one or more representative directions selected from 65 directional modes. have.

Meanwhile, input data may be configured in various combinations according to the direction of intra prediction. For example, when the intra prediction direction is a horizontal direction (horizontal direction), the input data may be composed of one or more blocks selected from the left neighboring blocks of the current block and an average block thereof. Conversely, when the intra prediction direction is a vertical direction (vertical direction), the input data may include one or more blocks selected from neighboring blocks above the current block and an average block thereof.

The CNN setting unit 1010 may set the filter coefficients transmitted from the image encoding apparatus as filter coefficients for intra prediction of the current block. In this case, the filter coefficient may be a value calculated in a predetermined unit, for example, a CU unit or a frame unit by the image encoding apparatus.

When the filter coefficients are set in units of frames, the corresponding filter coefficients may be commonly used for intra prediction of a plurality of current blocks included in the corresponding frame. In this case, there may be a plurality of prediction directionality information, which is one of the hint information. For example, the directionality information of intra prediction consists of one map, but a plurality of directionality values may be included in one map.

When the image encoding apparatus and the image decoding apparatus operate the same filter coefficient set, the CNN setting unit 1010 may set filter coefficients for intra prediction of the current block based on index information of the filter coefficients transmitted from the image encoding apparatus. have.

Meanwhile, although FIG. 10 shows that the CNN setting unit 1010 is included in the CNN prediction unit 1000, it should be noted that this is an example, and the present embodiment is not limited thereto. That is, the CNN setting unit 1010 may be implemented as a unit separate from the CNN prediction unit 1000 . Also, the CNN setting unit 1010 may be integrated with the CNN execution unit 1030 and implemented as a single unit.

The CNN execution unit 1030 performs a CNN-based reasoning process on the input data using the filter coefficients set by the CNN setting unit 1010, that is, the coefficient values of the convolution kernel, thereby generating output data, that is, for the current block. A prediction block can be generated.

Then, the generated prediction block is transmitted to the adder and added to the residual block, so that it can be used to reconstruct the current block.

Hereinafter, an exemplary method for performing CNN-based intra prediction according to the present embodiment will be described with reference to FIGS. 11 and 12 .

11 is a flowchart illustrating the operation of the CNN prediction unit of the image encoding apparatus of FIG. 5 .

Referring to FIG. 11 , in step S1110 , the CNN setting unit 510 may set input data and output labels of the CNN.

The input data may consist of a reference region encoded before the current block. For example, the input data may be composed of neighboring blocks of the same channel as the current block. Alternatively, the input data may be composed of neighboring blocks of the same channel as the current block and an average block thereof, and a current block of a channel different from the current block.

The data of the output layer may be a prediction block of the current block, and the label of the output layer may be composed of the original block of the current block.

The CNN setting unit 510 may set the direction information of prediction as hint information in order to improve the accuracy of intra prediction. The set hint information may be transmitted to the image decoding apparatus through a bitstream and used to decode the current block. In this case, the input data may be composed of various combinations according to the directionality of the intra prediction.

In step S1120, the CNN setting unit 510 may calculate filter coefficients through a learning process. The CNN setting unit 510 may repeat the learning process using an error backpropagation algorithm in order to improve the accuracy of intra prediction.

The filter coefficient calculation process may be performed in a predetermined unit, for example, in a frame unit or a block unit. The CNN setting unit 510 may configure a filter coefficient set by pre-calculating a plurality of filter coefficients using predetermined sample images. In this case, the CNN setting unit 510 may set one filter coefficient selected according to a predetermined criterion in the corresponding set as the filter coefficient for the current block.

In step S1130, the CNN execution unit 530 performs a CNN-based reasoning process on the input data using the filter coefficients set by the CNN setting unit 510, that is, the coefficient values of the convolution kernel, thereby outputting data, that is, A prediction block for the current block may be generated. In this case, the generated prediction block may be transmitted to a subtractor of the image encoding apparatus and used to generate a residual block from the current block.

12 is a flowchart illustrating an operation of the CNN prediction unit of the image decoding apparatus of FIG. 10 .

Referring to FIG. 12 , in step S1210 , the CNN setting unit 1010 may set filter coefficients for intra prediction of the current block based on information on filter coefficients transmitted from the image encoding apparatus.

The input data of the CNN may be composed of a reference region decoded before the current block, and the output data becomes a prediction block for the current block.

When hint information for intra prediction is delivered from the image encoding apparatus, the CNN setting unit 1010 may configure the hint information extracted by the decoder into one map and concatenate the hint information to the CNN layer.

Meanwhile, input data may be configured in various combinations according to the direction of intra prediction.

In step S1220, the CNN execution unit 1030 performs a CNN-based reasoning process on the input data using the filter coefficients set by the CNN setting unit 1010, that is, the coefficient values of the convolution kernel, thereby output data, that is, A prediction block for the current block may be generated. In this case, the generated prediction block may be transferred to the adder and added to the residual block to be used to reconstruct the current block.

11 and 12, it has been described that a plurality of steps are sequentially performed, but this is merely illustrative of the technical idea of the present embodiment. In other words, those of ordinary skill in the art to which this embodiment pertains may change the order described in FIGS. 11 and 12 or perform some of the plurality of steps in parallel without departing from the essential characteristics of the present embodiment. 11 and 12 are not limited to a time-series order, since various modifications and changes may be applied to the performance.

On the other hand, each step of the flowchart shown in FIGS. 11 and 12 may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded on a computer-readable recording medium. can The computer-readable recording medium includes all kinds of recording devices in which data readable by a computer system is stored. That is, the computer-readable recording medium includes a storage medium such as a magnetic storage medium (eg, ROM, floppy disk, hard disk, etc.) and an optically readable medium (eg, CD-ROM, DVD, etc.). In addition, the computer-readable recording medium may be distributed in a network-connected computer system to store and execute computer-readable codes in a distributed manner.

The above description is merely illustrative of the technical idea of this embodiment, and various modifications and variations will be possible by those skilled in the art to which this embodiment belongs without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain rather than limit the technical spirit of the present embodiment, and the scope of the technical spirit of the present embodiment is not limited by these embodiments. The protection scope of this embodiment should be interpreted by the following claims, and all technical ideas within the equivalent range should be interpreted as being included in the scope of the present embodiment.

Claims (15)

A video decoding method based on intra prediction, comprising:
Decodes block size information from the bitstream, decodes partition information for dividing a block having a size determined by the block size information into a tree structure, determines a current block to be decoded, and converts transform coefficients for the current block decrypting;
constructing input data using a reference region decoded before the current block;
generating prediction pixels of the current block by applying a 2-dimensional or 3-dimensional filter coefficient set to the input data;
generating residual signals for the current block by inverse transforming the transform coefficients; and
reconstructing the current block using the prediction pixels and the residual signals
video decoding method. According to claim 1,
The set of filter coefficients is an image decoding method, characterized in that the kernel coefficients of a CNN (Convolutional Neural Network). 3. The method of claim 2,
The input data is
Further comprising hint information for intra prediction of the current block,
The hint information is
Absolute sum of prediction directionality information, a quantization parameter (QP) of the current block or the reference region, and transform coefficients or residual signals of a reconstructed neighboring region adjacent to the current block
containing one or more of
video decoding method. According to claim 1,
Further comprising the step of decoding first information related to the configuration of the input data from the bitstream,
The input data is configured by combining pixel values obtained from reference regions adjacent to the left and upper sides of the current block based on the first information.
video decoding method. According to claim 1,
The reference area is
Reference pixels in a restored peripheral region adjacent to the current block, and at least one of already decoded components among luma and chroma components constituting the current block.
video decoding method. According to claim 1,
The input data is
It is constructed using pixel values generated by averaging pixel values in the reference region.
video decoding method. According to claim 1,
The method further comprises decoding, from the bitstream, second information for selecting a filter coefficient set to be applied to the current block from among a plurality of filter coefficient sets configured in two or three dimensions,
The prediction pixels of the current block are generated using a filter coefficient set determined based on the second information.
video decoding method. An intra prediction-based video encoding method, comprising:
determining a current block to be encoded by dividing a predetermined block into a tree structure;
constructing input data from a reference region encoded before the current block;
generating a prediction block of the current block by applying a 2-dimensional or 3-dimensional filter coefficient set to the input data;
generating a residual block by subtracting the prediction block from the current block; and
generating encoded data by encoding block size information indicating the size of the predetermined block, partition information related to the tree structure partitioning, and the residual block
Video encoding method. 9. The method of claim 8,
The set of filter coefficients is an image encoding method, characterized in that the kernel coefficients of a CNN (Convolutional Neural Network). 10. The method of claim 9,
The input data is
Further comprising hint information for intra prediction of the current block,
The hint information is
Absolute sum of prediction directionality information, a quantization parameter (QP) of the current block or the reference region, and transform coefficients or residual signals of a reconstructed neighboring region adjacent to the current block
containing one or more of
Video encoding method. 9. The method of claim 8,
The step of generating the encoded data includes:
encoding first information related to the configuration of the input data;
The input data is configured by combining pixel values obtained from reference regions adjacent to the left and upper sides of the current block so as to conform to the input data configuration method indicated by the first information. 9. The method of claim 8,
The reference area is
Reference pixels in a restored peripheral region adjacent to the current block, and at least one of already decoded components among luma and chroma components constituting the current block.
Video encoding method. 9. The method of claim 8,
The input data is
It is constructed using pixel values generated by averaging pixel values in the reference region.
Video encoding method. 9. The method of claim 8,
The step of generating a prediction block of the current block comprises:
selecting a filter coefficient set to be applied to the current block from among a plurality of filter coefficient sets configured in two or three dimensions; and
generating the prediction block using the selected filter coefficient set;
The step of generating the encoded data includes:
encoding second information for selecting a filter coefficient set to be applied to the current block from among the plurality of filter coefficient sets;
Video encoding method. A decoder-readable recording medium for storing an encoded bitstream of video data, comprising:
The bitstream is
determining a current block to be encoded by dividing a predetermined block into a tree structure;
composing input data from a reference region encoded before a current block to be encoded;
generating a prediction block of the current block by applying a 2-dimensional or 3-dimensional filter coefficient set to the input data;
generating a residual block by subtracting the prediction block from the current block; and
A decoder, characterized in that it is generated by executing an intra prediction-based video encoding method comprising encoding block size information indicating the size of the predetermined block, partition information related to the tree structure partitioning, and encoding the residual block, readable recording medium.

Images