KR102053242B1 - Machine learning algorithm using compression parameter for image reconstruction and image reconstruction method therewith - Google Patents

Abstract

The present invention uses compressed information and a degraded image as input data, and is configured to learn and derive an optimal model corresponding to various compressed information by using a machine learning algorithm aiming to restore the original image. It is possible to remarkably improve the image resilience and compression rate by applying the optimal model corresponding to the information, and to construct the loss function which is a function for calculating the difference between the reconstructed image and the original image during learning. The present invention provides a machine learning algorithm capable of precisely restoring an image for a specific region by assigning a weight and an image restoration method using the same.

Description

Machine learning algorithm using compression parameter for image reconstruction using image compression method and image reconstruction method

The present invention relates to a machine learning algorithm for image reconstruction using compression parameters and an image reconstruction method using the same. Specifically, the present invention reconstructs an image by using compression information during image encoding and decoding, and simultaneously machine learning. It is to provide a machine learning algorithm that can improve the image resilience and compression rate by being configured to learn and derive the optimal reconstruction method corresponding to the compressed information through the self and to provide an image reconstruction method using the same.

As the content industry expands and display technology develops, research on image compression technology is being actively conducted.

In particular, as broadcast services with high definition (HD) resolutions have recently been expanded, many users have become accustomed to high resolution and high definition video. Interest is soaring.

Currently, compression standards include JPEG, H.264, MPEG2, and HEVC. In the compression process of the compression standard, the image is divided into blocks of a certain size, and then quantized and predicted by each block. Compress the data through.

However, in the current compression standard, since image prediction and quantization are performed based on divided blocks, the boundary between blocks is degraded during compression.

As a method for solving this problem, loop filters, adaptive deblocking filters, and sample adaptive offset filter technologies have been studied and used. However, such a conventional method simply performs an image filter using only a few parameters determined by various studies and experiments. It is configured to.

In general, an image filter should be applied with optimal parameter values according to various conditions such as compression method, block size and quantity, and set values. However, since the image is reconstructed using only a few fixed parameter values, a reconstructed image The quality of the falling has a structural limit.

In particular, in the case of video compression such as HEVC (High Efficiency Video Coding), since the reconstructed image is encoded as a reference image in an adjacent frame, the compression rate decreases as the image quality of the reconstructed image decreases.

1 is a block diagram illustrating a decoding structure diagram of H. 264.

The H. 264 (100) of FIG. 1 processes data in units of macroblocks having a width of 16 × 16 pixels, and receives a bitstream and decodes the data in an intra mode or an inter mode. The reconstructed image is output.

In the intra mode, the switch is switched to intra, and in the inter mode, the switch is switched to inter.

In addition, the main flow of the decoding process is to first generate a prediction block, and then decode the input bitstream to add a block and a prediction block to generate a reconstructed block.

In addition, generation of the prediction block of the H.264 100 is performed according to the intra mode and the inter mode.

In the intra mode, the H.264 100 generates a prediction block by performing spatial prediction using the neighboring pixel values of the current block in the intra prediction process, and uses the motion vector in the inter mode. A prediction block is generated by searching for an area in the reference picture stored in the reference picture buffer and performing motion compensation.

Also, in the entropy decoding process, an quantized coefficient is output by performing entropy decoding according to a probability distribution of an input bitstream, and inverse quantization process and inverse transformation of the quantized coefficient are reconstructed through a predictive image and an adder. After generating a block, a blocking phenomenon is removed through a deblocking filter and then stored in a reference picture buffer.

However, in the conventional H.264 100, since the deblocking filter applied to remove the blocking artifact performs the image reconstruction using only a small number of preset parameters, various conditions of compression and deteriorated image are performed. It has a structural limitation that does not correspond to its characteristics.

The present invention has been made to solve such a problem, and the problem of the present invention is to use the machine learning algorithm which aims to restore the original image, which is the image before compression is degraded by compression information and deteriorated image as input data. Machine learning algorithm and image restoration method that can significantly improve image reconstruction power by applying the compressed information input to the trained model when reconstructing the image. It is to provide.

In addition, another problem of the present invention is to construct a loss function, which is a function for obtaining a difference value between a reconstructed image and an original image during learning, in units of blocks according to a block size, an inter / intra prediction mode of a block, a quantization parameter, and the like. Another object of the present invention is to provide a machine learning algorithm capable of precisely restoring an image on a specific region by assigning different weights and an image restoring method using the same.

In addition, another object of the present invention is to be applied to the image encoding step in consideration of the characteristic that the image quality of the reconstructed image is improved by the machine learning algorithm, so that the reconstructed image of the improved image quality is used for prediction of the adjacent frame image during image encoding. Therefore, it is to provide a machine learning algorithm that can increase the compression rate and an image restoration method using the same.

According to an aspect of the present invention, there is provided an image restoration method for restoring an image deteriorated by an image compression by an image encoder. The image restoration method uses a predetermined machine learning algorithm. A training step of deriving a set of optimal machine learning parameter values for improving image quality from the compressed information, the degraded image, and the original image; An inference step of restoring image quality by applying a set of parameter values determined in the learning step by using the deteriorated image and the compressed information reconstructed in the process of decoding from the compressed data as input values of the machine learning algorithm; The image encoder may include image segmentation information including block information, which is a unit for encoding an image during image compression, and location information at which blocking artifacts occur, and each block unit in an intra prediction mode. Extracts the compressed information including the intra prediction information determined by the step information, the motion vector information determined in the inter prediction mode, and the quantization parameter information generating the quantization parameter information applied during the quantization, and extracts the compressed information. The algorithm determines a difference value between the deteriorated image and the original image. After the calculation, a loss function is applied to reduce the calculated difference, and the objective function is obtained by assigning different weights according to image segmentation information which is a unit of image coding in the compressed information, and then calculating the difference value. The weight is formed to increase as the size of the divided block decreases,
The objective function is preferably defined by Equation 1 below.

delete

delete

[Equation 1]

L: objective function, w: image width, h: image height, D: degraded image, G: original image, Mi: matrix of image sizes with values of '0' or '1', Wi: weight

In addition, in the present invention, the machine learning algorithm is applied to a video encoder operated by a compression standard of H.262 and HEVC, which is a known deblocking filter, SAO (Sample AdaptiveOffset), and ALF (Adaptive Loop Filter). It is preferable to replace any one so that the reconstructed image is used for prediction of the adjacent frame image.

In the present invention, it is preferable that the machine learning algorithm is applied to an image encoder to be used as post-processing of an image reconstructed in a predetermined manner so that the reconstructed image is used for prediction of an adjacent frame image.

delete

According to the present invention having the above-mentioned problems and solving means, by using the compressed information and deteriorated image as input data, by using a machine learning algorithm aiming to restore the original image, by learning the optimal model corresponding to various compressed information It is configured to derive the image reconstruction power and compression ratio can be significantly improved by applying the optimal model corresponding to the compressed information when the image is reconstructed.

In addition, according to the present invention, in constructing a loss function that is a function for obtaining a difference value between the reconstructed image and the original image during learning, different weights are provided in units of blocks according to the block size, the inter / intra prediction mode of the block, the quantization parameter, and the like. By assigning, it is possible to precisely perform image restoration on a specific area.

In addition, the present invention is configured to be applied to the image encoding step in consideration of the characteristic that the image quality of the reconstructed image is improved by the machine learning algorithm. It can increase.

1 is a block diagram illustrating a decoding structure diagram of H. 264.
2 is a block diagram illustrating a typical video encoding apparatus for explaining the present invention.
3 is a block diagram illustrating a process of extracting compressed information by the subtractor of FIG. 2.
4 is a flowchart showing an image restoration method according to an embodiment of the present invention.
FIG. 5 is an exemplary diagram for describing formatting of block structure information (CU) which is one of image segmentation information of HEVC among the image segmentation information in the learning step of FIG. (B) is an illustration which shows the image which marked the specific value on the boundary surface of the image of (a).
FIG. 6 (a) is an exemplary diagram showing an input image divided into blocks according to CU information, and (b) shows a weight '1' in an area divided into 32 × 32 blocks in the image of (a) by the objective function. (C) is an exemplary diagram showing when the weight '1' is assigned to an area divided into 32 × 32 blocks in the image of (a) by the objective function, (d) ) Is an exemplary diagram showing when a weight '1' is assigned to an area divided into 32 × 32 blocks in the image of (a) by the objective function.
(A) of FIG. 7 is an exemplary diagram showing an original image, (b) is an exemplary diagram showing an image degraded during image compression, and (c) is a deblocking filter of a known HEVC and an image reconstructed by SAO. (D) is an illustration showing an image reconstructed by the present invention.

Hereinafter, with reference to the accompanying drawings will be described an embodiment of the present invention.

2 is a block diagram illustrating a typical video encoding apparatus for explaining the present invention.

The image encoding apparatus 200 needs to be decoded and stored in order to be used as a reference image by performing inter prediction encoding, that is, inter-frame prediction encoding.

Accordingly, the quantized coefficients are inversely quantized by the inverse quantizer 260 and inversely transformed by the inverse transformer 270. The inverse quantized and inverse transformed coefficients are added to the prediction block through the adder 275, and a reconstruction block is generated.

The reconstruction block passes through the filter unit 280, and the filter unit 280 applies at least one or more of a deblocking filter, a sample adaptive offset (SAO), and an adaptive loop filter (ALF) to the reconstruction block or the reconstruction picture. Can be.

 The filter unit 280 may be called an adaptive in-loop filter. The deblocking filter may remove block distortion or blocking artifacts generated at the boundary between blocks.

In addition, the subtractor 225, the transformer 235, and the quantizer 245 compress an image by detecting a difference value due to a difference between an input block and a generated prediction block, and then quantizing and storing the detected difference value. . In this case, compressed information (information on a difference value and a difference value) is stored in a stream in which the compressed image is stored.

3 is a block diagram illustrating a process of extracting compressed information by the subtractor of FIG. 2.

The subtractor 225 extracts the compressed information through the compressed information extraction step (S220).

As shown in FIG. 3, the compressed information extracting step S220 includes an image segmentation information extracting step 2210, an intra prediction information extracting step S2220, an inter prediction information extracting step S2230, and a quantization parameter information extracting step ( S2230) at least one or more.

The image segmentation information extraction step S2210 extracts the image segmentation information that is the structure information of the block which is the unit for encoding the image during the image compression by the compression step S210. In this case, the image segmentation information includes a coding unit (CU), a prediction unit (PU), and a transform unit (TU) information.

In this case, since the known image encoder 200 encodes an image in units of blocks during image compression, a blocking artifact occurs on the boundary of the block during the compression process. Accordingly, in the present invention, the image segmentation information, which is the structure information of the block, is extracted in consideration of the characteristics indicating the position of the boundary surfaces of the blocks. The image segmentation information may be utilized when the image is restored by the restoration method S1.

The intra prediction information extraction step (S2220) is a step of extracting the intra prediction information determined in each image segmentation information unit in the intra prediction mode in the compression step S210.

The inter prediction information extraction step S2230 is a step of extracting motion vector information determined in an inter prediction mode in the compression step S220.

The quantization parameter information extraction step S2230 is a step of extracting quantization parameter information applied when quantization is performed in the compression step S220.

In this case, the quantization parameter is important information for determining how much the original image is to be compressed. In general, when the quantization parameter value is large, the compression ratio increases, but the compressed image quality decreases. In other words, there is a close relationship between the quantization parameter and the degree of degradation of the compressed image.

That is, the information including at least one or more of the image segmentation information, the intra prediction information, the inter prediction information, and the quantization parameter information extracted by the compressed information extraction step S220 is called compressed information.

4 is a flowchart showing an image restoration method according to an embodiment of the present invention.

According to an embodiment of the present invention, an image reconstruction method (S1) uses compressed information, a decoded image (input image, a degraded image), or a reconstructed image as input data, and machine learning using the original image as output data. It is to improve the image resilience and compression rate by filtering the image based on the optimal filter value by improving the filtering technique of the in-loop filter used in the image reconstruction using the machine algorithm. .

In this case, various algorithms such as a linear regression artificial neural network and a support vector machine can be applied to the machine learning algorithm.

In addition, the machine learning algorithm may be used to detect the degraded region in the image or to restore the image from the compressed information input together with the degraded image.

In this case, the information output to the machine learning algorithm may be an image of which image quality is restored, or a feature vector for restoring image quality, or a residual image added to a deteriorated image.

In addition, as shown in FIG. 4, the image restoration method S1 includes a learning step S10, an image decoding limit S20, an input step S30, and an inference step S40.

In the learning step S10, the compressed information, the degraded image, and the original projection are extracted in advance from a large amount of image data and used as input data. In this case, the learning step S10 is a separate process not included in the image encoding and decoding process.

In addition, in the learning step S10, the compressed information and the deteriorated image are input data, and a model, which is a mapping relation between the compressed information and a given image, is given using a predetermined machine learning algorithm aimed at restoring the original image. Learn from a large number of data. In this case, the machine learning algorithm trains the model so that the input image is output as close as possible to the target image.

At this time, a variety of methods and techniques may be applied as a formatting method for using the compressed information as input data in the machine learning model in the learning step (S10). Let's explain.

FIG. 5 is an exemplary diagram for describing formatting of block structure information (CU) which is one of image segmentation information of HEVC among the image segmentation information in the learning step of FIG. 4, and (a) divides an input image into the size of a CU block. (B) is an illustration which shows the image which marked the specific value on the boundary surface of the image of (a).

In general, since image compression is performed by dividing an image into block units and then compressing them into divided block units, different blocking parameters are used for each block, and thus blocking artifacts occur in which unnatural tomographic planes occur at the boundary of blocks. Will appear.

In the present invention, in consideration of the characteristic that the block structure information (CU), which is one of the image segmentation information included in the compressed information, may indicate location information at which a blocking artifact occurs, as shown in FIG. After generating a matrix having the same size as the input image, as shown in (b) it is used as the input value of the model by marking the boundary of the block detected through the block structure information as a specific value.

The machine learning algorithm defines the loss function in the direction of reducing the difference after obtaining the difference between the image reconstructed by the model and the original image.

The loss function applied to the machine learning algorithm of the present invention is a modification of the Mean Squared Error function, which is a loss function that is typically used for image reconstruction. In one embodiment, a coding unit is used. It can be configured to give a greater weight to the area determined by the small block in.

The reason for this is that a region having a small size of the block structure information (CU), which is image segmentation information, has many characteristics in the image encoding process, so that when a difference value is calculated, a larger weight is given to a region having many high frequency components. It is possible to calculate a precise difference value for the component, and accordingly, the machine learning algorithm can improve the image quality of the reconstructed image by learning in a direction in which the reconstruction of the small area of the block structure information CU is better. .

The objective function applied to the learning step S10 of the present invention will be described in detail with reference to the following equations.

The objective function of the present invention is defined by the following equation.

Where L is the objective function, w is the image width, h is the image height, D is the degraded image, G is the original image, Mi is a matrix of image sizes with values of '0' or '1', Wi is a weight.

That is, the loss function applied to the learning step (S10) of the present invention assigns different weight (Wi) values only to a specific area so as to give different weights according to the size of the block. By assigning '0', the difference value can be calculated with different weights when calculating the difference value of each block area.

FIG. 6 (a) is an exemplary diagram illustrating an input image divided into blocks according to a CU, and (b) is a weight '1' in an area divided into 32 × 32 blocks in the image of (a) by an objective function. (C) is an exemplary diagram showing when the weight '1' is assigned to a region divided into 32x32 blocks in the image of (a) by the objective function, (d) Is an exemplary diagram showing when a weight '1' is assigned to a region divided into 32x32 blocks in the image of (a) by the objective function.

The objective function applied to the learning step (S10) of the present invention assumes that the CU is partitioned as shown in (a) of FIG. 6, the region M_0 divided into 32 × 32 blocks as shown in (b). A weight of 1 'may be given, and a weight of' 0 'may be given to the remaining area.

In addition, as shown in (c), the objective function may assign a weight of '2' to the region M_1 divided into 16 × 16 blocks, and give a weight of '0' to the remaining regions.

In addition, as shown in (d), the objective function may assign a weight of '3' to the region M_2 divided into 8 × 8 blocks, and may assign a weight of '0' to the remaining regions.

By assigning a weight '3' to the area M_2 having the smallest block size (area with many high frequency components), a precise difference value can be calculated for the high frequency component, thereby significantly improving the quality of the reconstructed image.

As described above, the learning step (S10) of the present invention uses the compressed information and the deteriorated image as input data, and learns the mapping relationship between the input image and the target image by using a machine learning algorithm aiming to restore the original image. At the same time, it is configured to calculate the difference value by assigning different weights according to the block size, thereby effectively removing the blocking artifact during image reconstruction, thereby maximizing image resilience.

4 again referring to the image decoding step S20, the image decoding step S20 is a step of reconstructing the image by releasing the compressed data that has already been compressed.

At this time, the image decoding step S20 inputs the reconstructed image (degraded image) and the compressed information included in the compressed data into the input step S30.

The input step S30 is a step of receiving the deteriorated image and the compressed information restored from the image decoding step S20.

The inference step S40 is a step of reconstructing the image by performing image filtering by applying the input data input from the input step S30 to the model learned by the learning step S10.

(A) of FIG. 7 is an exemplary diagram showing an original image, (b) is an exemplary diagram showing an image degraded during image compression, and (c) is a deblocking filter of a known HEVC and an image reconstructed by SAO. (D) is an illustration showing an image reconstructed by the present invention.

Looking at the present invention with reference to Figure 7, the deteriorated image of (b) compared to the original image of (a) not only the image quality is deteriorated, but also a blocking phenomenon remains, it can be seen that the wave-shaped artifacts are formed. .

In addition, as shown in (c), the deblocking filter of the known HEVC and the image reconstructed by SAO show that the blocking phenomenon is partially removed compared to the deteriorated image of (b), but compared with the original image of (a). It can be seen that not only the image quality deteriorates but also a lot of blocking and wave-like artifacts are formed.

In addition, as in (d), the reconstructed image to which the image reconstruction method (S1) of the present invention is applied not only has improved image quality when compared with (b) and (c), but also significantly reduces blocking and wave-like artifacts. It can be seen that.

Table 1 below is an experimental value to indicate the case of using the CU information as in the present invention, and the case of not using.

In Table 1, the residual block is a value proportional to the number of layers of the neural network in the neural network algorithm.

In addition, the average PSNR of the input image (deteriorated image) used in the experiment is 30.247 (db), and the PSNR of the image reconstructed by the in-loop filter (deblocking filter and SAI) of HEVC is 30.517 (db).

As shown in Table 1, when image restoration is performed without utilizing CU information, which is one of the image segmentation information, the signal-to-noise ratio (PSNR) is '31 .151 (db) 'when the number of residual blocks is five. When the gain is measured as' 0.905 (db) 'and the CU information is used under the same conditions, the signal-to-noise ratio (PSNR) is '31 .233 (db)' and the gain is' 0.986 '. (db) 'can be seen.

That is, the signal-to-noise ratio (PSNR) and the gain (gain) improved by '0.081 (db)' when using the CU information.

In addition, when image restoration is performed without using CU information, the signal-to-noise ratio (PSNR) is '31 .222 (db) 'for 15 residual blocks, and the PSNR gain (0.9) (0.9) compared to the input image is 0.97 (db). In the case of using CU information under the same conditions, it can be seen that the signal-to-noise ratio (PSNR) is measured as '31 .303 (db) 'and the gain is measured as' 1.056 (db)'.

That is, when the CU information is used, the signal-to-noise ratio (PSNR) and the PSNR gain compared to the input image are improved by '0.081 (db)'.

In addition, the machine learning algorithm of the present invention is applied to the image encoding step, and in detail, to the filter unit 260 of the image encoder 100 of FIG. 2, and thus, a conventional deblocking filter and a sample adaptive offset. ) Can be configured to replace

If the machine learning algorithm of the present invention is applied to the filter unit 260 of the image encoder, as described above with reference to FIGS. 4 to 6, the image quality of the reconstructed image is improved. Due to the excellent image quality, the compression rate can be significantly increased.

In addition, the machine learning algorithm may be configured to be applied to post-processing of the reconstructed image according to a conventionally known method and to utilize the reconstructed image for prediction of adjacent frame images.

In other words, the machine learning algorithm of the present invention may be configured to be applied only to an image decoder or 2) to both an image decoder and an image encoder. Objectives and effects can be expected, and if applied to the composition 2) can not only improve the image resilience but also can be expected to achieve the purpose and effect of increasing the compression ratio.

As described above, the image restoration method S1 according to an embodiment of the present invention uses compressed information and a degraded image as input data, and an optimal model corresponding to various compressed information using a machine learning algorithm aiming to restore the original image. It is configured to learn and derive by itself, so that the image resilience and compression rate can be remarkably improved by applying an optimal model corresponding to the compressed information during image reconstruction.

In addition, the image restoration method (S1) of the present invention in the configuration of a loss function that is a function for obtaining a difference value between the restored image and the original image during learning, restoring the image for a specific region by assigning different weights according to the size of the block Can be performed precisely.

S1: Image Restoration Method S10: Learning Steps
S20: Image decoding step S30: Input step
S40: Inference step S2210: Image segmentation information extraction step
S2220: extracting intra prediction information S2230: extracting inter prediction information S2240: generating quantization parameter

Claims (6)

In an image restoration method for restoring an image degraded by image compression by an image encoder:
The image restoration method
A training step of deriving a set of optimal machine learning parameter values for improving image quality from compressed information, a degraded image, and an original image by using a preset machine learning algorithm;
An inference step of restoring image quality by applying a set of parameter values determined in the learning step by using the deteriorated image and the compressed information reconstructed in the process of decoding from the compressed data as input values of the machine learning algorithm; ,
The video encoder
Image segmentation information including block information, which is a unit for encoding an image during image compression, and location information in which blocking artifacts occur, and intra prediction information determined in units of blocks in an intra prediction mode. And compressed information including motion vector information determined in an inter prediction mode and quantization parameter information for generating quantization parameter information applied during quantization,
The machine learning algorithm
After calculating a difference value between the deteriorated image and the original image, a loss function is applied to reduce the calculated difference value.
The objective function is formed to increase as the size of the block in which the weight is divided decreases by assigning different weights according to image segmentation information which is a unit of image coding in the compressed information, and calculating a difference value.
The objective function is defined by Equation 1 below.
[Equation 1]

L: objective function, w: image width, h: image height, D: degraded image, G: original image, Mi: matrix of image sizes with values of '0' or '1', Wi: weight delete delete The method according to claim 1, wherein the machine learning algorithm is applied to a video encoder that is operated as a compression standard of known H.262, HEVC, and the like. And a reconstructed image is used for prediction of an adjacent frame image by replacing one of the filters. The image reconstruction of claim 4, wherein the machine learning algorithm is applied to the image encoder to post-process an image reconstructed in a predetermined manner so that the reconstructed image is used for prediction of an adjacent frame image. Way. delete

Images