KR20210141303A - A method and an apparatus for extracting a noise of a compressed image and a method and an apparatus for reducing a noise of a compressed image - Google Patents

Abstract

Provided is a method for reducing noise of a compressed image. A first image is obtained by applying a convolution filter which sequentially performs down-convolution and up-convolution corresponding to the down-convolution to the compressed image of the original image. A second image obtained by subtracting the first image from the compressed image is obtained. Noise including high-frequency information and compression artifacts of the compressed image is obtained from the second image. A third image is obtained by removing compression artifacts by applying DNN for removing compression artifacts to the noise. And the compressed image is restored by summing the first image and the third image.

Description

A method and an apparatus for extracting a noise of a compressed image and a method and an apparatus for reducing a noise of a compressed image}

The present disclosure relates to a method and apparatus for extracting and reducing noise of a compressed image, and more specifically, the present disclosure relates to a method and apparatus for extracting and reducing noise of a compressed image, and extracting noise including compression artifacts generated during compression of an image based on artificial intelligence (AI), It relates to a method and apparatus for abatement.

The image is encoded by a codec conforming to a predetermined data compression standard, for example, a Moving Picture Expert Group (MPEG) standard, and then stored in a bitstream in the form of a recording medium or transmitted through a communication channel.

Compression artifacts may occur during image compression. Accordingly, there is a need to improve the quality of a compressed image by reducing such compression artifacts.

A method for extracting noise from a compressed image, a method for reducing noise from a compressed image, and an apparatus according to an embodiment separate structural information with a high probability of restoration and a high-frequency noise region with a high probability of damage from a compressed image based on AI Therefore, it is a technical task to improve the quality of a compressed image by reducing compression artifacts generated during image compression.

In order to solve the above technical problem, a method for extracting noise from a compressed image, proposed by the present disclosure, is a convolution filter that sequentially performs down-convolution on a compressed image of an original image and up-convolution corresponding to the down-convolution. obtaining a first image by applying obtaining a second image obtained by subtracting the first image from the compressed image; and obtaining noise including high frequency information and compression artifacts of the compressed image from the second image.

In order to solve the above technical problem, a method for reducing noise of a compressed image, proposed by the present disclosure, is a convolution of sequentially performing down-convolution on a compressed image of an original image and up-convolution corresponding to the down-convolution. obtaining a first image by applying a filter; obtaining a second image obtained by subtracting the first image from the compressed image; obtaining noise including high frequency information and compression artifacts of the compressed image from the second image; obtaining a third image by removing compression artifacts by applying DNN for removing compression artifacts to the noise; and restoring the compressed image by summing the first image and the third image.

In order to solve the above technical problem, an apparatus for extracting noise from a compressed image, which is proposed in the present disclosure, includes: a memory for storing one or more instructions; and a processor executing the one or more instructions stored in the memory, wherein the processor sequentially performs down-convolution and up-convolution corresponding to the down-convolution on the compressed image of the original image. By applying, a first image may be obtained, a second image obtained by subtracting the first image from the compressed image, and noise including high frequency information of the compressed image and compression artifacts may be obtained from the second image. .

In order to solve the above technical problem, an apparatus for reducing noise of a compressed image, which is proposed in the present disclosure, includes: a memory for storing one or more instructions; and a processor executing the one or more instructions stored in the memory, wherein the processor sequentially performs down-convolution and up-convolution corresponding to the down-convolution on the compressed image of the original image. obtaining a first image by applying, obtaining a second image obtained by subtracting the first image from the compressed image, and obtaining noise including high-frequency information and compression artifacts of the compressed image from the second image; A third image may be obtained by removing compression artifacts by applying DNN for removing compression artifacts to noise, and the compressed image may be reconstructed by summing the first image and the third image.

By using a convolution filter including down-convolution and up-convolution operations, and subtraction and summation operations, the compressed image is separated into an image containing low-frequency information and intermediate-frequency information and an image containing high-frequency information and noise containing compression artifacts. The quality of the compressed image is improved by extracting noise, inputting an image having noise including high-frequency information and compression artifacts into the DNN for compression artifact removal, removing compression artifacts, and resuming the image with low-frequency information and intermediate-frequency information. can be improved

In order to more fully understand the drawings cited herein, a brief description of each drawing is provided.
1 is a diagram for explaining a method of extracting and reducing noise of a compressed image according to an exemplary embodiment.
2 is an exemplary diagram illustrating down-convolution and up-convolution operations performed in a convolution filter.
3 is an exemplary diagram illustrating a DNN for removing compression artifacts.
4 is a block diagram illustrating a configuration of an apparatus for extracting noise from a compressed image according to an exemplary embodiment.
5 is a block diagram illustrating a configuration of an apparatus for reducing noise in a compressed image according to an exemplary embodiment.
6 is an exemplary diagram for explaining a method of training a convolution filter.
7 is an exemplary diagram for explaining a method of training a DNN for reducing AI artifacts.
8 is a flowchart illustrating a method of extracting noise from a compressed image according to an exemplary embodiment.
9 is a flowchart illustrating a method of reducing noise in a compressed image according to an exemplary embodiment.

Since the present disclosure can make various changes and can have various embodiments, specific embodiments are illustrated in the drawings and will be described in detail through the detailed description. However, this is not intended to limit the embodiments of the present disclosure, and it should be understood that the present disclosure includes all modifications, equivalents and substitutes included in the spirit and scope of various embodiments.

In describing the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the subject matter of the present disclosure, the detailed description thereof will be omitted. In addition, numbers (eg, first, second, etc.) used in the description process of the specification are only identifiers for distinguishing one component from other components.

In addition, in this specification, when a component is referred to as "connected" or "connected" with another component, the component may be directly connected or directly connected to the other component, but in particular It should be understood that, unless there is a description to the contrary, it may be connected or connected through another element in the middle.

In addition, in the present specification, components expressed as '~ unit (unit)', 'module', etc. are two or more components combined into one component, or two or more components for each more subdivided function. may be differentiated into In addition, each of the components to be described below may additionally perform some or all of the functions of other components in addition to the main functions that each component is responsible for, and some of the main functions of each of the components are different It goes without saying that it may be performed exclusively by the component.

In addition, in this specification, an 'image' or 'picture' may indicate a still image, a moving picture composed of a plurality of continuous still images (or frames), or a video.

Also, in the present specification, a 'deep neural network (DNN)' is a representative example of an artificial neural network model simulating a brain nerve, and is not limited to an artificial neural network model using a specific algorithm.

Also, in the present specification, a 'parameter' is a value used in a calculation process of each layer constituting a neural network, and may include, for example, a weight used when an input value is applied to a predetermined calculation expression. Also, the parameter may be expressed in a matrix form. A parameter is a value set as a result of training, and may be updated through separate training data if necessary.

In addition, in the present specification, the term 'convolution filter' refers to a trained filter used to obtain an image having only intermediate frequency and low frequency information of a compressed image by sequentially performing down-convolution and up-convolution on the compressed image 2 It is a type of DNN in that convolution operations are performed and trained, and “DNN for AI compression artifact removal” refers to a DNN used to remove compression artifacts from images having high frequency information and compression artifacts of compressed images. .

Also, in the present specification, 'filter setting information' includes the parameters described above as information related to a convolution filter. A convolution filter may be set using the filter setting information.

In addition, in the present specification, 'DNN configuration information' includes the aforementioned parameters as information related to elements constituting the DNN. A DNN for removing AI compression artifacts may be set using the DNN configuration information.

In addition, in this specification, 'original image' means an image subject to image compression, 'compressed image' means an image generated as a result of image compression of the original image, and 'first image' is It refers to an image to which a convolution filter in which up-convolution is sequentially performed is applied. In addition, 'second image' refers to an image obtained by subtracting the first image from the compressed image, and 'third image' refers to an image obtained by removing AI compression artifacts for the second image in the compression artifact removal process. , 'reconstructed image' means an image obtained by summing the first image and the third image.

In addition, in this specification, 'subtraction' means element-wise subtraction for subtracting pixel values for each pixel of an image, and 'summing' means element-wise subtraction for summing pixel values for each pixel of an image. wise sum).

In addition, in the present specification, 'compression artifact' means an error that occurs due to damage to edges and textures during image compression, and the compression artifacts generally include block artifacts, ringing artifacts, and the like.

1 is a diagram for explaining a method of extracting and reducing noise of a compressed image according to an exemplary embodiment.

Compressed images generated by image compression of the original image generate noise including compression artifacts generated during compression. Accordingly, it is necessary to effectively separate and extract such noise and effectively reduce compression artifacts included in the noise. .

The compressed image includes a region including structural information with high restoration potential (low frequency band information and intermediate frequency band information) and a high frequency noise region with high damage probability (high frequency band information and compression artifacts).

As shown in FIG. 1 , according to an embodiment of the present disclosure, the compressed image 105 is applied to the convolution filter 110 that sequentially performs down-convolution 115 and up-convolution 125 . A first image 135 is obtained, and a second image 145 is obtained by subtracting 120 of the first image 135 from the compressed image 105 . Then, the second image 145 is input to the DNN 130 for reducing artifacts to obtain a third image 155 , and the first image 135 and the third image 155 are summed 140 to obtain a reconstructed image. (165) is obtained.

The convolution filter 110 is a filter that sequentially performs down-convolution 115 and up-convolution 125 corresponding to down-convolution 125, and receives compressed image 105 as input and compressed image 105 ) is trained to acquire the first image 135 having a low frequency and an intermediate frequency. That is, as a result of applying the convolution filter 110 , a first image 135 having low frequency and intermediate frequency information of the compressed image 105 is obtained, and the first image 135 is subtracted from the compressed image 105 and compressed. A second image 145 having high frequency information of the image 105 and compression artifacts is obtained. The convolution operation of the convolution filter will be described later with reference to FIG. 2 , and the training method of the convolution filter will be described with reference to FIG. 6 .

The DNN 130 for reducing artifacts is trained to obtain a third image 155 with reduced compression artifacts by inputting the high frequency information of the compressed image 105 and the second image 145 having compression artifacts as inputs. That is, as a result of applying the DNN for reducing artifacts, a third image 155 with reduced compression artifacts is obtained, and the compression artifacts are reduced from the compressed image 105 by summing the first image 135 and the third image 155 . A restored image 165 is obtained. The structure of the DNN for reducing artifacts will be described later with reference to FIG. 3 , and the training method of the DNN for reducing artifacts will be described with reference to FIG. 7 .

The above-described method of extracting noise including compression artifacts and reducing compression artifacts may be performed several times. Also, it may be performed for each block of the image or may be performed several times for each block of the image. In addition, when it is necessary to remove compression artifacts after image compression is performed, the above-described method of extracting noise including compression artifacts and reducing compression artifacts may be applied at any stage.

If the compressed image is reduced based on AI without separating the compressed image into an image containing low-frequency and intermediate-frequency information and an image containing high-frequency information and compression artifacts, it is not possible to distinguish between the information of the compression artifact to be reduced and the information of the artifact to be maintained. Therefore, only compression artifacts cannot be separately reduced. However, as described above in FIG. 1 , the compressed image is separated into an image including low and intermediate frequency information and an image including compression artifacts and high frequency information related to compression artifacts using a trained convolution filter, and high frequency information and by removing compression artifacts using a DNN for reducing artifacts trained on an image including compression artifacts, compression artifacts generated during image compression that were not present in the original image can be reduced by concentration.

2 is an exemplary diagram illustrating down-convolution and up-convolution operations performed in a convolution filter according to the present invention.

As shown in FIG. 2 , the down convolution 115 is performed by applying the 3x3 down filter kernel 220 according to the stride to the 4x4 input feature map 210 through convolution to increase the size of the feature map. It is a convolution operation to obtain a reduced 2x2 output feature map 230, and the up convolution 125 is in a transposed relationship with the down convolution 115, and the 2x2 input feature map 240 with a reduced size of the feature map. ), a 4x4 output feature map 260 that increases the size of the feature map to its original size through convolution by applying a 3x3 up-filter kernel 250 corresponding to the down-filter kernel 220 according to the stride. Convolution operation to obtain.

An output feature map 230 having a reduced size compared to the input feature map 210 is obtained through down convolution 115, and input information corresponding to a high-frequency region is lost as much as pixels are deleted, but after that, up-convolution Through (125), the number of pixels is restored by increasing the size of the feature map again.

That is, a small-dimensional output feature map 230 in which high-frequency information is lost is obtained while reducing the input feature map 210 having large-dimensional information to a small dimension through down-convolution operation using a trained filter kernel, The original large-dimensional output feature map 260 is obtained while expanding the small-dimensional input feature map 240 to a large dimension through an up-convolution operation using a trained filter kernel corresponding to the down-convolution operation.

Accordingly, the first image 135 including low-frequency and intermediate-frequency information of the compressed image 105 input to the convolution filter is obtained.

According to an embodiment, the convolution filter may be a DNN including one convolutional layer performing down-convolution and one convolutional layer performing up-convolution.

Compressed images by using a trained convolution filter that sequentially performs a convolution operation that reduces the size and a transposed convolution that increases the size, rather than a fixed filter of the size reduction and enlargement method of the interpolation method. As a result of a subtraction operation for subtracting the convolution-filtered image from , it is possible to effectively separate an image in a noise region including high-frequency information and compression artifacts from the compressed image. Accordingly, compression artifacts can be effectively reduced by applying the separated noise region image to DNN for removing compression artifacts, which will be described later in FIG. 3 .

The convolution operations described with reference to FIG. 2 are only examples and are not limited thereto.

3 is an exemplary diagram illustrating a DNN for removing compression artifacts.

As shown in FIG. 3 , the second image 145 having high frequency information of the compressed image 105 and noise including compression artifacts is input to the first convolutional layer 310 . 3X3X4 displayed in the first convolution layer 310 shown in FIG. 3 exemplifies convolution processing on one input image using four filter kernels having a size of 3×3. As a result of the convolution process, four feature maps are generated by four filter kernels. Each feature map represents unique characteristics of the second image 145 . For example, each feature map may represent a vertical direction characteristic, a horizontal direction characteristic, an edge characteristic, and a characteristic indicated by a compression artifact of the second image 135 .

The feature maps generated as a result of the convolution process may be input to a batch normalization layer that calculates the average and standard deviation for each feature and normalizes them in batch units. The batch normalization method improves learning speed because it learns by normalizing in mini-batch units. Here, the batch means a portion of data.

The feature maps output from the first convolutional layer are input to the first batch normalization layer 310 and then input to the first activation layer 315 .

Also, as another example, batch normalization is an option, and feature maps output to the first convolutional layer may be directly input to the first activation layer 315 without batch normalization.

The first activation layer 315 may provide a non-linear characteristic to each feature map. The first activation layer 315 may include, but is not limited to, a sigmoid function, a Tanh function, a Rectified Linear Unit (ReLU) function, and the like.

Some samples of the feature map, which are the outputs of the first convolutional layer 305 or the first convolutional layer 305 and the first batch normalization layer 310, to give non-linear properties in the first activation layer 315 It means changing the value and outputting it. At this time, the change is performed by applying a non-linear characteristic.

The first activation layer 315 determines whether to transfer the sample values of the output feature maps to the second convolution layer 320 . For example, some sample values among the sample values of the feature maps are activated by the first activation layer 315 and transferred to the second convolution layer 320 , and some sample values are activated by the first activation layer 315 . It is deactivated and is not transmitted to the second convolutional layer 320 . Among the features of the second image 135 indicated by the feature maps, a feature corresponding to a compression artifact is inactivated, and a feature not corresponding to the compression artifact is emphasized by the first activation layer 315 .

The feature maps output from the first activation layer 315 are input to the second convolution layer 320 . 3X3X1 displayed in the second convolutional layer 320 shown in FIG. 3 exemplifies convolution processing to generate one output image using one filter kernel having a size of 3×3. The second convolutional layer 350 is a layer for outputting a final image and generates one output using one filter kernel. The second convolutional layer 320 is input to the second batch normalization layer 325 and then to the second activation layer 330 .

Also, as another example, batch normalization is an option, and feature maps output to the second convolution layer may be directly input to the second activation layer without batch normalization.

According to the example of the present disclosure, the second activation layer 330 may output the third image 155 from which the compression artifact included in the second image 145 is removed.

3 shows that the DNN 130 for reducing artifacts includes two convolutional layers 305 and 320 and two activation layers 315 and 330, and may optionally include two batch normalization layers 310 and 325. Although illustrated as being there, this is only an example, and the number of convolutional layers, activation layers, and batch normalization layers may be variously changed according to implementations. Also, according to an embodiment, the DNN 130 for reducing artifacts may be implemented through a recurrent neural network (RNN). In this case, it means changing the CNN structure of the DNN 130 for reducing artifacts according to the example of the present disclosure to the RNN structure.

As will be described later with reference to FIG. 7 , the DNN for removing compression artifacts takes as input a second image having noise including high frequency information and compression artifacts of the compressed image, and maintains the high frequency information included in the original image as much as possible while maintaining the original image. Since it is trained to obtain a third image in which compression artifacts not included in the image have been removed, compression artifacts are effectively removed. Thereafter, by summing the first image and the third image having low and intermediate frequency information of the compressed image, only compression artifacts of the compressed image are reduced to obtain a reconstructed image with improved quality to be close to the original image.

The convolution process described in FIG. 3 is only an example, and is not limited thereto.

4 is a block diagram illustrating a configuration of an apparatus for extracting noise from a compressed image according to an exemplary embodiment.

Referring to FIG. 4 , an apparatus 400 for extracting noise from a compressed image according to an embodiment includes a convolution filter 410 , a subtraction unit 420 , and a filter setting unit 430 .

The convolution filter 410 , the subtraction unit 420 , and the filter setting unit 430 may be implemented through one processor. In this case, the convolution filter 410 , the subtraction unit 420 , and the filter setting unit 430 may be implemented as a dedicated processor, and may include an application processor (AP), a central processing unit (CPU), or a graphic processing unit (GPU). ) may be implemented through a combination of a general-purpose processor and S/W. In addition, in the case of a dedicated processor, a memory for implementing an embodiment of the present disclosure or a memory processing unit for using an external memory may be included.

The convolution filter 410 , the subtraction unit 420 , and the filter setting unit 430 may include a plurality of processors. In this case, it may be implemented as a combination of dedicated processors, or may be implemented through a combination of S/W with a plurality of general-purpose processors such as an AP, CPU, or GPU. In one embodiment, the convolution filter 410 is implemented by a first processor, the subtraction unit 420 is implemented by a second processor different from the first processor, and the filter setting unit 430 is implemented by the first processor and the second processor. It may be implemented with a third processor different from the processor.

The filter setting unit 430 may provide trained filter setting information corresponding to the compressed image 105 to the convolution filter 410 based on a predetermined criterion.

The filter setting unit 430 may store filter setting information to obtain the first image 135 corresponding to the compressed image 105 .

The filter setting information is trained to acquire the first image 135 having low-frequency and intermediate-frequency information of the compressed image 105 .

The filter setting information may include the number of filter kernels, the size of the filter kernel, parameters of the filter kernel, and the like.

The convolution filter 410 sequentially performs down-convolution and up-convolution according to the filter setting information provided from the filter setting unit 430 .

The convolution filter 410 outputs the first image 135 by inputting the compressed image 105 using the filter setting information provided from the filter setting unit 430 .

The subtractor 420 subtracts the output first image by applying the convolution filter 410 from the compressed image 105 .

As a result of the subtraction, the subtraction unit 420 outputs the second image 145 having noise including high frequency information of the compressed image and compression artifacts.

5 is a block diagram illustrating a configuration of an apparatus for reducing noise in a compressed image according to an exemplary embodiment.

Referring to FIG. 5 , an apparatus 500 for reducing noise in a compressed image according to an exemplary embodiment includes a noise extractor 510 and an image restoration unit 550 . The noise extraction unit 510 includes a convolution filter 520 , a subtraction unit 530 , and a filter setting unit 540 , and the image restoration unit 550 includes a DNN 560 for reducing artifacts, and a summing unit 570 . ), and an AI setting unit 580 .

Although the noise extractor 510 and the image restorer 550 are illustrated as separate devices in FIG. 5 , the noise extractor 510 and the image restorer 550 may be implemented through one processor. In this case, the noise extraction unit 510 and the image restoration unit 550 may be implemented as a dedicated processor, and may be implemented with a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphic processing unit (GPU) and an S. It can also be implemented through the combination of /W. In addition, in the case of a dedicated processor, a memory for implementing an embodiment of the present disclosure or a memory processing unit for using an external memory may be included.

The noise extraction unit 510 and the image restoration unit 550 may be configured with a plurality of processors. In this case, it may be implemented as a combination of dedicated processors, or may be implemented through a combination of S/W with a plurality of general-purpose processors such as an AP, CPU, or GPU. In one embodiment, the convolution filter 520 and the subtraction unit 530 are implemented with a first processor, and the DNN for reducing artifacts 560 and the summing unit 570 are implemented with a second processor different from the first processor. , the filter setting unit 540 and the AI setting unit 580 may be implemented as a third processor different from the first processor and the second processor.

The filter setting unit 540 may provide trained filter setting information corresponding to the compressed image 105 to the convolution filter 520 based on a predetermined criterion.

The filter setting unit 540 may store filter setting information to obtain the first image 135 corresponding to the compressed image 105 .

The filter setting information is trained to acquire the first image 135 having low-frequency and intermediate-frequency information of the compressed image 105 .

The filter setting information may include the number of filter kernels, the size of the filter kernel, parameters of the filter kernel, and the like.

The convolution filter 520 sequentially performs down-convolution and up-convolution according to filter setting information provided from the filter setting unit 540 .

The convolution filter 520 uses the filter setting information provided from the filter setting unit 540 and outputs the first image 135 by inputting the compressed image 105 as an input.

The subtractor 530 subtracts the output first image by applying the convolution filter 520 from the compressed image 105 .

As a result of the subtraction, the subtractor 530 outputs the second image 145 having noise including high frequency information of the compressed image and compression artifacts.

The AI setting unit 580 provides the trained DNN setting information corresponding to the second image 145 to the DNN for reducing artifacts according to a predetermined criterion.

The AI setting unit 580 may store DNN setting information to obtain the third image 155 in which the compression artifact included in the second image 145 is reduced.

The DNN setting information is trained to obtain a third image 155 from which compression artifacts are removed from the second image 145 having high frequency information and compression artifacts of the compressed image 105 .

The DNN configuration information may include information on at least one of the number of convolution layers included in the DNN 560 for reducing artifacts, the number of filter kernels for each convolutional layer, and parameters of each filter kernel.

The DNN 560 for reducing artifacts receives the second image 145 as an input using the DNN setting information provided from the AI setting unit 580 to remove the compression artifacts included in the second image 145 ( 155) is output.

The summing unit 570 adds the first image output through the convolution filter 520 of the noise extractor 510 and the third image output through the DNN 560 for reducing artifacts to obtain a reconstructed image 165 . print out Since the first image has low frequency and intermediate frequency information of the compressed image and the third image has high frequency information of the compressed image, the summed reconstructed image is an image from which compression artifacts are removed from the compressed image. Accordingly, the apparatus 500 for reducing noise of a compressed image according to an embodiment obtains a reconstructed image 165 from which compression artifacts generated when an image included in the compressed image 105 is compressed.

6 is an exemplary diagram for explaining a method of training a convolution filter.

In FIG. 6 , an original training image 605 is an image subjected to image compression 610 , a compression training image 615 is an image compressed by an original training image 605 , and a first training image 635 . is an image to which a convolution filter 620 that sequentially performs down-convolution and up-convolution is applied to the compression training image 615 .

The original training image 605 includes a still image or a moving image composed of a plurality of frames. In an embodiment, the original training image 605 may include a still image or a luminance image extracted from a moving image including a plurality of frames. Also, according to an embodiment, the original training image 605 may include a still image or a patch image extracted from a moving picture including a plurality of frames. When the original training image 605 consists of a plurality of frames, the compressed training image 615 , the first training image 635 , and the low-frequency and intermediate-frequency training image 630 also include a plurality of frames. When a plurality of frames of the original training image 605 are compressed 610 , a plurality of frames of the compressed training image 615 are obtained, and a plurality of frames of the first training image 635 are obtained through the convolution filter 620 . this can be obtained.

For image compression of the original training image 605, any one of MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, and AV1 codecs may be used.

Referring to FIG. 6 , separately from outputting the first training image 635 through the convolution filter 620 by inputting the compressed training image 615 as an input, a legacy frequency band filter ( 625), the filtered low-frequency and intermediate-frequency training images 630 are obtained. Here, the cutoff frequency of the legacy frequency band filter 625 may vary according to characteristics of the compressed training image. Specifically, the use of the convolution filter is to separate low-frequency and intermediate-frequency information including structural information with a high probability of reconstructing a compressed image from high-frequency information and artifacts of high-frequency noise with a high probability of damage, so the cutoff frequency of the legacy frequency band filter may vary depending on the compression training image.

The low-frequency and intermediate-frequency training image 630 refers to an image including only low-frequency and intermediate-frequency information of the compressed training image through the legacy frequency band filter 625 . For training of the convolution filter 620 , a low-frequency and intermediate-frequency training image 630 including only low-frequency and intermediate-frequency information of the compressed training image is acquired.

Before training, the convolution filter 620 may be set with predetermined filter setting information. As training progresses, filter loss information 640 may be determined.

The filter loss information 640 may be determined based on a comparison result between the first training image 635 and the low-frequency and intermediate-frequency training images 630 .

The filter loss information 640 includes an L1-norm value, an L2-norm value, a structural similarity (SSIM) value, and a PSNR-HVS (Peak) value for the difference between the first training image 635 and the low-frequency and intermediate-frequency training image 630 . It may include at least one of a Signal-To-Noise Ratio-Human Vision System) value, a Multiscale SSIM (MS-SSIM) value, a Variance Inflation Factor (VIF) value, and a Video Multimethod Assessment Fusion (VMAF) value. The filter loss information 640 indicates how similar the first training image 635 is to the low-frequency and intermediate-frequency training images 630 . As the filter loss information 640 is smaller, the first training image 635 is more similar to the low-frequency and intermediate-frequency training images 630 .

The convolution filter 620 may update the parameter so that the filter loss information 640 is reduced or minimized.

The final loss information of the convolution filter 620 may be determined as in Equation 1 below.

[Equation 1]

LossCF = a*Filter loss information

In Equation 1, LossCF represents final loss information to be reduced or minimized for training of the convolution filter 620, and a may correspond to a predetermined weight.

That is, the convolution filter 620 updates the parameters in a direction in which the LossCF of Equation 1 is decreased. When the parameters of the convolution filter 620 are updated according to the LossCF derived in the training process, the first training image 635 obtained based on the updated parameters is different from the first training image 635 in the previous training process. will lose

7 is an exemplary diagram for explaining a method of training a DNN for reducing AI artifacts.

In FIG. 7 , the original training image 605 is an image to be compared with the reconstructed training image 730 , and the compressed training image 615 is a compressed image of the original training image 605 . The first training image 635 is an image in which the convolution filter 620 is applied to the compressed training image 615 , which has been trained through the training process in FIG. 6 . The second training image 710 is an image obtained by subtracting (705) the first training image 635 from the compressed training image 615, and the third training image 720 is the second training image 710 as an input. It is an image output through the DNN 715 for AI artifact reduction, and the training image 730 is an image obtained by summing 725 of the first training image 635 and the third training image 720 .

Here, the first training image 635 is a convolutional filter 620 that has been trained to the compressed training image 615 is applied. This is that the convolution filter 620 is trained to effectively separate the first image 135 having low-frequency and intermediate-frequency information and the second image 145 having high-frequency information and compression artifacts, and the AI artifact reduction DNN ( This is because 715 is trained to effectively remove the compression artifact of the second image 145 . That is, the convolution filter 620 and the DNN 715 for AI artifact reduction must be trained independently.

The original training image 605 includes a still image or a moving image composed of a plurality of frames. In an embodiment, the original training image 605 may include a still image or a luminance image extracted from a moving image including a plurality of frames. Also, according to an embodiment, the original training image 605 may include a still image or a patch image extracted from a moving picture including a plurality of frames. When the original training image 605 consists of a plurality of frames, the compressed training image 615, the first training image 635, the second training image 710, and the third training image 720 are also divided into a plurality of frames. is composed When a plurality of frames of the original training image 605 are compressed 610, a plurality of frames of a compressed training image 615 are obtained, and the first training image 635 is A plurality of frames are obtained, a plurality of frames of the second training image 710 are obtained through the subtraction 705 of the first training image 635 from the compressed training image 615, and a DNN 715 for reducing AI artifacts. A plurality of frames of the third training image 720 are acquired through this is obtained

The DNN 715 for reducing AI artifacts before training may be set with predetermined DNN configuration information. As training progresses, abatement loss information 740 may be determined.

The reduced loss information 740 may be determined based on a comparison result between the original training image 605 and the restored training image 730 .

The reduced loss information 740 includes an L1-norm value, an L2-norm value, a structural similarity (SSIM) value, and a peak signal-to- (PSNR-HVS) value for the difference between the original training image 605 and the reconstructed training image 730 . It may include at least one of a Noise Ratio-Human Vision System) value, a Multiscale SSIM (MS-SSIM) value, a Variance Inflation Factor (VIF) value, and a Video Multimethod Assessment Fusion (VMAF) value. The reduced loss information 740 indicates how similar the reconstructed training image 730 is to the original training image 605 . As the reduced loss information 740 is smaller, the restored training image 730 is more similar to the original training image 605 . That is, a reconstructed training image with reduced compression artifacts generated by image compression of the original training image is obtained.

The AI artifact reduction DNN 715 may update parameters so that the reduction loss information 740 is reduced or minimized.

The final loss information of the DNN 715 for reducing AI artifacts may be determined as in Equation 2 below.

[Equation 2]

LossAR = b*reduced loss information

In Equation 2, LossAR represents final loss information to be reduced or minimized for training of the DNN 715 for reducing AI artifacts, and b may correspond to a predetermined weight.

That is, the DNN 715 for reducing AI artifacts updates the parameters in a direction in which the LossAR of Equation 2 is reduced. When the parameters of the DNN 715 for reducing AI artifacts are updated according to the LossAR derived in the training process, the third training image 720 obtained based on the updated parameters is the third training image 720 in the previous training process. will be different from

8 is a flowchart illustrating a method of extracting noise from a compressed image according to an exemplary embodiment.

In step S810, the apparatus 400 for extracting noise of the compressed image applies a convolution filter that sequentially performs down-convolution and up-convolution corresponding to the down-convolution to the compressed image of the original image by applying a convolution filter to the compressed image of the original image. to acquire

According to an embodiment, the first image may include intermediate frequency information and low frequency information of the compressed image.

According to an embodiment, the convolution filter may be trained to obtain a first image including intermediate frequency information and low frequency information of the compressed image by receiving the compressed image as an input.

According to an embodiment, the down convolution may be a convolution operation trained to reduce the size of a compressed image.

According to an embodiment, the up-convolution may be a convolution operation trained to increase the down-convolved compressed image to the original size of the compressed image.

According to an embodiment, the down convolution and the up convolution may be transposed.

In operation S830, the apparatus 400 for extracting noise from the compressed image obtains a second image obtained by subtracting the first image from the compressed image.

In operation S850, the apparatus 400 for extracting noise of the compressed image obtains noise including high frequency information and compression artifacts of the compressed image from the second image.

9 is a flowchart illustrating a method of reducing noise in a compressed image according to an exemplary embodiment.

In step S910 , the apparatus 500 for reducing noise of the compressed image applies a convolution filter that sequentially performs down-convolution and up-convolution corresponding to the down-convolution to the compressed image of the original image by applying a convolution filter to the compressed image of the original image. to acquire

According to an embodiment, the first image may include intermediate frequency information and low frequency information of the compressed image.

According to an embodiment, the convolution filter may be trained to obtain a first image including intermediate frequency information and low frequency information of the compressed image by receiving the compressed image as an input.

According to an embodiment, the down convolution may be a convolution operation trained to reduce the size of a compressed image.

According to an embodiment, the up-convolution may be a convolution operation trained to increase the down-convolved compressed image to the original size of the compressed image.

According to an embodiment, the down convolution and the up convolution may be transposed.

In operation S930, the apparatus 500 for reducing noise of a compressed image obtains a second image obtained by subtracting the first image from the compressed image.

In operation S950 , the apparatus 500 for reducing noise of the compressed image obtains noise including high frequency information and compression artifacts of the compressed image from the second image.

In step S970 , the apparatus 500 for reducing the noise of the compressed image obtains the third image by removing the compression artifact by applying the DNN for removing the compression artifact to the noise.

According to an embodiment, the DNN for removing compression artifacts may be trained to reduce compression artifacts by receiving high frequency information and compression artifacts of the compressed image as inputs.

According to an embodiment, the DNN for compression artifact removal may include a convolution layer and an activation layer.

According to an embodiment, the DNN for compression artifact removal may further include a batch normalization layer.

In operation S990 , the apparatus 500 for reducing noise of the compressed image restores the compressed image by adding the first image and the third image.

Meanwhile, the above-described embodiments of the present disclosure can be written as a program or instruction that can be executed on a computer, and the written program or instruction can be stored in a medium.

The medium may continuously store a computer-executable program or instructions, or may be a temporary storage for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or several hardware combined, it is not limited to a medium directly connected to any computer system, and may exist distributed on a network. Examples of the medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floppy disk, and those configured to store program instructions, including ROM, RAM, flash memory, and the like. In addition, examples of other media may include recording media or storage media managed by an app store that distributes applications, sites that supply or distribute various other software, and servers.

Meanwhile, the above-described DNN-related model may be implemented as a software module. When implemented as a software module (eg, a program module including instructions), the DNN model may be stored in a computer-readable recording medium.

In addition, the DNN model may be integrated in the form of a hardware chip to become a part of the aforementioned AI decoding apparatus 200 or AI encoding apparatus 600 . For example, a DNN model may be built in the form of a dedicated hardware chip for artificial intelligence, or as part of an existing general-purpose processor (e.g., CPU or application processor) or graphics-only processor (e.g., GPU). it might be

In addition, the DNN model may be provided in the form of downloadable software. The computer program product may include a product (eg, a downloadable application) in the form of a software program distributed electronically through a manufacturer or an electronic market. For electronic distribution, at least a portion of the software program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a server of a manufacturer or an electronic market, or a storage medium of a relay server.

Above, the technical idea of the present disclosure has been described in detail with reference to preferred embodiments, but the technical idea of the present disclosure is not limited to the above embodiments, and those of ordinary skill in the art within the scope of the technical spirit of the present disclosure Various modifications and changes are possible by the person.

Claims (17)

obtaining a first image by applying a convolution filter that sequentially performs down-convolution and up-convolution corresponding to the down-convolution to the compressed image of the original image;
obtaining a second image obtained by subtracting the first image from the compressed image; and
and obtaining noise including high frequency information and compression artifacts of the compressed image from the second image. According to claim 1,
The method of extracting noise from a compressed image, wherein the first image includes intermediate frequency information and low frequency information of the compressed image. According to claim 1,
The method for extracting noise from a compressed image, wherein the convolution filter is trained to obtain the first image including intermediate frequency information and low frequency information of the compressed image by receiving the compressed image as an input. 4. The method of claim 3,
The down convolution is a convolution operation trained to reduce the size of the compressed image. 4. The method of claim 3,
The up-convolution is a convolution operation trained to increase the down-convolved compressed image to the original size of the compressed image. According to claim 1,
The down-convolution and the up-convolution are transposed, the method of extracting noise from a compressed image. obtaining a first image by applying a convolution filter that sequentially performs down-convolution and up-convolution corresponding to the down-convolution to the compressed image of the original image;
obtaining a second image obtained by subtracting the first image from the compressed image;
obtaining noise including high frequency information and compression artifacts of the compressed image from the second image;
obtaining a third image by removing compression artifacts by applying DNN for removing compression artifacts to the noise; and
and restoring the compressed image by summing the first image and the third image. 8. The method of claim 7,
The method for reducing noise in a compressed image, wherein the DNN for removing compression artifacts is trained to reduce the compression artifacts by inputting high-frequency information and compression artifacts of the compressed image as inputs. 8. The method of claim 7,
The method for reducing noise in a compressed image, wherein the DNN for removing compression artifacts includes a convolution layer and an activation layer. 10. The method of claim 9,
The DNN for removing compression artifacts further comprises a batch normalization layer. 8. The method of claim 7,
The method of reducing noise of a compressed image, wherein the first image includes intermediate frequency information and low frequency information of the compressed image. 8. The method of claim 7,
The method for reducing noise in a compressed image, wherein the convolution filter is trained to obtain the first image including intermediate frequency information and low frequency information of the compressed image. 8. The method of claim 7,
The down convolution is a convolution operation trained to reduce the size of the compressed image. 8. The method of claim 7,
The up-convolution is a convolution operation trained to increase the down-convolved compressed image to the original size of the compressed image. 8. The method of claim 7,
The down-convolution and the up-convolution are transposed, the method of reducing noise in a compressed image. a memory storing one or more instructions; and
a processor executing the one or more instructions stored in the memory;
The processor is
A first image is obtained by applying a convolution filter that sequentially performs down-convolution and up-convolution corresponding to the down-convolution to the compressed image of the original image,
obtaining a second image obtained by subtracting the first image from the compressed image,
The apparatus for extracting noise from a compressed image, characterized in that the noise including high frequency information and compression artifacts of the compressed image is obtained from the second image. a memory storing one or more instructions; and
a processor executing the one or more instructions stored in the memory;
The processor is
A first image is obtained by applying a convolution filter that sequentially performs down-convolution and up-convolution corresponding to the down-convolution to the compressed image of the original image,
obtaining a second image obtained by subtracting the first image from the compressed image,
obtaining noise including high-frequency information and compression artifacts of the compressed image from the second image;
A third image is obtained by removing compression artifacts by applying DNN for removing compression artifacts to the noise;
The apparatus for reducing noise in a compressed image, characterized in that the compressed image is restored by adding the first image and the third image.

Images