KR102467092B1 - Super-resolution image processing method and system robust coding noise using multiple neural networks - Google Patents

Abstract

A super resolution image processing method robust against coding noise using a plurality of neural networks is disclosed. The super-resolution image processing method that is robust to coding noise using a plurality of neural networks comprises the steps of: receiving, by a processor, a compressed low-resolution image; decoding the compressed low-resolution image by the processor; generating, by the processor, a noise-removed low-resolution image by removing noise from the decoded low-resolution image using a first neural network; when the difference between the noise level of the decoded low-resolution image and the noise level of the denoised low-resolution image is smaller than a certain value, converting, by the processor, the decoded low-resolution image into a decoded high-resolution image using a second neural network, and when the difference is greater than the arbitrary value, converting, by the processor, the low-resolution image from which the noise is removed into the decoded high-resolution image using the second neural network; encoding the decoded high-resolution image by the processor; and transmitting, by the processor, the encoded high-resolution image.

Description

Super-resolution image processing method and system robust coding noise using multiple neural networks

The present invention relates to a super-resolution image processing method and system robust against coding noise using a plurality of neural networks, and more particularly, to a super-resolution image that can be generated according to the level of coding noise generated in the process of compressing a low-resolution image. A super-resolution image processing method and system robust against coding noise using a plurality of neural networks.

Image upscaling, or image super-resolution, refers to a technique for generating a high resolution image from a low resolution image.

The image upscaling operation may also be performed on a low-resolution image obtained by decoding a compressed low-resolution image. Noise can be created in the process of compressing an image. The noise is defined as coding noise. When an image upscaling operation is performed on a low-resolution image from which the coding noise has not been removed, the coding noise may become an artifact. That is, coding noise may be noticeable in the converted high-resolution image.

Therefore, when an image upscaling operation is performed on a low-resolution image including coding noise, a new method capable of removing coding noise and generating a high-resolution image is required. At this time, a method for not removing the texture included in the low-resolution image is required.

Korean Registered Patent Publication No. 10-1996730 (2019.06.28.)

A technical problem to be achieved by the present invention is to provide a super-resolution image processing method and system robust against coding noise by using a plurality of neural networks capable of generating high-resolution images according to the level of coding noise.

A super-resolution image processing method robust against coding noise using a plurality of neural networks according to an embodiment of the present invention includes the steps of: receiving, by a processor, a compressed low-resolution image; decoding, by the processor, the compressed low-resolution image; Generating a noise-removed low-resolution image by removing noise from the decoded low-resolution image using a first neural network, wherein the processor determines the noise level of the decoded low-resolution image and the noise level of the noise-removed low-resolution image. converting the decoded low-resolution image or the noise-removed low-resolution image into a decoded high-resolution image using a second neural network, the processor encoding the decoded high-resolution image, and the processor performing the encoding Transmitting the high-resolution image.

When the difference between the noise level of the decoded low-resolution image and the noise level of the noise-removed low-resolution image is smaller than a certain value, the processor converts the decoded low-resolution image into the decoded high-resolution image using the second neural network. convert to

When the difference is greater than the predetermined value, the processor converts the noise-removed low-resolution image into the decoded high-resolution image using the second neural network.

The noise level of the decoded low-resolution image is generated using a third neural network.

The noise level of the noise-removed low-resolution image is generated using a fourth neural network.

According to an embodiment, the super-resolution image processing method robust against coding noise using the plurality of neural networks includes, by the processor, an image similarity between the noise-removed low-resolution image and the lossless low-resolution training image as a first score Calculating, calculating, by the processor, an image similarity between the decoded high-resolution image and the high-resolution training image as a second score; 1 Further training of the neural network may be included.

The first score is lower as the low resolution image from which the noise is removed is similar to the lossless low resolution training image.

The second score is lower as the decoded high-resolution image and the high-resolution training image are similar.

A super-resolution image processing system robust against coding noise using a plurality of neural networks according to an embodiment of the present invention includes an image processing device.

The image processing device includes a processor that executes commands, and a memory that stores the commands.

The instructions generate a noise-removed low-resolution image by removing noise from the low-resolution image using a first neural network, and the low-resolution image, or The low-resolution image from which the noise has been removed is converted into a high-resolution image using a second neural network.

When a difference between the noise level of the low-resolution image and the noise level of the noise-removed low-resolution image is smaller than a certain value, the instructions convert the low-resolution image into the high-resolution image using the second neural network.

When the difference is greater than the arbitrary value, the instructions are implemented to convert the noise-removed low-resolution image into the high-resolution image using the second neural network.

The noise level of the low-resolution image is generated using a third neural network.

The noise level of the noise-removed low-resolution image is generated using a fourth neural network.

The instructions calculate image similarity between the noise-removed low-resolution image and the lossless low-resolution training image as a first score, calculate image similarity between the high-resolution image and the high-resolution training image as a second score, and When a score of 1 is greater than the second score, the first neural network may be trained more than the second neural network.

The first score is lower as the low-resolution image from which the noise is removed is similar to the lossless low-resolution training image, and the second score is lower as the high-resolution image and the high-resolution training image are similar.

A super-resolution image processing method and system robust against coding noise using a plurality of neural networks according to an embodiment of the present invention selectively converts either a low-resolution image or a low-resolution image from which coding noise is removed into a high-resolution image according to a coding noise level. This has the effect of preventing the texture from being removed as well.

A detailed description of each drawing is provided in order to more fully understand the drawings cited in the detailed description of the present invention.
1 shows a block diagram of a super-resolution image processing system using a neural network according to an embodiment of the present invention.
FIG. 2 shows a block diagram of a network for explaining a training operation of the neural network module shown in FIG. 1 .
Figure 3 shows a detailed block diagram of the generator network shown in Figure 2;
4 shows images for explaining the effect of the super-resolution image processing method using a neural network according to an embodiment of the present invention.
5 is a flowchart of a super-resolution image processing method using a neural network according to an embodiment of the present invention.
6 is a block diagram of a network for explaining a training operation of a neural network module according to another embodiment of the present invention.
7 is a flowchart of a super-resolution image processing method robust against coding noise using a plurality of neural networks according to another embodiment of the present invention.

Specific structural or functional descriptions of the embodiments according to the concept of the present invention disclosed in this specification are only illustrated for the purpose of explaining the embodiments according to the concept of the present invention, and the embodiments according to the concept of the present invention It can be embodied in various forms and is not limited to the embodiments described herein.

Embodiments according to the concept of the present invention can apply various changes and can have various forms, so the embodiments are illustrated in the drawings and described in detail in this specification. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosure forms, and includes all changes, equivalents, or substitutes included in the spirit and technical scope of the present invention.

Terms such as first or second may be used to describe various components, but the components should not be limited by the terms. The above terms are only for the purpose of distinguishing one component from another component, e.g., without departing from the scope of rights according to the concept of the present invention, a first component may be termed a second component, and similarly The second component may also be referred to as the first component.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle. Other expressions describing the relationship between elements, such as "between" and "directly between" or "adjacent to" and "directly adjacent to", etc., should be interpreted similarly.

Terms used in this specification are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. As used herein, “includes.” or "to have." is intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, but is intended to indicate that one or more other features or numbers, steps, operations, components, parts, or combinations thereof are present. It should be understood that it does not preclude the possibility of existence or addition of one or the other.

Unless defined otherwise, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this specification, it should not be interpreted in an ideal or excessively formal meaning. don't

Hereinafter, the present invention will be described in detail by describing preferred embodiments of the present invention with reference to the accompanying drawings.

1 shows a block diagram of a super-resolution image processing system using a neural network according to an embodiment of the present invention.

Referring to FIG. 1 , a super-resolution image processing system 100 using a neural network means a system that converts a low-resolution image transmitted by a user into a high-resolution image and transmits the converted high-resolution image to the user.

A super resolution image processing system 100 using a neural network includes an image processing device 10 and a computing device 20 . The image processing device 10 and the computing device 20 communicate with each other through the network 101 .

The image processing device 10 may be called various terms such as a computing device or a server. The image processing device 10 receives the compressed low-resolution image CLR from the computing device 20, converts the compressed low-resolution image CLR into an encoded high-resolution image ESR, and outputs the compressed low-resolution image CLR to the computing device 20. . A conversion operation from a compressed low-resolution image (CLR) to an encoded high-resolution image (ESR) may be defined as an upscaling operation. Detailed operations of the image processing device 10 will be described later.

Computing device 20 refers to an electronic device such as a notebook, tablet PC, or desktop. According to embodiments, the computing device 20 may be referred to as a client. Computing device 20 is used to create a high-resolution image from a low-resolution image. The computing device 20 compresses the low-resolution image under a user's control to obtain a high-resolution image from the low-resolution image and transmits the compressed low-resolution image CLR to the image processing device 10 . Noise can be created in the process of compressing low-resolution images. The noise is defined as coding noise. In the process of converting a low-resolution image into a high-resolution image, the coding noise must be removed. If the low-resolution image is converted into a high-resolution image without removing coding noise, the quality of the high-resolution image will be low. That is, coding noise can become an artifact. In the present invention, methods for removing coding noise included in a low-resolution image in a process of up-scaling a compressed low-resolution image to a high-resolution image are proposed.

The image processing device 10 includes a processor 11 and a memory 13 . The processor 11 executes instructions related to super-resolution image processing methods using a neural network. The memory 13 contains the instructions. The memory 13 includes a decoding module 15 , a neural network module 17 , and an encoding module 19 . The decoding module 15, the neural network module 17, and the encoding module 19 refer to instructions executed by the processor 11. Hereinafter, operations of the decoding module 15 , the neural network module 17 , and the encoding module 19 may be understood to be performed by the processor 11 .

The decoding module 15 decodes the compressed low resolution image CLR. That is, the decoding module 15 receives the compressed low-resolution image CLR and outputs the decoded low-resolution image LR.

The neural network module 17 converts the decoded low-resolution image LR into a decoded high-resolution image SR using a pair of neural networks.

FIG. 2 shows a block diagram of a network for explaining a training operation of the neural network module shown in FIG. 1 .

Referring to FIGS. 1 and 2 , a pair of neural networks 300 and 400 must first be trained to convert a decoded low-resolution image LR into a decoded high-resolution image SR. The operations of network 200 are performed by neural network module 17 .

The network 200 shown in FIG. 2 represents a generative adversarial network (GAN). The network 200 includes a generator network 500 and a discriminator network 550 .

Generator network 500 is not trained to minimize the loss function of generator network 500, but rather to trick discriminator network 500. This allows the network 200 to learn in an unsupervised manner.

In the first step the generator network 500 is trained, and in the second step the discriminator network 550 is trained. The first step and the second step are repeatedly performed.

The generator network 500 is trained to receive a decoded low-resolution image LR containing coding noise and output a decoded high-resolution image SR.

The discriminator network 550 receives the high resolution training image HR and the decoded high resolution image SR output from the generator network 500, and calculates the distribution of the high resolution training image HR and the decoded high resolution image SR ) outputs the Wasserstein loss (EM), which is the difference in the distribution. Generator network 500 and discriminator network 550 can be trained according to the Wasserstein loss (EM). That is, parameters (weights) of the generator network 500 and the discriminator network 550 may be adjusted. The high resolution training image HR and the high resolution image SR may have the same resolution. The discriminator network 550 includes convolutional layers and activation layers.

Generator network 500 includes a pair of neural networks 300 and 400 . A pair of neural networks 300 and 400 are sequentially connected.

Figure 3 shows a detailed block diagram of the generator network shown in Figure 2;

Referring to FIGS. 1 to 3 , the first neural network 300 receives the decoded low-resolution image LR including coding noise and outputs a noise-removed low-resolution image DLR. The first neural network 300 is used to remove coding noise from the low-resolution image LR. The first neural network 300 may be referred to as a coding noise reduction network.

The first neural network 300 receives the decoded low-resolution image LR including coding noise and includes a plurality of layers 310-1 to 310-N; N to output a noise-removed low-resolution image DLR. is a natural number), a shuffle layer 320, and an adder 330. The plurality of layers 310-1 to 310-N are convolution layers or activation layers. The activation layer may be a Rectified Linear Unit (ReLU) activation function. Some of the plurality of layers 310-1 to 310-N form a residual block. Depending on embodiments, the number of the plurality of layers 310-1 to 310-N may vary.

According to an embodiment, the first neural network 300 converts the low-resolution image LR expressed in the form of (H, W, C) into the form of (H / 2, W / 2, C * 2 * 2) Space-to-depth (Space to depth) layer 305 may be further included. The space to depth layer 305 is used to improve the processing speed of the first neural network 300 . The H represents the height of the image, the W represents the width of the image, and the C represents the channel.

The shuffle layer 320 shuffles tensors including a plurality of channels to generate an image including one channel.

The adder 330 adds the image output from the shuffle layer 320 and the decoded low-resolution image LR, and outputs a noise-removed low-resolution image DLR.

The first loss function 340 calculates a loss function by receiving the noise-removed low-resolution image DLR and the lossless low-resolution training image LLR. The loss function may be calculated as a mean square error of a noise-removed low-resolution image (DLR) and a lossless low-resolution training image (LLR). The lossless low-resolution training image (LLR) means an image whose resolution is reduced from the high-resolution training image (HR). A lossless low-resolution training image (LLR) and a high-resolution training image (HR) are images used for neural network training.

The second neural network 400 receives the noise-removed low-resolution image DLR and outputs the decoded high-resolution image SR. The second neural network 400 is used to convert a low resolution image DLR into a high resolution image SR. The second neural network 400 may be referred to as a video super resolution network.

The second neural network 400 receives the noise-removed low-resolution image DLR and outputs the decoded high-resolution image SR through a connection layer 405 and a plurality of layers 410-1 to 410-M. ; M is a natural number), a shuffle layer 420, an upscaling layer 425, and an adder 430.

The connection layer 405 concatenates the high resolution image PSR of the previous frame output through the second neural network 400 and the noise-removed low resolution image DLR output from the first neural network 300. According to an embodiment, the second neural network 400 may further include a space to depth layer 403 . The high-resolution image (PSR) of the previous frame may be input to the space-to-depth layer 403 . The high-resolution image (PSR) of the previous frame output through the second neural network 400 is expressed as (H*scale factor, W*scale factor, C). The noise-removed low-resolution image DLR output from the first neural network 300 is expressed as (H, W, C). The H represents the height of the image, the W represents the width of the image, and the C represents the channel. The scale factor represents a constant. That is, the high-resolution image (PSR) of the previous frame output through the second neural network 400 and the noise-removed low-resolution image (DLR) output from the first neural network 300 have different sizes. Therefore, the high resolution image (PSR) of the previous frame expressed as (H*scale factor, W*scale factor, C) through the space-to-depth layer 403 is changed to (H, W, C*scale factor*scale factor) do.

An image output from the space-to-depth layer 403 and a low-resolution image (DLR) from which noise is removed may be input to the connection layer 405 . The connection layer 405 may connect the image output from the space-to-depth layer 403 and the low-resolution image (DLR) from which noise is removed.

The plurality of layers 410-1 to 410-M are convolution layers or activation layers. The activation layer may be a Rectified Linear Unit (ReLU) activation function. Some of the plurality of layers 410-1 to 410-M form a residual block. Depending on embodiments, the number of the plurality of layers 410-1 to 410-M may vary.

The shuffle layer 420 shuffles tensors including a plurality of channels to generate an image including one channel.

The upscaling layer 425 uses bicubic interpolation to perform upscaling by receiving the noise-removed low-resolution image DLR output from the first neural network 300 .

The adder 430 adds the image output from the shuffle layer 420 and the image output from the upscale layer 425 and outputs the decoded high-resolution image SR.

The second loss function 440 calculates a loss function by receiving the decoded high resolution image SR and the high resolution training image HR. The loss function may be calculated as a mean square error of the decoded high-resolution image (SR) and the high-resolution training image (HR).

According to an embodiment, LPIPS (Learned Perceptual Image Patch Similarity) units 335 and 435 may be used to efficiently train the first neural network 300 and the second neural network 400 .

The first LPIPS unit 335 is included in the first neural network 300 . The first LPIPS unit 335 calculates the image similarity between the noise-removed low-resolution image DLR and the lossless low-resolution training image LLR as a first score (a). The first score (a) is lower as the low-resolution image (DLR) from which noise is removed is similar to the low-resolution training image (LLR) without loss. The first LPIPS unit 335 is implemented as a neural network.

The second LPIPS unit 435 is included in the second neural network 400 . The second LPIPS unit 435 calculates the image similarity (b) between the decoded high-resolution image (SR) and the high-resolution training image (HR) as a second score (b). The second score (b) is lower as the decoded high-resolution image SR and the high-resolution training image HR are similar.

When the first score (a) is greater than the second score (b), the processor 11 trains the first neural network 300 more than the second neural network 400. The processor 11 normalizes the first score (a) and the second score (b) so that the sum of the first score (a) and the second score (b) becomes 1.

By using the LPIPS units 335 and 435, it is possible to determine a neural network that needs more training, and more efficient training is possible by learning more about the neural network that needs more training.

A loss function of the generator network (500) is proposed to train the generator network (500). The processor 11 may train the generator network 500 using a loss function expressed as in Equation 1 below. The loss function of the generator network 500 is expressed by the equation below.

The LossFunction is the loss function of the generator network 500, the a and the b are the first and second scores, the LossFunction1 is the first loss function 340 of the first neural network 300, and the LossFunction2 is In the second loss function 440 of the second neural network 400, the corr1 denotes the correlation of the first neural network 300, and the corr2 denotes the correlation of the second neural network 400.

The correlation of the first neural network 300 is defined by the following equation.

The corr1 denotes the correlation of the first neural network 300, and the crosscorrelation1 denotes the cross-correlation between channels of the feature map of the first neural network 300. The maskmap _c (x, y) is expressed by the following equation.

The LR denotes a pixel value of the decoded low-resolution image LR, which is an input of the first neural network 300, and the gaussianblurkernel LR denotes a pixel value of a Gaussian kernel used to blur the low-resolution image LR. That is, the processor 11 calculates a first absolute value of a difference between a pixel value of the decoded low-resolution image LR, which is an input of the first neural network 300, and a pixel value of the Gaussian kernel.

When the region (x,y) of the decoded low-resolution image LR includes a flat region with low contrast, the correlation of features may be disregarded. When the region (x,y) of the decoded low-resolution image LR includes a texture region with high contrast, the correlation of features should not be neglected.

Equation 3 is used to determine whether the area (x, y) of the image LR includes a flat area or a texture area.

When the difference between the pixel value of the low-resolution image LR decoded in Equation 3 and the pixel value of the Gaussian kernel used to blur the low-resolution image LR is smaller than the first threshold value, the processor 11 generates an image ( It is determined that the flat area is included in the area (x, y) of the LR). Therefore, when the difference between the pixel value of the low-resolution image LR decoded in Equation 3 and the pixel value of the Gaussian kernel used to blur the low-resolution image LR is smaller than the first threshold value, the processor 11 Outputs 0 as the value of maskmap _c (x,y). The processor 11 sets the first absolute value to 0 when the first absolute value is smaller than the first threshold value. The (x,y) represents a pixel position of the image LR.

When the difference between the pixel value of the low-resolution image LR decoded in Equation 3 and the pixel value of the Gaussian kernel used to blur the low-resolution image LR is greater than the first threshold value, the processor 11 produces an image ( It is determined that the texture area is included in the area (x, y) of the LR). Therefore, when the difference between the pixel value of the low-resolution image LR decoded in Equation 3 and the pixel value of the Gaussian kernel used to blur the low-resolution image LR is greater than the first threshold value, the processor 11 Outputs 1 as the value of maskmap _c (x,y). The processor 11 sets the first absolute value to 1 when the first absolute value is greater than the first threshold value. The value of the maskmap _c (x,y) is 0 or 1.

The processor 11 calculates a first cross correlation value between channels of the feature map FM1 of the first neural network 300 . In Equation 2, the crosscorrelation1 represents a cross correlation between channels of the feature map of the first neural network 300 . The feature map FM1 of the first neural network 300 means a feature map input from the first neural network 300 to the shuffle layer 320 . The feature map FM1 input to the shuffle layer 320 includes more features than feature maps output from other layers (eg, 310-1, 310-2, ..., or 310-4). are doing Features refer to points, edges, or objects in an image. A feature is represented by (x, y, channel). The x and y denote positions of pixels in the image, and the channel denotes a channel in the image. When the cross-correlation of features included in each channel is high, each channel need not include all features similar to each other. This is inefficient in obtaining a high-resolution image SR. Therefore, in order to determine the cross-correlation of the features included in each channel, the processor 11 calculates a first cross-correlation value between the channels of the feature map FM1 of the first neural network 300. do.

The correlation of the second neural network 400 is defined by the following equation.

The corr2 denotes the correlation of the second neural network 400, and the crosscorrelation2 denotes the cross-correlation between channels of the feature map of the second neural network 400. The maskmap _s (x, y) is expressed by the following equation.

The DLR is a pixel value of a low-resolution image (DLR) from which noise is removed, which is an input of the second neural network 400, and the gaussianblurkernel (DLR) is a Gaussian kernel used to blur the low-resolution image (DLR) from which noise is removed. represents the pixel value. That is, the processor 11 calculates a second absolute value of a difference between a pixel value of the noise-removed low-resolution image DLR, which is an input of the second neural network 400, and a pixel value of the Gaussian kernel.

When the area (x, y) of the low-resolution image (DLR) from which noise is removed includes a flat area with low contrast, correlation between features may be disregarded. When the area (x, y) of the low-resolution image (DLR) from which noise is removed includes a high-contrast texture area, the correlation of features should not be neglected.

Equation 5 is used to determine whether the area (x, y) of the noise-removed low-resolution image DLR includes a flat area or a texture area.

When the difference between the pixel value of the noise-removed low-resolution image DLR in Equation 5 and the pixel value of the Gaussian kernel used to blur the noise-removed low-resolution image DLR is smaller than the second threshold value, the processor (11) determines that the flat area is included in the area (x, y) of the low-resolution image DLR from which noise has been removed. Therefore, in Equation 5, when the difference between the pixel value of the noise-removed low-resolution image DLR and the pixel value of the Gaussian kernel used to blur the noise-removed low-resolution image DLR is smaller than the second threshold value, The processor 11 outputs 0 as the value of the maskmap _s (x, y). The processor 11 sets the second absolute value to 0 when the second absolute value is smaller than the second threshold value. The (x,y) represents a pixel position of the image LR.

When the difference between the pixel value of the noise-removed low-resolution image DLR in Equation 5 and the pixel value of the Gaussian kernel used to blur the noise-removed low-resolution image DLR is greater than the second threshold value, the processor (11) determines that the texture area is included in the area (x, y) of the low-resolution image DLR from which noise is removed. Therefore, in Equation 5, when the difference between the pixel value of the noise-removed low-resolution image DLR and the pixel value of the Gaussian kernel used to blur the noise-removed low-resolution image DLR is greater than the second threshold value, The processor 11 outputs 1 as the value of the maskmap _s (x, y). The processor 11 sets the second absolute value to 1 when the second absolute value is greater than the second threshold value. The value of the maskmap _s (x, y) is 0 or 1.

The processor 11 calculates a second cross correlation value between channels of the feature map FM2 of the second neural network 400 . In Equation 4, the crosscorrelation2 represents a cross correlation between channels of the feature map of the second neural network 400 . The feature map FM2 of the second neural network 400 means a feature map input from the second neural network 400 to the shuffle layer 420 . The feature map FM2 input to the shuffle layer 420 includes more features than feature maps output from other layers (eg, 410-1, 410-2, ..., or 410-4). Similarly, when the cross-correlation of features included in each channel is high, each channel need not include all features similar to each other. Therefore, in order to determine the cross-correlation of the features included in each channel, the processor 11 calculates the value of the second cross-correlation between the channels of the feature map FM2 of the second neural network 300. do.

The generator network 500 may be referred to as a third neural network 500. The third neural network 500 includes the first neural network 300 and the second neural network 400 .

The loss function of the third neural network 500 is defined as in Equation 1 above. That is, the loss function of the third neural network 500 is a value obtained by multiplying a first absolute value according to the first threshold value by the value of the first cross-correlation, a second absolute value according to the second threshold value, and the It is defined by the value multiplied by the value of the second cross-correlation.

The first absolute value according to the first threshold value represents 0 or 1, which is the value of maskmap _c (x,y) according to Equation 3 above. The value of the first cross-correlation represents crosscorrelation1, which is a cross-correlation between channels of the feature map of the first neural network 300 in Equation 2 above. A value obtained by multiplying the first absolute value according to the first threshold value by the value of the first cross-correlation represents the corr1, which is the correlation of the first neural network 300 in Equation 2.

The second absolute value according to the second threshold value represents 0 or 1, which is a value of maskmap _s (x, y) according to Equation 5 above. The value of the second cross-correlation represents crosscorrelation2, which is a cross-correlation between channels of the feature map of the second neural network 400 in Equation 4 above. A value obtained by multiplying the second absolute value according to the second threshold value by the value of the second cross-correlation represents the corr2, which is the correlation of the second neural network 400 in Equation 4.

4 shows images for explaining the effect of the super-resolution image processing method using a neural network according to an embodiment of the present invention. (a) of FIG. 4 shows a portion of an image transmitted to the image processing device 10 by the computing device 20 . Figure 4 (b) is a low-resolution image. Figure 4 (b) shows a high-resolution image converted from a low-resolution image by the prior art. Figure 4 (c) shows a high-resolution image converted from a low-resolution image according to the present invention. The high-resolution image shown in (b) of FIG. 4 is lower in quality than the high-resolution image shown in (c) of FIG. 4 because conventional coding noise is not removed.

5 is a flowchart of a super-resolution image processing method using a neural network according to an embodiment of the present invention.

Referring to FIGS. 1 to 5 , the processor 11 receives the compressed low-resolution image CLR (S10).

The processor 11 decodes the compressed low-resolution image CLR (S20).

The processor 11 converts the decoded low-resolution image LR into a decoded high-resolution image SR using a pair of neural networks 50 and 60 (S30).

The processor 11 encodes the decoded high-resolution image SR (S40).

The processor 11 transmits the encoded high-resolution image ESR to the computing device 20 through the network 101 (S50).

6 is a block diagram of a network for explaining a training operation of a neural network module according to another embodiment of the present invention.

The network 600 shown in FIG. 6 represents a generative adversarial network (GAN). Referring to FIG. 6 , the network 600 includes a generator network 700 and a discriminator network 770 . The network 600 shown in FIG. 6 is similar in structure and operation to the network 200 shown in FIG. 2 . Therefore, only differences from the network 200 shown in FIG. 2 except for similar parts will be described.

It differs from the generator network 700 shown in FIG. 6 and the generator network 500 shown in FIG. 2 . The generator network 500 shown in FIG. 2 may have a problem in that not only coding noise removal but also texture included in the decoded low-resolution image LR may be removed.

The generator network 700 shown in FIG. 6 is intended to solve this problem. When the coding noise included in the decoded low-resolution image LR is high, the generator network 700 shown in FIG. 6 removes noise from the decoded low-resolution image LR, and generates the noise-removed low-resolution image DLR. The second neural network 720 is used to convert the decoded high-resolution image SR. When the coding noise included in the decoded low-resolution image LR is low, the generator network 700 shown in FIG. 6 converts the decoded low-resolution image LR, not the noise-removed low-resolution image DLR, to the second neural network. It is converted into a decoded high-resolution image (SR) using 720. Therefore, the generator network 700 shown in FIG. 6 can solve the problem of removing the texture included in the decoded low-resolution image LR.

The generator network 700 is trained to receive a decoded low-resolution image LR containing coding noise and output a decoded high-resolution image SR.

The discriminator network 770 receives the high resolution training image HR and the decoded high resolution image SR output from the generator network 700, and calculates the distribution of the high resolution training image HR and the decoded high resolution image SR ) outputs the Wasserstein loss (EM), which is the difference in the distribution of

The generator network 700 includes a first neural network 710, a second neural network 720, a third neural network 730, a fourth neural network 740, a selection signal generator 750, and a selector 760.

The first neural network 710 includes a plurality of layers. The first neural network 710 corresponds to the first neural network 300 shown in FIG. 2 . Accordingly, the first neural network 710 may be implemented as the first neural network 300 shown in FIG. 3 . According to an embodiment, the first neural network 710 may be implemented by modifying the first neural network 300 shown in FIG. 3 .

The second neural network 720 includes a plurality of layers. The second neural network 720 corresponds to the second neural network 400 shown in FIG. 2 . Accordingly, the second neural network 720 may be implemented as the second neural network 400 shown in FIG. 3 . According to an embodiment, the second neural network 720 may be implemented by modifying the second neural network 400 shown in FIG. 3 .

In order to efficiently train the first neural network 710 and the second neural network 720, the LPIPS units 335 and 435 shown in FIG. 3 may be used. Since operations of the LPIPS units 335 and 435 have been described above, a detailed description thereof will be omitted.

Equation 1 described above may be applied to the generator network 700. That is, the loss function of the generator network 700 can be expressed as Equation 1.

In Equation 1, the LossFunction is the loss function of the generator network 700, the a and the b are the first and second scores, the LossFunction1 is the first loss function of the first neural network 710, LossFunction2 is the second loss function of the second neural network 720, corr1 is the correlation of the first neural network 710, and corr2 is the correlation of the second neural network 720. As Equation 1 is applied to the generator network 700, Equations 2 to 5 are also applied to the generator network 700.

The technical characteristics of the first network 300, the second network 400, and the generator network 500 shown in FIG. 2 are the first network 710, the second network 720, and the generator network shown in FIG. It is applied to the technical characteristics of the network 700 as it is.

Each of the third neural network 730 and the fourth neural network 740 may be implemented as a general convolutional neural network (CNN). The third neural network 730 may be trained while adjusting the encoding bitrate in the process of encoding the original image using the original image before encoding and the decoded low-resolution image LR as training data. The fourth neural network 740 may be trained while adjusting the encoding bitrate in the process of encoding the original image before encoding and the low-resolution image (DLR) from which noise is removed as training data.

The processor 11 removes noise from the decoded low-resolution image LR using the first neural network 710 to generate a noise-removed low-resolution image DLR.

The processor 11 outputs the decoded low-resolution image DLR or the noise-removed low-resolution image DLR according to the noise level of the decoded low-resolution image LR and the noise level of the noise-removed low-resolution image DLR. It is converted into a decoded high-resolution image (SR) using the 2 neural network 720.

The selector 760 outputs either the decoded low-resolution image DLR or the noise-removed low-resolution image DLR to the second neural network 720 according to the selection signal SEL. Selector 760 may be implemented with program instructions. According to embodiments, the selector 760 may be implemented in hardware.

When the difference between the noise level of the decoded low-resolution image LR and the noise level of the noise-removed low-resolution image DLR is smaller than a certain value, the processor 11 converts the decoded low-resolution image LR to the second neural network ( 720) to convert into a decoded high-resolution image (SR).

When the difference between the noise level of the decoded low-resolution image LR and the noise level of the noise-removed low-resolution image DLR is greater than the predetermined value, the processor 11 outputs the noise-removed low-resolution image DLR. It is converted into a decoded high-resolution image (SR) using the 2 neural network 720.

The noise level of the decoded low-resolution image LR is generated using the third neural network 730. The processor 11 generates a noise level of the decoded low-resolution image LR by using the third neural network 730 from the decoded low-resolution image LR. The noise level may have a value between 0 and 1.

The noise level of the noise-removed low-resolution image DLR is generated using the fourth neural network 740 . The processor 11 generates a noise level of the noise-removed low-resolution image DLR by using the fourth neural network 740 . The noise level may have a value between 0 and 1.

The selection signal generator 750 compares the noise level of the decoded low-resolution image LR with the noise level of the noise-removed low-resolution image DLR, and generates the selection signal SEL according to the comparison result.

When the difference between the noise level of the decoded low-resolution image LR and the noise level of the noise-removed low-resolution image DLR is smaller than a certain value, the processor 11 generates a selection signal generated by the selection signal generator 750. According to (SEL), the decoded low-resolution image LR is selected from among the decoded low-resolution image LR and the noise-removed low-resolution image DLR, and the decoded low-resolution image LR is used by the second neural network 720. and converts it into a decoded high-resolution image (SR).

When the difference between the noise level of the decoded low-resolution image LR and the noise level of the noise-removed low-resolution image DLR is greater than the predetermined value, the processor 11 selects the signal generated by the selection signal generator 750. According to the signal (SEL), a low-resolution image (DLR) with noise removed is selected from among the decoded low-resolution image (LR) and the noise-removed low-resolution image (DLR), and the noise-removed low-resolution image (DLR) is converted to the second neural network. It is converted into a decoded high-resolution image (SR) using 720.

7 is a flowchart of a super-resolution image processing method robust against coding noise using a plurality of neural networks according to another embodiment of the present invention.

Referring to FIGS. 1, 6 and 7, the processor 11 receives the compressed low-resolution image LR (S100).

The processor 11 decodes the compressed low-resolution image LR (S110).

The processor 11 removes noise from the decoded low-resolution image LR using the first neural network 710 to generate a noise-removed low-resolution image DLR (S120).

The processor 11 determines the decoded low-resolution image LR or the noise-removed low-resolution image ( The DLR) is converted into a decoded high-resolution image SR using the second neural network 720 (S130).

The processor 11 encodes the decoded high-resolution image SR (S140).

The processor 11 transmits the encoded high-resolution image ESR to the computing device 20 (S150).

Although the present invention has been described with reference to an embodiment shown in the drawings, this is only exemplary, and those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the attached claims.

100: super-resolution image processing system using a neural network;
101: network;
10: image processing unit;
11: processor;
13: memory;
15: decoding module;
17: neural network module;
19: encoding module;
20: computing device;

Claims (10)

receiving, by a processor, a compressed low-resolution image;
decoding, by the processor, the compressed low-resolution image;
generating, by the processor, a noise-removed low-resolution image by removing noise from the decoded low-resolution image using a first neural network;
When the difference between the noise level of the decoded low-resolution image and the noise level of the denoised low-resolution image is smaller than a certain value, the processor converts the decoded low-resolution image into a decoded high-resolution image using a second neural network. and converting, by the processor, the low-resolution image from which the noise is removed into the decoded high-resolution image using the second neural network when the difference is greater than the predetermined value;
encoding, by the processor, the decoded high-resolution image; and
The super-resolution image processing method that is robust to coding noise using a plurality of neural networks comprising the step of transmitting, by the processor, the encoded high-resolution image. delete The method of claim 1, wherein the noise level of the decoded low-resolution image is
It is generated using the third neural network,
The noise level of the low-resolution image from which the noise has been removed is
A super-resolution image processing method that is robust against coding noise using a plurality of neural networks generated using a fourth neural network. The method of claim 1, wherein the super-resolution image processing method robust to coding noise using the plurality of neural networks comprises:
calculating, by the processor, an image similarity between the noise-removed low-resolution image and the lossless low-resolution training image as a first score;
calculating, by the processor, an image similarity between the decoded high-resolution image and the high-resolution training image as a second score; and
When the first score is greater than the second score, the processor further training the first neural network than the second neural network,
The lossless low-resolution training images are images used for training of the first neural network, and the high-resolution training images are images used for training of the second neural network,
The first score is lower as the low-resolution image from which the noise is removed is similar to the low-resolution training image,
The second score is lower as the decoded high-resolution image and the high-resolution training image are similar,
The high-resolution training image and the decoded high-resolution image are robust to coding noise using a plurality of neural networks having the same resolution. delete Including an image processing device,
The image processing device,
a processor that executes instructions; and
a memory for storing the instructions;
These commands are
Noise is removed from the low-resolution image using a first neural network to generate a low-resolution image from which the noise is removed;
When the difference between the noise level of the low-resolution image and the noise level of the low-resolution image from which the noise has been removed is smaller than an arbitrary value,
These commands are
It is implemented to convert the low-resolution image into a high-resolution image using a second neural network,
When the difference is greater than the above arbitrary value,
These commands are
A super-resolution image processing system robust against coding noise using a plurality of neural networks implemented to convert the low-resolution image from which the noise is removed into the high-resolution image using the second neural network. delete The method of claim 6, wherein the noise level of the low-resolution image is,
It is generated using the third neural network,
The noise level of the low-resolution image from which the noise has been removed is
A super-resolution image processing system robust against coding noise using a plurality of neural networks generated using a fourth neural network. The method of claim 6, wherein the instructions,
Calculate the image similarity between the noise-removed low-resolution image and the lossless low-resolution training image as a first score;
Calculate the image similarity between the high-resolution image and the high-resolution training image as a second score,
When the first score is greater than the second score, the first neural network is further trained than the second neural network,
The first score is lower as the low-resolution image from which the noise is removed is similar to the lossless low-resolution training image,
The second score is lower as the high-resolution image and the high-resolution training image are similar,
The high-resolution training image and the decoded high-resolution image are robust to coding noise using a plurality of neural networks having the same resolution.



delete

Images