KR20240071720A - Method and apparatus for improving the resolution of images - Google Patents

Abstract

An image resolution improvement method and system for generating a first feature map by inputting a first image into a feature generator and generating a second image by inputting the first feature map into an image generator are provided. At this time, the second image is an image with a higher resolution than the first image, and the image generator includes at least one deconvolution layer that performs up-sampling.

Description

Method and apparatus for improving the resolution of images {METHOD AND APPARATUS FOR IMPROVING THE RESOLUTION OF IMAGES}

This disclosure relates to a method and apparatus for improving the resolution of an image. More specifically, it relates to a method of generating a high resolution (HR) image from a low resolution (LR) image and a system to which the method is applied.

Since high-resolution images or videos have a large capacity, they may be transmitted as low-resolution images or videos depending on network traffic conditions.

In the process of up-sampling a low-resolution image or video to a high-resolution image or video, problems such as partial image quality degradation and overall blurriness may occur. To overcome this, various interpolation methods based on video or image processing are used. Interpolation or deep learning-based super-resolution technology is being used.

However, when upsampling a low-resolution image or video by a certain percentage or more, there are limits to improving image quality, such as the readability of text being reduced and some areas of the image or video appearing blurry, even if deep learning-based super-resolution technology is applied.

Accordingly, there is a need for a technology that can effectively improve the resolution of an image or video even when a low-resolution image or video is upsampled by a certain percentage or more.

US Patent No. 113280905 (published on April 9, 2020)

The technical problem that the present disclosure aims to solve is to provide a method and device for improving image resolution that can generate a high-resolution image by inputting a low-resolution image.

Another technical problem that the present disclosure aims to solve is to provide a method and device for improving image resolution that can solve the problem of partial image quality degradation that may occur when a low-resolution image is upsampled by a certain rate or more.

Another technical problem that the present disclosure aims to solve is to provide images or videos with improved resolution compared to images or videos generated by conventional images or various interpolation methods based on image processing or deep learning-based super-resolution technology. To provide a method and device.

The technical problems of the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by those skilled in the art from the description below.

A method for improving image resolution according to an embodiment of the present disclosure for solving the above technical problem includes inputting a first image into a feature generator to generate a first feature map, and converting the first feature map into an image. It may include generating a second image by inputting it into a generator. At this time, the second image may be an image with a higher resolution than the first image, and the image generator may include at least one deconvolution layer that performs up-sampling. there is.

In one embodiment, the feature generator is a feature extractor that extracts an intermediate feature map from the first image and a feature enhancer that outputs the first feature map by enhancing the intermediate feature map by reflecting the feature map extracted from the correct answer image. It may include a group. At this time, the correct answer image may be a higher resolution image than the first image.

In one embodiment, the feature enhancer It may contain multiple Resnet blocks.

In one embodiment, the feature extractor may include an upscaling block, a downscaling block, and a feature aggregator. At this time, the upscaling block may increase the scale of the input feature map, the downscaling block may decrease the scale of the input feature map, and the feature aggregator may increase the scale of the input feature map. Aggregation of the map and the feature map output from the downscaling block can be performed.

In one embodiment, the upscaling block may be implemented based on a combination of a convolution layer and a deconvolution layer, and the number of deconvolution layers included in the upscaling block may be greater than the convolution layer. .

In one embodiment, the downscaling block may be implemented based on a combination of a convolution layer and a deconvolution layer, and the number of deconvolution layers included in the downscaling block may be less than the convolution layer. .

In one embodiment, the image generator may further include a convolution layer that receives the first feature map, and the at least one deconvolution layer receives the feature map output from the convolution layer. It may be configured to perform upsampling on a basis.

In one embodiment, the method may further include calculating an adversarial loss value using an image discriminator and updating the feature generator and the image generator using the adversarial loss value.

In one embodiment, the method may further include calculating a first loss value between the second image and the correct image and updating the feature generator and the image generator using the first loss value. .

In one embodiment, the first loss value may be one of Mean Square Error (MSE), Perceptual Loss, and Total Variation Loss.

In one embodiment, updating the image generator may include calculating a second loss value and updating the image generator using the second loss value only if a preset condition is met. You can. At this time, the second loss value may be the L1 loss value.

In one embodiment, the preset condition may be that the Peak Signal-to-Noise Ratio (PSNR) value of the second image exceeds a reference value.

In one embodiment, the preset condition may be that the first loss value is less than or equal to a reference value.

In one embodiment, the preset condition may be that the reduction rate of the first loss value is less than or equal to a reference value.

In one embodiment, the preset condition may be that the reduction rate of PSNR for the second image is greater than or equal to a reference value.

In one embodiment, updating the image generator may include updating a weight parameter of the image generator while the feature generator is frozen.

In one embodiment, updating the image generator may include training the image generator based on a weight assigned to the second loss value. At this time, the second loss value may be the L1 loss value. Additionally, the first weight assigned to the second loss value at a first time may be lower than the second weight assigned to the second loss value at a second time after the first time.

In one embodiment, the first image may be a frame image of a video conference video received in real time.

An image resolution improvement system according to another embodiment of the present disclosure for solving the technical problem may include a processor and a memory for storing instructions, wherein the instructions, when executed by the processor, cause the processor to generate a first image. An operation of generating a first feature map can be performed by inputting a to a feature generator, and an operation of generating a second image can be performed by inputting the first feature map into an image generator. At this time, the second image may be an image with a higher resolution than the first image, and the image generator may include at least one deconvolution layer that performs up-sampling. there is.

According to this embodiment, user satisfaction and usability can be improved by providing an image or video with improved resolution than an image or video upsampled by the prior art.

According to this embodiment, the frame-by-frame quality of an image or video can be effectively improved by updating the feature map using a plurality of convolution layers, deconvolution layers, and Resnet blocks.

1 is an exemplary diagram of an image generated according to a conventional image resolution improvement method and an image resolution improvement method according to an embodiment of the present disclosure.
Figure 2 is a flowchart showing a method for improving image resolution, according to an embodiment of the present disclosure.
3 is an exemplary diagram of the configuration of a feature generator, which may be referenced in some embodiments of the present disclosure.
FIG. 4 is an example diagram for explaining some of the configurations shown in FIG. 3.
Figure 5 is an example diagram for explaining a feature map supplementation process by a feature extractor, which may be referred to in some embodiments of the present disclosure.
FIG. 6 is an example diagram for explaining some operations shown in FIG. 2.
Figure 7 is a flowchart for explaining the inference process of an image resolution improvement model, which may be referenced in some embodiments of the present disclosure.
Figure 8 is an example diagram of the configuration of an image discriminator, which may be referenced in some embodiments of the present disclosure.
Figure 9 is a flowchart showing a method of updating an image resolution improvement model according to another embodiment of the present disclosure.
FIG. 10 is a detailed flowchart of some operations described with reference to FIG. 9.
FIG. 11 is an example diagram for explaining some operations shown in FIG. 9.
FIG. 12 is a flowchart illustrating the learning process of an image resolution improvement model using an image discriminator, which may be referenced in some embodiments of the present disclosure.
FIG. 13 is a flowchart illustrating the learning process of an image resolution improvement model using a correct answer image, which may be referred to in some embodiments of the present disclosure.
14 is a hardware configuration diagram of a computing system according to some embodiments of the present disclosure.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the attached drawings. The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the technical idea of the present invention is not limited to the following embodiments and may be implemented in various different forms. The following examples are merely intended to complete the technical idea of the present invention and to be used in the technical field to which the present invention pertains. It is provided to fully inform those skilled in the art of the scope of the present invention, and the technical idea of the present invention is only defined by the scope of the claims.

In describing the present disclosure, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description will be omitted.

Unless otherwise defined, terms (including technical and scientific terms) used in the following embodiments may be used with meanings that can be commonly understood by those skilled in the art to which this disclosure pertains. It may vary depending on the intentions or precedents of engineers working in related fields, the emergence of new technologies, etc. The terminology used in this disclosure is for describing embodiments and is not intended to limit the scope of this disclosure.

The singular expressions used in the following embodiments include plural concepts, unless the context clearly specifies singularity. Additionally, plural expressions include singular concepts, unless the context clearly specifies plurality.

In addition, terms such as first, second, A, B, (a), (b) used in the following embodiments are only used to distinguish one component from another component, and the terms The nature, sequence, or order of the relevant components are not limited.

Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the attached drawings.

1 is an exemplary diagram of an image generated according to a conventional image resolution improvement method and an image resolution improvement method according to an embodiment of the present disclosure.

The video conference system server program can encode video streams of various resolutions depending on the network transmission situation, and can select and transmit video streams of appropriate resolution. For example, a video conference system server program can encode an original video frame with 720p resolution into 360p or 180p depending on the network transmission situation, 720p when network transmission is smooth, and 360p or 180p when network transmission is not smooth. It can be transmitted to .

At this time, when the server transmits a stream with a resolution lower than the target resolution of the client program, the client program that receives it can upsample the low-resolution stream to the size of the target resolution and output it on the screen.

As shown in FIG. 1, when network transmission conditions are smooth, a high-resolution original image 1a can be transmitted. Additionally, if the network transmission situation is not smooth due to reasons such as an excessively large number of concurrent users, a low-resolution image (1b) may be transmitted. At this time, since the quality of the image is too low to output the low-resolution image 1b on the client program screen, an upsampled image 1c of the low-resolution image may be displayed on the screen.

However, referring to the image (1c) in which the low-resolution image is upsampled, there is a problem with the readability of the text due to distortion of some letters (e), overlapping parts of some letters (f, a), and blurring of some letters. There may be.

Accordingly, according to an image resolution improvement method according to some embodiments of the present disclosure, which will be described later, an image 1d with higher resolution than the image 1c in which a low-resolution image is upsampled can be generated.

Hereinafter, a method for improving image resolution according to some embodiments of the present disclosure will be described with reference to FIGS. 2 to 13. However, for convenience of explanation, this specification focuses on a method for improving the resolution of a specific image. However, the present disclosure is not limited thereto, and the resolution of a specific image can also be improved by methods described later.

Figure 2 is a flowchart showing a method for improving image resolution, according to an embodiment of the present disclosure. However, this is only a preferred embodiment for achieving the purpose of the present disclosure, and of course, some steps may be added or deleted as needed.

As shown in FIG. 2, the method for improving image resolution according to an embodiment of the present disclosure begins at step S100 of generating a first feature map by inputting a first image into a feature generator. At this time, the first image may be an image transmitted in low resolution due to poor network transmission. Additionally, the first image may be a frame image of a video conference video received in real time.

The feature generator may include a feature extractor that extracts an intermediate feature map from the first image and a feature enhancer that outputs the first feature map by reflecting the feature map extracted from the correct answer image to enhance the intermediate feature map. . At this time, the correct answer image may be an image before the first image is transmitted. That is, the correct answer image may be a higher resolution image than the first image.

Additionally, the feature enhancer may include a plurality of Resnet blocks, and the feature extractor may include an upscaling block, a downscaling block, and a feature aggregator. At this time, the upscaling block may increase the scale of the input feature map, the downscaling block may decrease the scale of the input feature map, and the feature aggregator may increase the scale of the input feature map. Aggregation of the map and the feature map output from the downscaling block can be performed. A detailed description of this will be provided later with reference to FIGS. 3 and 4.

Referring again to FIG. 2, in step S200, a second image may be generated as a result of inputting the first feature map to the image generator. At this time, the image generator may include at least one deconvolution layer that performs upsampling. Additionally, the image generator may further include a convolution layer that receives the first feature map.

Meanwhile, the feature generator and the image generator can be updated using the loss calculated between the second image and the correct image, and can also be updated using the loss calculated using the image discriminator. Accordingly, the resolution of the image can be improved by updating the feature generator and the image generator based on the loss. A detailed description of this will be provided later with reference to FIGS. 8 to 13.

3 is an exemplary diagram of the configuration of a feature generator, which may be referenced in some embodiments of the present disclosure.

As shown in Figure 3, the feature generator 30 reflects the feature extractor 10 that extracts the intermediate feature map (F1) from the first image (LR) and the feature map extracted from the correct answer image (HR) to create the intermediate feature map (F1). It may include a feature enhancer 20 that outputs the first feature map F2 by enhancing the feature map F1. At this time, the feature extractor 10 may include an upscaling block 12, a downscaling block 13, and a feature aggregator 14, and the feature enhancer 20 may include a plurality of Resnet blocks 21, 22) may be included.

First, the convolution block 11 included in the feature extractor 10 can generate a feature map for the first image using a plurality of convolution layers. At this time, the convolution block may be a network unit composed of two or more convolution layers. Additionally, the feature map for the generated first image may be sequentially input to the upscaling block 12, the downscaling block 13, and the upscaling block 12, and a plurality of feature maps are generated from each block. It can be. Furthermore, the generated plurality of feature maps may be aggregated through the feature aggregator 14. At this time, the acgregating may be, for example, concatenation.

At this time, in the deconvolution process, a low-frequency feature map that may disappear in the convolution process may be created, and conversely, in the convolution process, a high-frequency feature map may be created that may become blurred in the deconvolution process. . Furthermore, the feature map can be strengthened through back projection using the generated low-frequency feature map and high-frequency feature map.

Meanwhile, the feature layer in each block may include a convolution layer capable of compressing the feature map and a deconvolution layer capable of enlarging the feature map, and the output value of each feature layer may be activated. Therefore, PReLU (Parametric ReLU) can be used as an activation function for updating the weights of parameters. Those working in the relevant technical field will already be familiar with PReLU, so detailed explanations thereof will be omitted.

FIG. 4 is an example diagram for explaining some of the configurations shown in FIG. 3. More specifically, FIG. 4 is an enlarged example of the upscaling block and downscaling block included in the feature extractor shown in FIG. 3.

As shown in FIG. 4, the upscaling block 12 may be a block composed of a deconvolution layer 12a, a convolution layer 12b, and a deconvolution layer 12a sequentially. At this time, when the upscaling block 12 receives the feature map F4, it can generate a feature map F5 in which the feature map F4 is enlarged.

Likewise, the downscaling block 13 may be a block composed of a convolution layer 13b, a deconvolution layer 13a, and a convolution layer 13b in succession. At this time, when the downscaling block 13 receives the feature map F6, it can generate a feature map F7 in which the feature map F6 is reduced.

Meanwhile, the combination of the convolution layer and deconvolution layer constituting the upscaling block 12 and the downscaling block 13 is not limited to what is shown in FIG. 4, and the upscaling block 12 and the downscaling block 13 Block 13 may be configured in various combinations.

That is, the upscaling block can be implemented based on a combination of a convolution layer and a deconvolution layer, and in particular, the number of deconvolution layers included in the upscaling block can be implemented so that it is greater than the convolution layer. . Additionally, the downscaling block may be implemented based on a combination of a convolution layer and a deconvolution layer. In particular, the downscaling block may be implemented so that the number of deconvolution layers included is smaller than the convolution layer. there is.

Figure 5 is an example diagram for explaining a feature map supplementation process by a feature extractor, which may be referred to in some embodiments of the present disclosure.

As shown in Figure 5, the deconvolution layers 12a and 13a included in the upscaling block 12 or the downscaling block 13 can generate a feature map for a high-resolution image, and the convolution layer ( 12b, 13b) can generate a feature map for a low-resolution image. Afterwards, the feature map can be supplemented through a process of backprojecting the difference between the feature map for the low-resolution image and the feature map for the high-resolution image at each step. Furthermore, feature maps generated through the backprojection process can be used as input values for a feature enhancer through a convolution layer after concatenation. If you are working in the relevant technical field, you will already be familiar with back projection and concatenation, so detailed explanations regarding them will be omitted.

Referring again to FIG. 3, the feature enhancer 20 included in the feature generator 30 may include a plurality of Resnet blocks 21 and 22.

At this time, the Resnet blocks 21 and 22 may be configured to supplement the feature map through back projection using the correct answer image. More specifically, the Resnet deconvolution block 21 and the Resnet convolution block 22 can supplement the feature map by backprojecting the difference between the feature map output from each block and the correct image. Additionally, image resolution can be further improved by using the concatenated feature map as an input value to the feature extractor 10.

Meanwhile, the improvement in image resolution can be measured by Peak Signal-to-Noise Ratio (PSNR). At this time, the process of inputting the first image into the feature extractor 10 to generate an intermediate feature map and inputting the generated intermediate feature map into the feature enhancer 20 to generate the first feature map is repeatedly performed. In this case, image resolution can be improved and PSNR values can also be increased. Anyone working in the relevant technical field will already be familiar with PSNR, so detailed explanations thereof will be omitted.

FIG. 6 is an example diagram for explaining some operations shown in FIG. 2. More specifically, FIG. 6 is an example diagram showing an image generator in addition to the configuration of the feature generator shown in FIG. 3.

As shown in FIG. 6, the feature map generated through the feature extractor 10 and the feature enhancer 20 may be input to the image generator 40, and a second image may be generated as a result of the input. there is. At this time, the second image may be an image with a higher resolution than the first image input to the feature extractor 10.

Image generator 40 may include at least one deconvolution layer (not shown) that performs upsampling. Additionally, the image generator 40 may further include a convolution layer (not shown) that receives the feature map generated through the feature extractor 10 and the feature enhancer 20.

The image generator 40 according to an embodiment of the present disclosure may have a structure similar to that of a fully convolutional network (FCN) composed of a plurality of convolutional layers. However, while FCN upsamples the feature map using bilinear interpolation in the last stage, the image generator 40 upsamples the feature map using a deconvolution layer in the last stage, thereby improving further. You can create images with high resolution.

Meanwhile, the loss of the feature map concatenated in the feature enhancer 20 from the correct image can be calculated through the image generator 40, and the image generator 40 can be updated based on the calculated loss. there is. A detailed description of the update of the image generator 40 according to some embodiments of the present disclosure will be described later with reference to FIGS. 10 and 11.

Figure 7 is a flowchart for explaining the inference process of an image resolution improvement model, which may be referenced in some embodiments of the present disclosure.

As shown in FIG. 7, a low-resolution image input to the feature generator 30 may be output as a high-resolution image through the feature generator 30 and the image generator 40. More specifically, as a result of inputting a low-resolution first image to the feature extractor 10 from a low-resolution image, an intermediate feature map may be generated through a process of enlarging or compressing the feature map for the first image. Next, as a result of inputting the intermediate feature map to the feature enhancer 20, a first feature map may be generated through a process of supplementing the intermediate feature map. Finally, as a result of inputting the first feature map to the image generator 40, a high-resolution second image may be generated through a process of supplementing and upsampling the first feature map.

So far, a method for improving image resolution according to some embodiments of the present disclosure has been described with reference to FIGS. 2 to 7 . According to the image resolution improvement method described above, a high-resolution image is generated through a process of changing the size of the feature map using a plurality of convolution layers and a deconvolution layer and a process of supplementing the feature map using a plurality of reznet blocks. can be created. Accordingly, the frame-by-frame quality of images or videos transmitted in low-resolution sizes can be effectively improved depending on the network transmission situation, and user satisfaction and usability can be improved.

Hereinafter, a method of learning an image resolution improvement model according to another embodiment of the present disclosure will be described with reference to FIGS. 8 to 13. At this time, the image resolution improvement model may include a feature generator and an image generator, and in some cases may further include an image discriminator.

Figure 8 is an example diagram of the configuration of an image discriminator, which may be referenced in some embodiments of the present disclosure.

As shown in FIG. 8, the image resolution improvement model according to an embodiment of the present disclosure may include a feature extractor 10, a feature enhancer 20, an image generator 40, and an image discriminator 50. there is. Since the feature extractor 10, feature enhancer 20, and image generator 40 have been described with reference to FIGS. 2 to 7, detailed description thereof will be omitted.

The image discriminator 50 can determine whether the high-resolution second image generated by the image generator 40 is a correct image or an incorrect image and calculate an adversarial loss value. Afterwards, the feature extractor 10, feature enhancer 20, and image generator 40 can be updated using the calculated adversarial loss value. At this time, the adversarial loss value can be calculated using the formula below.

Anyone working in the relevant technology field will already be familiar with the super-resolution algorithm based on GAN (Generative Adversarial Nets), so detailed explanations thereof will be omitted.

Figure 9 is a flowchart showing a method of updating an image resolution improvement model according to another embodiment of the present disclosure. However, this is only a preferred embodiment for achieving the purpose of the present disclosure, and of course, some steps may be added or deleted as needed.

As shown in FIG. 9, the image resolution improvement method according to another embodiment of the present disclosure calculates a first loss value between the second image generated according to the image resolution improvement method described with reference to FIG. 2 and the correct image. It may include step S300 and step S400 of updating the feature generator and the image generator using the first loss value. At this time, the correct answer image may be a higher resolution image than the second image.

The first loss value may be one of Mean Square Error (MSE), Perceptual Loss, and Total Variation Loss.

As can be seen in the formula below, MSE loss is a loss value calculated as the average value of the total sum after squaring the difference between the high-resolution image generated from the image resolution improvement model and the correct image.

As can be seen in the formula below, the perceptual loss is similar to the MSE loss described above, and there is a similarity between the high-resolution image generated from the image resolution improvement model and the feature map obtained by passing the correct image through the pre-trained VGG-19. This is the loss value calculated from the Euclidean distance.

Total Variation Loss is calculated using the difference between the reference pixel (i, j) and adjacent pixels ((i, j+1), (i+1, j)), as can be seen in the formula below. This is the loss value.

Meanwhile, in the process of updating the feature generator and image generator using Mean Square Error (MSE), Perceptual Loss, Adversarial Loss, and Total Variation Loss, Each of the above losses can be used in training of the feature generator and image generator in various proportions. For example, the first loss values can be used for learning the feature generator and the image generator at the following ratio.

Those working in the relevant technical field will already be familiar with Mean Square Error (MSE), Perceptual Loss, and Total Variation Loss, so detailed explanations thereof will be omitted. do.

FIG. 10 is a detailed flowchart of some operations described with reference to FIG. 9.

As shown in FIG. 10, step S400 of updating the feature generator and image generator using the first loss value includes step S410 of calculating a second loss value only when a preset condition is met, and the second loss value. Step S420 may include updating the image generator using the loss value. At this time, the second loss value is an L1 loss value, which can be calculated through the L1 distance between the second image generated by the image generator and the correct image as shown in the formula below.

Step S420 of updating the image generator using L1 loss according to an embodiment of the present disclosure can be performed only when a preset condition is met. For example, as described above with reference to FIG. 9, the image resolution improvement model consisting of a feature extractor, feature enhancer, and image generator may be updated using the first loss value, and if preset conditions are met, Insofar as the L1 loss value is used, only the image generator can be additionally updated during the construction of the image resolution improvement model. At this time, the preset condition may be that the difference between the second image generated through the updated image resolution improvement model and the correct image is less than or equal to a reference value. A detailed description of the preset conditions will be described later with reference to FIG. 11.

The reason why step S420 of updating the image generator using L1 loss is performed only when preset conditions are met is that a total of 5 types of loss values (L1 loss, MSE, Perceptual loss), including L1 loss from 1-epoch, are performed. This is because when end-to-end machine learning for the image generator is performed using Adversarial loss (TV loss), the L1 loss can converge to the overall loss. Therefore, L1 loss for updating the image generator can be performed after the change in loss becomes gradual.

That is, the feature extractor, feature enhancer, and image generator can continue to be learned with a combination of the four types of first loss values as described in [Equation 5], and the image generator will be updated when preset conditions are met. You can. At this time, the update of the image generator using L1 loss can be performed with the same proportion as the formula below.

As shown in the formula above, the proportion of L1 loss can be adjusted by multiplying by an arbitrary value. If the edge contrast of the image or objects distributed in the image is clear (e.g. black text on a white background), the error or loss is calculated to be large. It can be. In this case, the arbitrary value may be set as small as 0.001. Additionally, when the edge contrast of an object is gentle (e.g. similar colored background and foreground or gradient area), the error or loss may be calculated to be small and may be reflected as about 0.01. In other words, the reflection rate of L1 loss can be adaptively adjusted depending on the degree of error or loss.

Accordingly, after the feature extractor, feature enhancer, and image generator are sufficiently updated using the first loss value (MSE, Perceptual loss, TV loss) and Adversarial loss (MSE), an additional second loss value (L1 By updating the image generator using loss), more elaborate and higher resolution images can be created.

Meanwhile, in the process of updating the image generator using L1 loss, not only can the weight parameters of the feature generator and the image generator be updated simultaneously, but only the weight parameters of the image generator can be updated while the feature generator is frozen. It may be possible.

Additionally, in the process of updating the image generator using L1 loss, the image generator may be updated based on the weight given to L1 loss. More specifically, the first weight assigned to the L1 loss at the first time point may be lower than the second weight assigned to the L1 loss at the second time point. At this time, the first time point may be before the second time point.

FIG. 11 is an example diagram for explaining some operations shown in FIG. 9. In more detail, Figure 11 is an example diagram of preset conditions corresponding to the standard for updating the image generator using L1 loss.

First, the preset condition may be that the Pick Signal-to-Noise Ratio (PSNR) value of the high-resolution second image generated from the image generator exceeds the standard value. That is, as shown in the graph shown in the upper left corner of FIG. 11, L1 loss can be calculated at the time (t1) when the PSNR value of the second image exceeds the reference value, and the image generator can be updated using the calculated L1 loss. You can.

Second, the preset condition may be that the first loss value calculated between the high-resolution second image generated from the image generator and the correct image is below the reference value. At this time, the first loss value may be one of MSE, Perceptual loss, and TV loss values. That is, as shown in the graph shown in the upper right corner of FIG. 11, L1 loss can be calculated at a time point (t2) when the first loss value calculated between the second image and the correct image is less than the reference value, and using the calculated L1 loss The image generator may be updated.

Third, the preset condition may be that the reduction ratio of the first loss value calculated between the high-resolution second image generated from the image generator and the correct image is below the standard value. At this time, the first loss value may be one of MSE, Perceptual loss, and TV loss values. That is, as shown in the graph shown in the lower left corner of FIG. 11, L1 loss can be calculated at a time when the reduction rate of the first loss value calculated at a plurality of time points (t3, t4) is below the reference value, and the calculated L1 loss is The image generator can be updated using

Fourth, the preset condition may be that the Pick Signal-to-Noise Ratio (PSNR) reduction ratio for the high-resolution second image generated from the image generator is greater than or equal to the reference value. That is, as shown in the graph shown in the upper left corner of FIG. 11, L1 loss can be calculated at a time when the PSNR reduction ratio for the second image calculated at a plurality of time points (t5, t6) is greater than the reference value, and the calculated L1 loss The image generator can be updated using .

FIG. 12 is a flowchart illustrating the learning process of an image resolution improvement model using an image discriminator, which may be referenced in some embodiments of the present disclosure.

As shown in FIG. 12, the image discriminator 50 may calculate an adversarial value of the second image (SR image) generated by the image generator 40. At this time, if the adversarial loss value does not converge to the standard error, the feature extractor 10, the feature enhancer 20, and the image generator 40 may be continuously trained using the adversarial loss value. Additionally, when the adversarial loss value converges to the reference error, training of the image resolution improvement model may be terminated.

FIG. 13 is a flowchart illustrating the learning process of an image resolution improvement model using a correct answer image, which may be referred to in some embodiments of the present disclosure.

As shown in FIG. 13, the first loss values (MSE loss, Perceptual loss, Total-Variation loss) between the second image (SR image) generated from the image generator 40 and the correct answer image (HR image) are calculated. You can. At this time, if the first loss value does not converge to the reference error, the feature extractor 10, the feature enhancer 20, and the image generator 40 may be continuously trained using the first loss value. Additionally, when the first loss value converges to the reference error, training of the image resolution improvement model may be terminated.

Meanwhile, the image generator 40 may be additionally trained only when preset conditions are met during the learning process of the image resolution improvement model using the first loss value. If the preset condition is met, a second loss value (L1 loss) between the second image (SR image) generated by the image generator 40 and the correct answer image (HR image) may be calculated, and the second loss value An image generator can be learned using . The case where the preset conditions are met has been described above with reference to FIG. 11 and will therefore be omitted.

So far, a method for learning an image resolution improvement model according to another embodiment of the present disclosure has been described with reference to FIGS. 8 to 13 . According to the above-described learning method of the image resolution improvement model, an image with higher resolution can be generated by additionally training the image generator based on L1 loss only when preset conditions are met. Accordingly, the frame-by-frame quality of images or videos transmitted in low-resolution sizes can be effectively improved depending on the network transmission situation, and user satisfaction and usability can be improved.

14 is a hardware configuration diagram of an image resolution improvement system according to some embodiments of the present disclosure. The image resolution improvement system 1000 shown in FIG. 14 loads one or more processors 1100, a system bus 1600, a communication interface 1200, and a computer program 1500 performed by the processor 1100. ) and a storage 1300 that stores a computer program 1500.

The processor 1100 controls the overall operation of each component of the image resolution improvement system 1000. The processor 1100 may perform operations on at least one application or program to execute methods/operations according to various embodiments of the present disclosure. The memory 1400 stores various data, commands and/or information. The memory 1400 may load one or more computer programs 1500 from the storage 1300 to execute methods/operations according to various embodiments of the present disclosure. Bus 1600 provides communication between components of image resolution improvement system 1000. The communication interface 1200 supports Internet communication of the image resolution improvement system 1000. Storage 1300 may non-temporarily store one or more computer programs 1500. The computer program 1500 may include one or more instructions implementing methods/operations according to various embodiments of the present disclosure. When the computer program 1500 is loaded into the memory 1400, the processor 1100 can perform methods/operations according to various embodiments of the present disclosure by executing the one or more instructions.

In some embodiments, the image resolution improvement system 1000 described with reference to FIG. 14 is configured using one or more physical servers included in a server farm based on cloud technology such as a virtual machine. It can be. In this case, at least some of the processor 1100, memory 1400, and storage 1300 among the components shown in FIG. 15 may be virtual hardware, and the communication interface 1200 may also be a virtual switch. It may be composed of virtualized networking elements such as switches.

So far, various embodiments of the present disclosure and effects according to the embodiments have been mentioned with reference to FIGS. 1 to 14 . The effects according to the technical idea of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below.

The technical ideas of the present disclosure described so far can be implemented as computer-readable code on a computer-readable medium. The computer program recorded on the computer-readable recording medium can be transmitted to another computing device through a network such as the Internet and installed on the other computing device, and thus can be used on the other computing device.

Although operations are shown in the drawings in a specific order, it should not be understood that the operations must be performed in the specific order shown or sequential order or that all illustrated operations must be performed to obtain the desired results. In certain situations, multitasking and parallel processing may be advantageous. Although embodiments of the present disclosure have been described above with reference to the attached drawings, those skilled in the art will understand that the present invention can be implemented in other specific forms without changing the technical idea or essential features. I can understand that there is. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. The scope of protection of the present invention should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be construed as being included in the scope of rights of the technical ideas defined by this disclosure.

Claims (19)

In a method performed by a computing device,
Generating a first feature map by inputting the first image into a feature generator; and
Including inputting the first feature map into an image generator to generate a second image,
The second image is a higher resolution image than the first image,
The image generator includes at least one deconvolution layer that performs up-sampling,
How to improve image resolution. According to claim 1,
The feature generator is,
a feature extractor that extracts an intermediate feature map from the first image; and
A feature enhancer that outputs the first feature map by enhancing the intermediate feature map by reflecting the feature map extracted from the correct answer image,
The correct answer image is a higher resolution image than the first image,
How to improve image resolution. According to clause 2,
The feature enhancer includes a plurality of Resnet blocks,
How to improve image resolution. According to clause 2,
The feature extractor includes an upscaling block, a downscaling block, and a feature aggregator,
The upscaling block increases the scale of the input feature map,
The downscaling block reduces the scale of the input feature map,
The feature aggregator performs aggregation of the feature map output from the upscaling block and the feature map output from the downscaling block,
How to improve image resolution. According to clause 4,
The upscaling block is implemented based on a combination of a convolution layer and a deconvolution layer,
The number of deconvolution layers included in the upscaling block is greater than the convolution layer,
How to improve image resolution. According to clause 4,
The downscaling block is implemented based on a combination of a convolution layer and a deconvolution layer,
The number of deconvolution layers included in the downscaling block is less than the convolution layer,
How to improve image resolution. According to claim 1,
The image generator further includes a convolution layer that receives the first feature map,
The at least one deconvolution layer is configured to perform upsampling based on the feature map output from the convolution layer,
How to improve image resolution. According to claim 1,
Calculating an adversarial loss value using an image discriminator; and
Further comprising updating the feature generator and the image generator using the adversarial loss value,
How to improve image resolution. According to claim 1,
calculating a first loss value between the second image and the correct image; and
Further comprising updating the feature generator and the image generator using the first loss value,
The correct answer image is a higher resolution image than the second image,
How to improve image resolution. According to clause 9,
The first loss value is,
One of Mean Square Error (MSE), Perceptual Loss, and Total Variation Loss,
How to improve image resolution. According to clause 9,
The step of updating the image generator is,
calculating a second loss value only when preset conditions are met; and
Updating the image generator using the second loss value,
The second loss value is the L1 loss value,
How to improve image resolution. According to claim 11,
The preset conditions are:
The PSNR (Peak Signal-to-Noise Ratio) value of the second image exceeds the standard value,
How to improve image resolution. According to claim 11,
The preset conditions are:
Wherein the first loss value is less than or equal to the reference value,
How to improve image resolution. According to claim 11,
The preset conditions are:
The reduction rate of the first loss value is less than or equal to the standard value,
How to improve image resolution. According to claim 11,
The preset conditions are:
The reduction rate of PSNR for the second image is greater than or equal to the reference value,
How to improve image resolution. According to clause 9,
The step of updating the image generator is,
In a state of freezing the feature generator, updating weight parameters of the image generator,
How to improve image resolution. According to clause 9,
The step of updating the image generator is,
Updating the image generator based on the weight assigned to the second loss value,
The second loss value is the L1 loss value,
The first weight assigned to the second loss value at a first time point is lower than the second weight assigned to the second loss value at a second time point after the first time point,
How to improve image resolution. According to claim 1,
The first image is a frame image of a video conference video received in real time,
How to improve image resolution. processor; and
Includes a memory for storing instructions,
When executed by the processor, the instruction causes the processor to:
An operation of generating a first feature map by inputting a first image into a feature generator; and
An operation of generating a second image is performed by inputting the first feature map into an image generator,
The second image is a higher resolution image than the first image,
The image generator includes at least one deconvolution layer that performs up-sampling,
Image resolution improvement system.

Images