KR20220102811A - Apparatus and method for reconstructing single image super-resolution - Google Patents

Abstract

A single image super-resolution reconstruction device according to the present invention includes: an image input unit for receiving a low-resolution image and providing the image to a super-resolution network; and an image reconstruction unit generating a high-resolution image by enlarging the low-resolution image by a predefined magnification using predefined super-resolution networks. Here, the super-resolution network may be configured such that a residual dense channel attention block (RDCAB) including a residual dense structure and a channel concentration structure is recursively repeated a predetermined number of times.

Description

Single image super-resolution restoration apparatus and method

The present invention relates to a single image super-resolution technology based on deep learning that can generate (or restore) a high-resolution image from a low-resolution image.

A single image super-resolution technique is a technique for generating (or restoring) a high-resolution image from a low-resolution image. Recently, a high-resolution display of an Ultra-High Definition (UHD) level has been widely distributed, and interest in a super-resolution algorithm capable of converting a low-resolution image of a Full-High Definiton (FHD) level into a high-resolution image is growing.

Representative methods of increasing the resolution of a single image can be classified into interpolation, reconstruction, and deep learning methods in a wide range. Interpolation is a technique for generating new pixels where there is no information using a given surrounding pixel value. Specifically, nearest pixel interpolation method, linear interpolation method, bilinear interpolation method, and high-dimensional interpolation method exist. Here, the interpolation method has an advantage in that the amount of calculation is small, but due to the low-frequency filter characteristic, the image is blurred as a whole, and thus the image quality of detailed parts may be deteriorated and deteriorated. Compared to the interpolation method, the reconstruction method restores high-frequency components well, but it has a relatively complex amount of computation.

Recently, due to the technological development of deep learning methods, deep learning-based super-resolution algorithms have shown higher performance compared to existing image processing methods, and various networks have been proposed and actively studied to this day. Super-resolution techniques applying deep learning have been developed in various ways, such as linear, residual learning, recursive, dense, and concentrated techniques, and various networks have been proposed.

RDN (Residual Dense Network), which is a representative technique among densely connected network structures, is suitable for extracting a feature map of an image by using information from all previous layers. Here, the RDN shows excellent performance by reusing the learned various characteristic information to make the flow of information flexible. However, in the case of RDN, it has a large number of parameters as a large number of residual dense blocks are stacked, so it takes a lot of time to learn the network, and it is difficult to apply to a mobile system because a large storage space is required.

Currently, it will be necessary to develop a new super-resolution technique that can solve these problems.

Korean Patent Registration No. 10-2059529

The present invention has been derived to solve the above-described problems, and it is intended to provide a single image super-resolution technology based on deep learning, which can exhibit a fast processing speed by significantly reducing the amount of calculation and have excellent performance at the same time.

A single image super-resolution restoration apparatus according to an aspect of the present invention includes: an image input unit for receiving a low-resolution image and providing it to a super-resolution network; and an image restoration unit that generates a high-resolution image by magnifying the low-resolution image by a predefined magnification using predefined Super-Resolution Networks. Here, the super-resolution network may be configured such that a Residual Dense Channel Attention Block (RDCAB) including a residual density structure and a channel concentration structure is recursively repeated a predetermined number of times.

In an embodiment, the super-resolution network may include: a feature extraction unit including first and second convolutional layers; a recursive block including the RDCAB and Concat layers; convolutional bottleneck; a third convolutional layer; and a restoration unit; may be configured to be connected in series.

In an embodiment, the feature extraction unit extracts a predetermined number of feature maps from the first and second convolutional layers, respectively, and the output of the first convolutional layer is a second convolutional layer and a last layer. input, and the output of the second convolution layer may be configured to be input to the recursive block.

In an embodiment, the recursive block may be configured such that the RDCAB is recursively repeated a predetermined number of times, and features outputted for each number are connected by a Concat layer.

In an embodiment, the RDCAB is configured by connecting the residual dense structure and the channel intensive structure in series, wherein the residual dense structure includes a plurality of predefined convolutional layers and concat layers, and the channel intensive structure is It can be constructed including an average pooling layer and a sigmoid function.

In an embodiment, the reconstructor may perform an up-scaling process using an Efficient Sub-Pixel Convolutional Neural Network (ESPCN), and reconstruct features generated through the up-scaling process into an RGB image through a synthesis product.

According to another aspect of the present invention, the single image super-resolution restoration method is performed in a single image super-resolution restoration apparatus. Here, the single image super-resolution restoration method includes the steps of: an image providing unit, receiving a low-resolution image and providing it to a super-resolution network; and generating, by an image restoration unit, a high-resolution image by magnifying the low-resolution image by a predefined magnification using a predefined super-resolution network (Super-Resolution Networks). and RDCAB (Residual Dense Channel Attention Block) including a channel concentration structure may be configured to recursively repeat a predetermined number of times.

The present invention exhibits a faster processing speed and at the same time has excellent restoration performance, as compared with existing super-resolution techniques such as SRCNN and VDSR.

1 is a reference diagram for explaining a super-resolution network (Super-Resolution Networks) according to the present invention.
2 is a reference diagram for explaining the RDCAB structure according to the present invention.
3 is a reference diagram for explaining a single image super-resolution restoration apparatus according to the present invention.
4 to 7 are reference diagrams for explaining the performance of the single image super-resolution reconstruction technique according to the present invention.

Since the present invention can apply various transformations and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. In describing the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted.

Terms such as first, second, etc. may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It is to be understood that this does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof. Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1. Introduction

As described above, various networks for super-resolution techniques to which deep learning is applied have been proposed.

RDN (Residual Dense Network), which is a representative technique among densely connected network structures, is suitable for extracting a feature map of an image by using information from all previous layers. Here, the RDN shows excellent performance by reusing the learned various characteristic information to make the flow of information flexible. However, in the case of RDN, it has a large number of parameters as a large number of residual dense blocks are stacked, so it takes a lot of time to learn the network, and it is difficult to apply to a mobile system because a large storage space is required.

In order to solve this problem, the present applicant is a single image super-resolution technique using a recursive method of DRRN (Image Super-Resolution via Deep Recursive Residual Network), and RDCAB (Residual Dense Channel Attention Block) using residual density and channel concentration. -RecursiveSRNet We propose a single-image super-resolution network.

In the present invention, in order to deliver the initial feature information to the last layer and to allow subsequent layers to use the input information of the previous layers well, the importance level according to the feature map of the image is utilized by utilizing the residual density structure and the entire area of the image. By using a recursive method while blocking the channel concentration method that determines The network according to the present invention can exhibit similar performance with 1.18 times faster processing speed compared to RDN, and has about 10 times fewer parameters, so that the amount of computation can be greatly reduced.

Hereinafter, a super-resolution network according to the present invention and a single image super-resolution technique using the same will be described in more detail.

2. Conventional super-resolution technique

Various algorithms for single image super-resolution reconstruction technology have been proposed.

Image Super-Resolution Using Deep Convolutional Networks (SRCNN), the first technology to apply deep learning to super-resolution techniques, uses only three convolutions, and shows higher performance than methods that do not apply deep learning. It showed the possibility that deep learning can be applied to super-resolution restoration technology. However, in the case of SRCNN, as it has a pre-up-sampling structure, a large amount of computation is required, and as it has a small number of layers, there is a limit in terms of accuracy.

Accurate Image Super-Resolution Using Very Deep Convolutional Networks (VDSR) improves performance by creating a deep network using residual learning and learning only high-frequency information. There are downsides to using it.

DRRN pointed out that although a deep network can show good performance, it is difficult to tune hyperparameters as the number of parameters increases significantly with each layer increase. DRRN uses global residual learning and local residual learning and a recursive block composed of multiple residual units to create a deep network without additional parameters.

SRDenseNet (Image Super-Resolution Using Dense Skip Connections) is a network applied to super-resolution by using dense connections of DenseNet, and it solves the problem of loss of gradient convergence to 0 as the depth increases and restores performance by connecting the features of all layers. Also improved.

RDN extracts the feature map of the image by utilizing information from all previous layers using the existing residual learning and dense connection used in SRDenseNet to prevent the increase in the amount of computation of the residual block as the size of the synthesis product increases.

3. Single image super-resolution restoration technology according to the present invention

1 is a reference diagram for explaining a super-resolution network (Super-Resolution Networks) according to the present invention. Referring to FIG. 1 , the super-resolution network according to the present invention can be viewed as a structure of RDCAB-RecursiveSRNet in which the depth of the network is increased without additional parameters by using RDCAB in a recursive manner. In the single image super-resolution reconstruction method according to the present invention, a recursive method is applied to reduce the number of parameters, and a residual density structure and a channel concentration technique are applied to make high performance. The single image super-resolution reconstruction method to which RDCAB-RecursiveSRNet according to the present invention is applied may consist of a feature extraction step, a feature extraction step through a recursive block, an upscaling and a reconstruction step.

3.1. Feature extraction step

The super-resolution network (RDCAB-RecursiveSRNet) according to the present invention has a post-upsampling structure in order to prevent an increase in a complex amount of computation. Here, the super-resolution network (RDCAB-RecursiveSRNet) according to the present invention, when a low-resolution image is input, a predefined number (for example, 32 or more, preferably 64) in the first and second synthesis products, respectively ), the output of the first convolution product is used for the last global residual learning, and the output of the second convolution product is configured to be used for local residual learning of RDCAB.

3.2. Feature extraction step through recursive block

The super-resolution network (RDCAB-RecursiveSRNet) according to the present invention is a Residual Dense Channel Attention Block (RDCAB) composed of a residual density structure (RD) and a channel concentration method (CA) in order to reduce the number of parameters as much as possible and maintain high performance. ) is configured to be used recursively. Here, the residual dense structure (RD) utilizes information from all layers through continuous memory structure, local feature combining, and local residual learning, and the channel concentration method (CA) redefines the channels according to the importance between the channels. It has the function of giving.

2 is a reference diagram for explaining the RDCAB structure according to the present invention. 2, the RDCAB according to the present invention is a 3x3 convolutional product of a predefined number (eg, 8) (on the other hand, the more the number of convolutional layers is 12, etc., the better in terms of performance, but the learning time is Since there is a disadvantage of increasing, it can be configured by including the 1x1 convolutional product which is the bottleneck layer proposed by Szegedy et al.) and Szegedy et al. Here, the input of each layer uses a value connecting the outputs of the previous layer, and the output may be set to 64, and the growth rate may be set to 64 to maintain the highest possible performance. As shown in FIG. 2 , RDCAB is repeated as many times as the set number of recursion, and the characteristics of RDCAB that appear each number of times are connected to pass through the 1x1 bottleneck layer once again, and the number of channels can be reduced to 64, thereby reducing the amount of computation.

Here, in order to utilize the information of the entire area, the channel concentration method (CA) that assigns weights according to the importance between each channel can be performed using global average pooling and sigmoid, and A method of dividing and re-multiplying using a reduction rate of 16 is applied, and global residual learning can be performed with the output value passed through the first convolutional product of the network.

3.3. Upscale and restore steps

The up-scaling step is performed by the method proposed by the conventional ESPCN (Efficient Sub-Pixel Convolutional Neural Network) (H x W x (C x r x r) is transformed into rH x rW x C by using shuffle for each channel through sub-pixel convolutional product. , where r means magnification). In the final restoration step, the features obtained through the up-scaling step may be reconstructed into an RGB image through a 3x3 synthesis product.

3. 4. Configuration of single image super-resolution restoration apparatus according to the present invention

Hereinafter, a single image super-resolution restoration apparatus for performing the single image super-resolution restoration technique according to the present invention described above will be described. 3 is a reference diagram for explaining a single image super-resolution restoration apparatus according to the present invention.

Referring to FIG. 3 , the single image super-resolution restoration apparatus 100 is configured to include an image input unit 110 and an image restoration unit 120 , and corresponds to a computing device performing the single image super-resolution restoration technique described above. .

The image input unit 110 receives the low-resolution image and provides it to the super-resolution network.

The image restoration unit 120 generates a high-resolution image obtained by magnifying a low-resolution image by a predefined magnification using super-resolution networks according to the present invention.

Here, the super-resolution network (RDCAB-RecursiveSRNet) according to the present invention is configured such that the RDCAB (FIG. 2) including the residual dense structure (RD) and the channel concentration structure (CA) is recursively repeated a predefined number of times. (see Fig. 1) can be.

In one embodiment, the super-resolution network (RDCAB-RecursiveSRNet) according to the present invention, as shown in Figure 1, a feature extraction unit comprising a first and second convolution (Conv) layers; Recursive Block containing RDCAB and Concat layers; convolution bottleneck (Conv 1x1); a third convolutional layer (Conv); and a restoration unit (UpScale & Reconstruction) connected in series.

In an embodiment, as shown in FIG. 1 , the feature extraction unit extracts a predetermined number of feature maps from each of the first and second convolutional layers, and the output of the first convolutional layer is the second convolutional layer. and the last layer, and the output of the second convolution layer may be configured to be input to the recursive block.

In one embodiment, the recursive block may be configured such that RDCAB is recursively repeated a predefined number of times, and features outputted for each number are connected by a Concat layer.

In one embodiment, as shown in FIG. 2 , the RDCAB is configured by connecting a residual dense structure and a channel concentrated structure in series, wherein the residual dense structure includes a plurality of predefined convolutional layers and a concat layer, and a channel A centralized structure can be constructed including a global average pooling layer and a sigmoid function.

In an embodiment, the reconstructor (UpScale & Reconstruction) may perform an up-scaling process using an Efficient Sub-Pixel Convolutional Neural Network (ESPCN), and convert features generated through the up-scaling process into an RGB image through a synthesis product.

4. Comparative experiment

Hereinafter, comparative experimental results for confirming the performance of the single image super-resolution reconstruction method to which the super-resolution network (RDCAB-RecursiveSRNet) according to the present invention is applied will be described. In the comparison experiment, 800 DIV2K sheets were used as the training data set and Set5 consisting of 5 sheets was used as the validation set. For training, a total of 4,000 images were trained by extracting 5 random 32x32-sized patch units from the input image for training, and then passing random horizontal flip, vertical flip, and 90 degree rotation. Set5, Set14, BSD100, and Urban100 were used as a set, and 2, 3, and 4 times compared to the existing methods using peak signal to noise (PSNR) and structural similarity (SSIM). Comparisons were made at magnification. The loss function used L1 Loss as it showed better performance in L1 Loss than L2 Loss when reconstructing an image. For comparison experiments, GTX 1660 Super GPU and Intel i5-9400F CPU were used as hardware.

Tables 1 and 2 are tables comparing the signal-to-noise ratio, structural similarity, and number of parameters of each network between the existing networks and the network according to the present invention using each test set. It can be confirmed that the single image super-resolution reconstruction method to which the super-resolution network (RDCAB-RecursiveSRNet) is applied according to the present invention exhibits better performance than the existing methods except for RDN. On the other hand, as shown in Table 1, RDN obtained 32.34 dB in the BSD100 test set at 2x magnification, and the restoration method according to the present invention obtained 32.23 dB and made lower performance by 0.11 dB, and 4x magnification Urban100 test In the set, RDN obtained 26.61 dB, and the restoration method according to the present invention obtained 26.18 dB and made performance as low as 0.43 dB, but as shown in Table 2, RDN has 22M number of parameters and the network according to the present invention has 2.1M Taking the number of parameters, it can be seen that the network according to the present invention has about ten times fewer parameters.

4 is a comparison of the performance of the Set5 test set according to the 4x magnification of Table 1 with the existing networks in terms of the number of parameters. Referring to FIG. 3 , the network of the present invention is superior to the existing networks It shows performance, and it can be confirmed that it shows similar performance with fewer parameters than RDN.

Table 3 above is a table comparing the total time taken to learn the network and the performance of the Set14 test set image when processing the image at 4x magnification. The RDN network takes about 25 hours to learn, and the present invention It can be seen that the following network takes about 14 hours. The network according to the present invention exhibits 0.15 dB less performance, but the processing speed is 1.22 times faster than before, and on average, 1.18 times faster in all test sets for 4 times magnification.

Table 4 is a table comparing parameters and performance according to the number of recursions using the Urban100 test set. Here, RB means the number of recursive blocks and RDB means the number of times RDCAB is used recursively. It can be seen that the result of setting the number of recursions to 9 had 24,576 more parameters than the result of setting the number of recursions to 3, but the performance was 0.21 dB higher.

Table 5 shows the performance comparison according to whether the channel concentration scheme configured in the network according to the present invention and global residual learning are used. When all techniques are applied from the 4x magnification Set5 test set, performance is improved compared to when each technique is not used.

5 to 7 show the results of comparing the network according to the present invention with different magnifications (2 times in FIG. 5, 3 times in FIG. 6, and 4 times in FIG. 7) of the network according to the present invention, and in the case of bicubic, all enlarged It can be seen that the magnification is blurred due to the low-frequency characteristics, and in the conventional network at 2x magnification, there is a line between 'y' and 'o' and deterioration exists, whereas the network according to the present invention is 'y' You can see that there is no connection between ' and 'o'. In addition, at 3X magnification, the existing network restores the window frame dividing the upper left window into a curved or unrecognizable window frame, whereas the network according to the present invention restores the window frame of the window dividing the five windows. have. At 4x magnification, it can be seen that the network according to the present invention precisely restores the diagonal line in the network according to the present invention, whereas the existing network is restored in a diagonal pattern opposite to the diagonal direction to float the checker board shape.

The single image super-resolution restoration method according to the present invention described above may be implemented as a computer-readable code on a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices in which data readable by a computer system is stored. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. In addition, the computer-readable recording medium is distributed in a computer system connected through a network, so that the computer-readable code can be stored and executed in a distributed manner.

The above-described preferred embodiments of the present invention have been disclosed for the purpose of illustration, and various modifications, changes, and additions may be made by those skilled in the art with respect to the present invention within the spirit and scope of the present invention, and such modifications, changes and Additions should be considered to fall within the scope of the following claims.

100: single image super-resolution restoration device
110: video input unit
120: image restoration unit

Claims (7)

an image input unit that receives a low-resolution image and provides it to a super-resolution network; and
An image restoration unit for generating a high-resolution image magnified by a predefined magnification by using a predefined super-resolution network (Super-Resolution Networks);
The super-resolution network,
RDCAB (Residual Dense Channel Attention Block) including the residual dense structure and the channel concentration structure is characterized in that it is configured to recursively repeat a predetermined number of times,
Single image super-resolution restoration device.
According to claim 1, wherein the super-resolution network,
a feature extraction unit including first and second convolutional layers; a recursive block including the RDCAB and Concat layers; convolutional bottleneck; a third convolutional layer; and a single image super-resolution restoration device, characterized in that the restoration unit is connected in series.
The method of claim 2, wherein the feature extraction unit,
Each of the first and second convolutional layers extracts a predefined number of feature maps,
The output of the first convolutional layer is input to the second convolutional layer and the last layer,
and the output of the second convolutional layer is configured to be input to the recursive block. Video super-resolution restoration device.
The method of claim 3, wherein the recursive block comprises:
The RDCAB is recursively repeated for a predefined number of times, and the features output for each number are configured to be connected by a Concat layer. Video super-resolution restoration device.
5. The method of claim 4, wherein the RDCAB is
The remaining dense structure and the channel concentration structure are connected in series,
The residual dense structure includes a plurality of predefined convolutional layers and Concat layers,
The channel concentration structure comprises a global average pooling layer and a sigmoid function. Video super-resolution restoration device.
The method of claim 2, wherein the restoration unit,
An up-scaling process is performed using ESPCN (Efficient Sub-Pixel Convolutional Neural Network), and the features obtained through the up-scaling process are reconstructed into an RGB image through a synthesis product. Video super-resolution restoration device.
In the single image super-resolution restoration method performed in the single image super-resolution restoration apparatus,
An image providing unit, receiving a low-resolution image and providing it to a super-resolution network; and
Generating, by an image restoration unit, a high-resolution image magnified by a predefined magnification by using a predefined super-resolution network (Super-Resolution Networks);
The super-resolution network,
RDCAB (Residual Dense Channel Attention Block) including the residual dense structure and the channel concentration structure is characterized in that it is configured to recursively repeat a predetermined number of times,
A single image super-resolution restoration method.

Images