KR102541603B1 - Apparatus and method for generating super-resolution images using display gradient information based on deep learning - Google Patents

Abstract

An apparatus for generating a super-resolution image using deep learning-based display tilt information according to an embodiment of the present invention includes an image information extraction branch for extracting image information from a low-resolution image; a gradient information extraction branch for extracting gradient information from the low-resolution image; a first adder for adding an output of the image information extraction branch and an output of the gradient information extraction branch; a deconvolution unit configured to deconvolve an output of the first adder; an upsampling unit for upsampling the low resolution image; and a second adder for forming a super-resolution image by adding an output of the deconvolution unit and an output of the upsampling unit, wherein the gradient information extraction branch includes intermediate image information of the image information extraction branch and a gradient of the low-resolution image. The gradient information is extracted based on the feature.

Description

Apparatus and method for generating super-resolution images using deep learning-based display gradient information

The present invention relates to an apparatus and method for generating a super-resolution image using deep learning-based display tilt information.

Single image super-resolution (SISR), a classic problem in low-level computer vision tasks, aims to reconstruct a high-resolution (HR) image from a low-resolution (LR) image. .

SISR can be used for a variety of image processing projects such as surveillance, medical and satellite imagery. SISR is a well-known and poorly established problem because the HR solutions corresponding to LR images may differ.

In recent years, many methods have been proposed in the literature on SISR research, including interpolation-based methods, reconstruction-based methods, and example-based methods. Because the recovery performance of the first two methods often decreases significantly at larger scale factors, example-based methods have been widely used with deep learning developments aimed at learning prior knowledge from LR and HR pairs.

Today, with the development of deep learning, such as deep convolutional neural networks (CNNs), many SR methods based on deep learning aim to train deep networks to obtain clearer reconstruction details. Despite achieving significant performance, most deep networks still have some drawbacks.

First of all, to get better performance, researchers prefer to use deeper and wider networks. However, these methods obviously require high computational cost and memory consumption. It is not suitable for real-world applications such as mobile and embedded systems, limiting the widespread use of SISRs. Also, most networks are only pursuing depth deepening without making full use of the image's rich geometric structure information. This tends to make the edges of the reconstructed SR image too smooth and blurry. Such reconstruction results are clearly not good enough.

Deep convolutional neural networks (CNNs) have played an increasingly important role in single image super-resolution (SR). However, as the depth and breadth of networks increase, super-resolution methods based on convolutional neural networks face training problems, memory consumption, slow execution and other problems. Moreover, most methods do not make sufficient use of image tilt information leading to loss of image geometry information.

KR 10-2077215 B1

Hui, Z., Wang, X., Gao, X., "Fast and accurate single image super-resolution via information distillation network" CVPR (2018)

The problem to be solved by the present invention is to provide a super-resolution image generation device and method using deep learning-based display tilt information that can sharpen edge details of super-resolution images and reduce calculation and execution time. .

A super-resolution image generating device using deep learning-based display tilt information according to an embodiment of the present invention for solving the above problems is,

an image information extraction branch for extracting image information from a low-resolution image;

a gradient information extraction branch for extracting gradient information from the low-resolution image;

a first adder for adding an output of the image information extraction branch and an output of the gradient information extraction branch;

a deconvolution unit configured to deconvolve an output of the first adder;

an upsampling unit for upsampling the low resolution image; and

A second addition unit for forming a super-resolution image by adding the output of the deconvolution unit and the output of the upsampling unit;

The gradient information extraction branch extracts the gradient information based on intermediate image information of the image information extraction branch and gradient characteristics of the low-resolution image.

In the super-resolution image generating apparatus using deep learning-based display tilt information according to an embodiment of the present invention, the image information extraction branch,

a feature map extraction module for extracting a feature map from the low-resolution image; and

first to nth image information extraction modules connected in series for receiving the feature map output from the feature map extraction module and outputting the image information, wherein n is an integer greater than or equal to 2;

The intermediate image information of the image information extraction branch may be outputs of the first through n-th imager information extraction modules.

In addition, in the super-resolution image generating apparatus using deep learning-based display tilt information according to an embodiment of the present invention, the tilt information extraction branch,

a gradient map extraction module for extracting a gradient map from the low-resolution image;

a gradient feature extraction module for extracting a gradient feature from the gradient map output from the gradient map extraction module;

a first addition module configured to add a gradient feature output from the gradient feature extraction module and an output of the first image information extraction module;

a first gradient information extraction module configured to receive an output of the first addition module and output gradient information;

second to n-th gradient information extraction modules serially connected to the first gradient information extraction module; and

And an M-th addition module for adding the output of the (M-1)th gradient information extraction module and the output of the M-th image information extraction module,

The output of the nth addition module is input to the nth gradient information extraction module,

M is an integer from 2 to n, and n may be an integer of 2 or more.

In addition, in the apparatus for generating super-resolution images using deep learning-based display tilt information according to an embodiment of the present invention, the first to nth image information extraction modules and the first to nth tilt information extraction modules are respectively ,

an enhancement module that includes two shallow convolutions; and

and a compression module coupled to the output of the enhancement module.

In addition, in the super-resolution image generating apparatus using deep learning-based display tilt information according to an embodiment of the present invention, the enhancement module,

A first 3×3 convolution module receiving an input signal;

a slicing module for slicing the output of the first 3×3 convolution module into two signals;

a second 3×3 convolution module receiving the first output of the slicing module;

a connection module connecting the second output of the slicing module and the input signal; and

An addition module for adding the output of the second 3 × 3 convolution module and the output of the convolution module,

The compression module may include a 1×1 convolution module.

A super-resolution image generation method using deep learning-based display tilt information according to an embodiment of the present invention for solving the above problems is,

(A) an image information extraction branch extracting image information from a low-resolution image;

(B) a gradient information extraction branch extracting gradient information from the low-resolution image;

(C) adding, by a first adder, an output of the image information extraction branch and an output of the gradient information extraction branch;

(D) deconvoluting the output of the first adder by a deconvolution unit;

(E) upsampling the low resolution image by an upsampling unit; and

(F) forming, by a second adder, a super-resolution image by adding the output of the deconvolution unit and the output of the upsampling unit;

The gradient information extraction branch extracts the gradient information based on intermediate image information of the image information extraction branch and gradient characteristics of the low-resolution image.

In the super-resolution image generation method using deep learning-based display tilt information according to an embodiment of the present invention, the image information extraction branch,

a feature map extraction module for extracting a feature map from the low-resolution image; and

first to nth image information extraction modules connected in series for receiving the feature map output from the feature map extraction module and outputting the image information, wherein n is an integer greater than or equal to 2;

The intermediate image information of the image information extraction branch may be outputs of the first through n-th imager information extraction modules.

In addition, in the super-resolution image generation method using deep learning-based display tilt information according to an embodiment of the present invention, the tilt information extraction branch,

a gradient map extraction module for extracting a gradient map from the low-resolution image;

a gradient feature extraction module for extracting a gradient feature from the gradient map output from the gradient map extraction module;

a first addition module configured to add a gradient feature output from the gradient feature extraction module and an output of the first image information extraction module;

a first gradient information extraction module configured to receive an output of the first addition module and output gradient information;

second to n-th gradient information extraction modules serially connected to the first gradient information extraction module; and

And an M-th addition module for adding the output of the (M-1)th gradient information extraction module and the output of the M-th image information extraction module,

The output of the nth addition module is input to the nth gradient information extraction module,

M is an integer from 2 to n, and n may be an integer of 2 or more.

In addition, in the super-resolution image generation method using deep learning-based display tilt information according to an embodiment of the present invention, the first to nth image information extraction modules and the first to nth tilt information extraction modules are respectively ,

an enhancement module that includes two shallow convolutions; and

and a compression module coupled to the output of the enhancement module.

In addition, in the super-resolution image generation method using deep learning-based display tilt information according to an embodiment of the present invention, the enhancement module,

A first 3×3 convolution module receiving an input signal;

a slicing module for slicing the output of the first 3×3 convolution module into two signals;

a second 3×3 convolution module receiving the first output of the slicing module;

a connection module connecting the second output of the slicing module and the input signal; and

An addition module for adding the output of the second 3 × 3 convolution module and the output of the convolution module,

The compression module may include a 1×1 convolution module.

According to an apparatus and method for generating a super-resolution image using deep learning-based display tilt information according to an embodiment of the present invention, edge details of a super-resolution image can be made clearer and calculation and execution time can be reduced.

In addition, according to the super-resolution image generation device and method using deep learning-based display tilt information according to an embodiment of the present invention, a tilt feature is extracted from a low-resolution tilt map and geometric structure information of an image is converted into super-resolution It provides better performance than existing super-resolution methods in terms of parameter quantity and computing speed.

1 is a diagram showing a super-resolution image generating apparatus using deep learning-based display tilt information according to an embodiment of the present invention.
2 is a diagram showing detailed blocks of each of the first to n th image information extraction modules and the first to n th gradient information extraction modules shown in FIG. 1;
3 is a flowchart of a super-resolution image generation method using deep learning-based display tilt information according to an embodiment of the present invention.
4 is a diagram illustrating a device capable of executing a super-resolution image generation method using deep learning-based display tilt information according to an embodiment of the present invention.
5 is a diagram comparing a super-resolution image generation method using display tilt information based on deep learning according to an embodiment of the present invention and existing methods.
6 is a diagram comparing a super-resolution image generation method using deep learning-based display tilt information according to an embodiment of the present invention and conventional methods in relation to parameters and PSNR for a Set5 data set for 2x SR.
7 is a diagram comparing a super-resolution image generation method using deep learning-based display tilt information according to an embodiment of the present invention and conventional methods in relation to parameters and PSNR for a Set5 data set for 4x SR.
8 is a visual representation of GIDN-L and GIDN and existing state-of-the-art methods according to a super-resolution image generation method using deep learning-based display tilt information according to an embodiment of the present invention for 4x SR for BSD100 and Urban100 data sets. A drawing showing the comparison.

Objects, specific advantages and novel features of the present invention will become more apparent from the following detailed description and preferred embodiments taken in conjunction with the accompanying drawings.

Prior to this, the terms or words used in this specification and claims should not be interpreted in a conventional and dictionary sense, and the inventor may appropriately define the concept of the term in order to explain his or her invention in the best way. It should be interpreted as a meaning and concept consistent with the technical spirit of the present invention based on the principles.

In adding reference numerals to components of each drawing in this specification, it should be noted that the same components have the same numbers as much as possible, even if they are displayed on different drawings.

In addition, terms such as “first”, “second”, “one side”, and “other side” are used to distinguish one component from another component, and the components are limited by the terms. It is not.

Hereinafter, in describing the present invention, detailed descriptions of related known technologies that may unnecessarily obscure the subject matter of the present invention will be omitted.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

In the present invention, we propose a gradient information extraction network. On the one hand, the advantage of fast and lightweight processing is maintained through information extraction.

SR performance is improved by gradient information. The network of the present invention has two branches: a gradient information distillation branch (GIDB) and an image information distillation branch (IIDB).

To combine the functions of the two branches, we also introduce a residual feature transfer mechanism (RFT). Under the functions of GIDB and RFT, an apparatus and method for generating super-resolution images using deep learning-based display tilt information according to an embodiment of the present invention provide rich geometric structure information that can make edge details of a reconstructed image clearer. can keep

Experimental results show that the super-resolution image generation method using deep learning-based display tilt information according to an embodiment of the present invention is superior to existing methods, but limits model parameters, calculation, and execution time. A super-resolution image generation method using deep learning-based display tilt information according to an embodiment of the present invention provides the possibility of real-time image processing and mobile application.

In the super-resolution image generation method using display gradient information based on deep learning according to an embodiment of the present invention, a gradient information distillation network (GIDN) is proposed. GIDN is controlled to avoid significant increases in the amount of parameters and calculations in order to maintain its fast and lightweight advantage.

An apparatus and method for generating super-resolution images using deep learning-based display gradient information according to an embodiment of the present invention include a gradient information distillation branch (GIDB) and an image information distillation branch (IIDB) includes

GIDB takes a gradient map of an image as input. At the same time, the tilt information is used as a guide to restore the super-resolution of the image. To bridge the two, we also introduce a residual feature transfer mechanism. This mechanism can add geometry information from the same rich images in IIDB to training in GIDB. As long as the edges of the image can be reconstructed clearly through rich geometric structure information, the SR process is regarded as a color filling operation that the LR image can produce strongly.

The main contribution of the super-resolution image generation method using deep learning-based display tilt information according to an embodiment of the present invention can be summarized as follows.

1. The super-resolution image generation method using deep learning-based display tilt information according to an embodiment of the present invention explicitly proposes the concept of extracting tilt information for the first time, and divides the distillation information into an image super-resolution convolutional neural network applied to. The present invention has the advantage of being fast and lightweight.

2. A residual feature transmission mechanism is applied to the network. In our network, advanced residual features of LR images are added to the training network of gradient residuals. And the gradient feature enhancement is guided by the LR image to enhance the effect of the gradient information extraction branch.

Experimental results show that the super-resolution image generation method using deep learning-based display tilt information according to an embodiment of the present invention can effectively improve the performance of super-resolution images on the premise of fast and accurate guarantee.

An object of the present invention is to reduce geometric distortion generated by a deep convolutional neural network (CNN)-based method by using a gradient map as a structure guide and obtaining sufficient gradient edge information through a lightweight and fast gradient information extraction branch. As a result, the PSNR value of the reconstructed image can be improved. The goal of the present invention is to further improve the CNN-based SR method using gradient information extraction branching.

outline

An apparatus and method for generating a super-resolution image using deep learning-based display tilt information according to an embodiment of the present invention selects an information distillation network (IDN) as a reference line. As a result, the present invention proposes a network that extracts gradient information to improve performance. Since most super-resolution convolutional neural networks do not fully utilize the gradient information of images, the extraction of gradient information is helpful for high-resolution reconstruction of image edges.

In the full network, the gradient map is first extracted from a low-resolution (LR) image. Then, feature extraction and cumulative information extraction are performed in the gradient information distillation branch (GIDB). Then, the gradient residuals of the gradient information extraction branch (GIDB) and the image residuals of the Image Information Distillation Branch (IIDB) are added together for image reconstruction. Finally, a high-resolution (HR) image is created by adding upsampling.

1 is a diagram illustrating an apparatus for generating a super-resolution image using deep learning-based display tilt information according to an embodiment of the present invention.

Referring to FIG. 1, an apparatus for generating a super-resolution image using deep learning-based display tilt information according to an embodiment of the present invention includes an image information extraction branch 100 for extracting image information from a low-resolution (LR) image, A gradient information extraction branch 102 for extracting gradient information from the low-resolution image, a first adder 104 for adding an output of the image information extraction branch 100 and an output of the gradient information extraction branch 102 , the deconvolution unit 106 for deconvoluting the output of the first adder 104, the upsampling unit 108 for upsampling the low-resolution image, and the output of the deconvolution unit 106 and the and a second adder 110 for adding the outputs of the upsampling unit 108 to form a super-resolution image.

The gradient information extraction branch 102 extracts the gradient information based on the intermediate image information of the image information extraction branch 100 and the gradient characteristics of the low-resolution image. Referring to FIG. 1, the image information extraction branch 100 The intermediate image information of ) is input to gradient residual training to help the gradient information extraction branch 102 obtain better image information.

The image information extraction branch 100 receives the feature map output from the feature map extraction module 112 and the feature map extraction module 112 for extracting a feature map from the low-resolution image and outputs the image information. and first to n-th image information extraction modules 114_1 to 114_n connected in series, wherein n is an integer greater than or equal to 2, and the intermediate image information of the image information extraction branch 100 is the first to n-th image information extraction modules. This is the output of the imager information extraction modules 114_1 to 114_n. The first to n th image information extraction modules 114_1 to 114_n connected in series are referred to as information extraction blocks.

The gradient information extraction branch 102 includes a gradient map extraction module 116 for extracting a gradient map from the low-resolution image and a gradient feature for extracting a gradient feature from the gradient map output from the gradient map extraction module 116. An extraction module 118, a first addition module 122_1 for adding the gradient feature output from the gradient feature extraction module 118 and the output of the first image information extraction module 114_1, the first addition module ( A first tilt information extraction module 120_1 for receiving the output of 122_1 and outputting tilt information, and second to nth tilt information extraction modules 120_2 to 120_n serially connected to the first tilt information extraction module 120_1 ), and the Mth addition module for adding the outputs of the (M−1)th gradient information extraction modules 120_1 to 120_(n−1) and the outputs of the Mth image information extraction modules 114_2 to 114_n ( 122_2 to 122_n). The output of the n-th addition module 122_n is input to the n-th gradient information extraction module 120_n, where M is an integer from 2 to n, and n is an integer greater than or equal to 2.

Meanwhile, FIG. 3 is a flowchart of a super-resolution image generation method using deep learning-based display tilt information according to an embodiment of the present invention.

1 and 3, in the super-resolution image generation method using deep learning-based display tilt information according to an embodiment of the present invention, in step S300, the image information extraction branch 100 performs a low-resolution (LR) image Extract image information from

In step S302, the tilt information extraction branch 102 extracts tilt information from the low-resolution (LR) image.

In step S304, the first adder 104 adds the output of the image information extraction branch 100 and the output of the gradient information extraction branch 102.

In step S306, the deconvolution unit 106 deconvolves the output of the first addition unit 104.

In step S308, the upsampling unit 108 upsamples the low resolution image.

In step S310, the second adder 110 adds the output of the deconvolution unit 106 and the output of the upsampling unit 108 to form a super-resolution (SR) image.

Hereinafter, an apparatus and method for generating a super-resolution image using deep learning-based display tilt information according to an embodiment of the present invention will be described in detail.

Obtaining the Gradient Map

An apparatus for generating a super-resolution image using deep learning-based display tilt information according to an embodiment of the present invention is composed of an image information extraction branch 100 and a tilt information extraction branch 102 as shown in FIG. 1 . It pre-performs two operations on the LR image at the same time. Meanwhile, in order to reconstruct a high resolution (HR) image, the upsampling unit 108 upsamples the low resolution (LR) image by bicubic. On the other hand, the gradient map is extracted from the low resolution (LR) image by the gradient map extraction module 116 .

As shown in Equation 1, the gradient map is obtained by calculating the difference between adjacent pixels.

Here, G(*) denotes the gradient map extraction operation, and the element of the gradient map is the gradient length of coordinates A = (a, b) pixels. The operation to obtain the gradient can be easily implemented through a convolutional layer using a fixed kernel. At the same time, the gradient direction information is ignored because the gradient strength is sufficient to reflect the sharpness of the local area in the reconstructed image. Therefore, the intensity map is used as the gradient map.

In particular, we define x as the input of the gradient information extraction branch 102 and y as the super-resolution (SR) image that is the output of the entire system. First, the gradient map is extracted as in Equation 2.

Here, x _g represents the gradient map of the low-resolution (LR) image input by the G(*) operation.

Branch image information extraction

In the apparatus and method for generating a super-resolution image using deep learning-based display tilt information according to an embodiment of the present invention, residual features of a low-resolution (LR) image are extracted using fast and accurate branching of low-resolution (LR) image processing. In particular, an information extraction network is used. A branch is two parts.

The first is the feature extraction block (FBlock) 112, which extracts the feature map of the original low resolution (LR) image using two 3x3 convolutional layers. This process can be expressed as:

Here, F(x) denotes a feature extraction function, x _i0 denotes a feature extracted in the image information extraction branch 100, and is applied as an input to the next step.

The next part is a chain structure composed of a plurality of information extraction blocks (DBlocks) 114_1 to 114_n and 120_1 to 120_n to which the gradient information extraction branch 102 and the image information extraction branch 100 are simultaneously applied.

As shown in FIG. 2, information extraction modules (DBlocks) 114_1 to 114_n and 120_1 to 120_n include an enhancement module 200 that includes two shallow convolutions and a compression module connected to the output of the enhancement module 200. (202).

The enhancement module 200 includes two shallow convolutional networks, the main purpose of which is to improve the expressiveness of the network. In the case of compressed units, the redundant information of the enhanced unit function is compressed into a simple convolutional layer.

The enhancement module 200 includes a first 3 × 3 convolution module 204 receiving an input signal and a slicing module 206 for slicing the output of the first 3 × 3 convolution module 204 into two signals , a second 3 × 3 convolution module 210 receiving the first output of the slicing module, a connection module 208 connecting the second output of the slicing module 206 and the input signal, and the second 3 and an addition module 212 for adding the output of the ×3 convolution module 210 and the output of the concatenation module 208, and the compression module 202 includes a 1 × 1 convolution module.

In the image information extraction branch 100, this process can be described as follows.

과

은 각각 이미지 정보 추출 분기(100)에서 n번째 정보 추출 모듈(DBlock)의 출력과 입력을 나타낸다.Here, D _n (*) represents a function of the n-th information extraction module (DBlock) of the image information extraction branch 100.

class

represents the output and input of the nth information extraction module (DBlock) in the image information extraction branch 100, respectively.

Gradient Information Extraction Branch

In fact, the generated gradient map can be considered as a different kind of image, so we can learn the mapping between the two modes using image-to-image conversion techniques. The essence of this transformation is a spatial distribution transformation from LR contour geometries to HR contour geometries.

Since most of the area values in the gradient map are near zero, the convolutional neural network can pay more attention to the geometric information of the image contours. It is easier to capture the structural correlation of images and provide structural information for SR images. Therefore, gradient features are extracted from the gradient map using the gradient feature extraction module 118 .

As in the image information extraction branch 100, this process can be described as follows.

은 기울기 정보 추출 분기(102)에서 추출한 특징을 나타낸다.where F(*) denotes the feature extraction function

represents the feature extracted in the gradient information extraction branch 102.

Then, the following operations are performed through the first to n th image information extraction modules 120_1 to 120_n and the residual feature transmission mechanism. The experimental results show that the features extracted from the gradient map play an important role in the final SR reconstruction.

Residual feature transfer mechanism

Since the image information extraction branch 100 can deliver rich structural information of the image, the Image Information Distillation Branch (IIDB) 100 The residual feature of is important.

This is important for the gradient information extraction branch 102 to obtain gradient features. Therefore, as shown in FIG. 1, the present invention transmits residual features (RFT: Residual Feature Transfer).

In the present invention, by guiding the training of the gradient information of the gradient information extraction branch 102 by the image residual feature of the image information extraction branch 100, the SR gradient feature can be easily obtained. In this case, the efficiency of the gradient information extraction branch 102 can be improved by using image features without increasing the parameters.

Since the residual feature transmission mechanism and the gradient information extraction modules 120_1 to 120_n are at the same time, the corresponding procedure of the gradient information extraction branch 102 is formulated as follows.

은 기울기 정보 추출 분기(102)에서 n번째 정보 추출 모듈(DBlock)의 출력을 나타낸다.Like the image information extraction branch 100, D _n (*) represents a function of the n-th information extraction module (DBlock) of the tilt information extraction branch 102.

represents the output of the n-th information extraction module (DBlock) in the tilt information extraction branch 102.

Finally, the first adder 104 adds the results of the image information extraction branch 100 and the gradient information extraction branch 102 . The sum value is input to the deconvolution unit 106, which is a reconstruction block in which the convolution is transposed by the activation function.

An output (y) of the super-resolution image generating device using deep learning-based display tilt information according to an embodiment of the present invention is formed as in Equation 7.

Here, R(*) represents a reconstruction function (RBlock) by the deconvolution unit 106, and B(*) represents a bicubic upsampling operation of the input low resolution (LR) image by the upsampling unit 108.

loss function

The main purpose of introducing the loss function is to predict and minimize the difference between the real image and the reconstructed HR image. In the present invention, the mean square error is used as the loss function. The loss function can be expressed as:

는 실제 이미지를 나타내고 S(*)는 매핑 함수를 나타내는데, 이 매핑 함수는 저해상도(LR) 이미지 훈련에서 해당 초해상도(SR) 이미지를 재구성한다. N은 훈련 패치의 수를 나타낸다.here

denotes the real image and S(*) denotes the mapping function, which reconstructs the corresponding super-resolution (SR) image from low-resolution (LR) image training. N represents the number of training patches.

Meanwhile, a super-resolution image generation method using deep learning-based display tilt information according to an embodiment of the present invention may be executed using the device shown in FIG. 4 .

Referring to FIG. 4 , an apparatus in which a super-resolution image generation method using deep learning-based display tilt information according to an embodiment of the present invention is executed includes a processor 400 and a memory 402 .

The memory 402 stores data and various data in the form of program codes for a super-resolution image generation method using display tilt information based on deep learning according to an embodiment of the present invention.

The processor 400 performs a super-resolution image generation method using deep learning-based display tilt information according to an embodiment of the present invention stored in the memory 402 according to data stored in the form of program codes. run

In addition, the processor 400 is based on a super-resolution image formed by a super-resolution image generation method using deep learning-based display tilt information according to an embodiment of the present invention, a ground truth image, and a loss function , The convolutional neural network shown in FIG. 1 is trained to optimally determine the parameters of filters used for convolution, thereby reconstructing an optimal super-resolution (SR) image from a low-resolution (LR) image.

Experiment

The present invention reports many details of the experiments. It is then compared with previous superior methods. Then, the present invention introduces the ablation experiment of the present invention in two aspects of RFT and gradient information distillation branch (GIDB), respectively. Finally, we conclude the experimental performance and contribution.

training data set

The training data set of this experiment contains 800 high-quality training images and DIV2K, which is widely used in the current study, was selected. We downsampled the high-resolution image with bicubic to get the low-resolution input. At the same time, to make full use of the training data, we reduce the images by a factor of 0.9/0.8/0.7/0.6 and generate 10 series images from the native images as training using the horizontal flipping method. The data enhancement method of 90 degree rotation is performed prior to training.

Data set test

To evaluate the SR performance of the model, the average peak signal-to-noise (PSNR) and structure similarity index (SSIM) were calculated based on five standard benchmarks: Set5, Se14, BSD100, Urban100 and Manga109. ) was selected. For a fair comparison with previous studies, we evaluate the performance results only in the luminance (Y) channel in the transformed YCbCr space.

training options

At each training iteration, we extract 16 RGB low-resolution image (LR) patches as network inputs. For a 2x upscaling factor, the LR size is 96×96. When the image super-resolution magnification is 3x, the LR size is 64x64. At the same time, in the case of 4x, it corresponds to 48 × 48. Therefore, the size of the Ground-Truth High Resolution Image (HR) patch is 192×192.

The ADAM optimizer with the settings beta1 = 0.9, beta2 = 0.999 is applied to the parameter update. We set the initial learning rate to 0.0001. Considering the balance between recovery performance and execution time, we propose a fully structured network with 8 DBlocks in a total of 61 layers. To reduce the parameters, we also designed a lightweight network consisting of 4 DBlocks with a total of 33 layers.

We implemented the model by PyTorch and trained it on a single NVIDIA 1660 GPU. It is worth mentioning that the training process of the present invention does not require any special weight initialization or regularization techniques or other tricks.

model details

The present invention proposes two models, a complete version and a lightweight version. Their difference is that a different number of DBlocks are used. Experimental results show that both models obtained satisfactory results. It can be seen in Figures 6 and 7.

Based on the IDN, we propose a complete model called the Gradient Information Distillation Network (GIDN), which uses four DBlocks with the same number as the IDN in GIDB. We also use the residual feature transfer operation between the two branches to get good results. On the other hand, we propose a lighter version GIDN-L to better ensure the lightweight and fast advantages of IDN and better demonstrate the role of the proposed GIDN. This model reduces the number of DBlocks in GIDB to two. Also, only two DBlocks are used in IIDB to ensure consistency with the IDN in the total number of DBlocks. The results show that this lightweight version also has a significant performance improvement.

Compare and analyze

As shown in Fig. 5, the present invention was compared with major state-of-the-art methods by PSNR and SSIM indicators. As shown in Fig. 5, it can be seen that the method according to the present invention better reconstructs the structural details of the rail than the existing methods.

quantitative comparison

We quantitatively compare it with other excellent SR methods through PSNR and SSIM metrics. These include bicubic, SRCNN, and FSRCNN. Table 1 reflects the effect of the method according to the present invention in five benchmarks (Set5, Set14, etc.) and three sizes of upscaling factors (2, 3, and 4).

Not only the performance of the network model (PSNR and SSIM) is shown in Table 1, but also reflects the model complexity as well as parameter quantities. Finally, it can be seen that the two models proposed by the present invention obtained the best results in most cases.

In addition, in GIDN-L in Table 1, compared to IDN, only a very small number of parameters were increased, but the performance was greatly improved. Therefore, the GIDN method proposed by the present invention achieves excellent SR performance when securing lightweight parameters and significantly outperforms other methods such as IDN in large-scale cases.

subjective comparison

8, it can be seen that the method according to the present invention can obtain the best performance. In the case of 8023 of BSD100, the method according to the present invention clearly restored marks such as stripes without artifacts. In the case of 86000 of BSD100, both of the present invention can clearly express the three window shapes of a building without distortion. On the other hand, in the img059 of Urban100, the method according to the present invention can clearly see the lines in the building. What is clearer from img083 is that the method according to the present invention retains the shape of the structure well, while other methods deform and destroy the texture of the ceiling. Overall, compared with other methods with different degrees of misinformation, the method according to the present invention can obtain clearer contours and no serious artifacts.

Ablation experiments

A series of ablation experiments were conducted to individually confirm the effectiveness of the method according to the present invention. Then, the details of the experiment are introduced. The experimental results are shown in Table 2.

GIDB's Ablation Study

To prove the effectiveness of gradient map and gradient information extraction branch (GIDB), the following ablation experiment was performed.

To make the parameters match the IDN as much as possible, we propose a lightweight version of the GIDN-L network that reduces the number of DBlocks in GIDB and IIDB to two. The number of DBlocks is 4 which matches the IDN. Experiments prove that the performance of the present invention outperforms IDN and that GIDB exerts an adequate effect.

Meanwhile, we design another model that replaces the GIDB input with the same LR image as the IIDB input after canceling the GIDN contrast gradient map extraction task. The name of this model is GIDN-C.

Finally, as shown in Table 2 below, we can conclude that gradient map and GIDB achieve satisfactory performance results.

Ablation studies of RFT

Design a network structure using only GIDB and IIDB and no residual feature transfer (RFT) called GIDN-A. It is compared with the complete model of the present invention. After comparing the experiments, it can be concluded that the RFT mechanism of the full model performs its function and affects the improvement of PSNR and SSIM values.

Comparison of resection studies

This section presents a comparison of methods on the x4 scale including IDN and the model described above. The results show that the GIDN according to the present invention achieved satisfactory results in both PSNR and SSIM indicators. GIDN-L also offers significant performance improvements over IDN. At the same time, MAC values are used to represent the computational cost of complex multiplication and accumulation operations on a single image. MAC is calculated assuming that the resolution of the generated SR image is 720P (1280 × 720).

As can be seen in Table 2, it is demonstrated that the method according to the present invention achieves excellent performance without significantly increasing the number of parameters and computation.

Experiment summary

The above experiments show that the method proposed by the present invention achieves strong competitive results in PSNR and SSIM. Compared to many large model networks simultaneously among the five benchmark datasets, it is far ahead of other existing advanced methods in terms of parameter quantity and speed of operation. Comparing the reconstructed SR images, satisfactory results were obtained in detail recovery by the method of the present invention. Finally, ablation experiments prove that the contribution of the present invention has played a significant role in improving the super-resolution effect of a single image.

conclusion

In the present invention, we propose a new network called a gradient information distillation network (GIDN), including a lightweight version (GIDN-L) based on IDN. The network extracts rich gradient information through fast and accurate gradient information extraction branches. It is then used to guide the reconstruction of high-resolution images. The gradient information extraction branch aims to extract gradient features from low-resolution gradient maps and provide geometric structural information at super-resolution with unambiguous structural guidelines. The method achieved strong competitive results in terms of PSNR and SSIM on five benchmark datasets. At the same time, compared with many large model networks, the method according to the present invention is far ahead of many advanced methods in terms of parameter quantity and computing speed. This gradient information extraction concept will be more widely used in real-time practice.

Although the present invention has been described in detail through specific examples, this is for explaining the present invention in detail, the present invention is not limited thereto, and within the technical spirit of the present invention, those skilled in the art It will be clear that the modification or improvement is possible by

All simple modifications or changes of the present invention fall within the scope of the present invention, and the specific protection scope of the present invention will be clarified by the appended claims.

100: image information extraction branch 102: tilt information extraction branch
104: first addition unit 106: deconvolution unit
108: upsampling unit 110: second addition unit
112: feature map extraction module
114_1 to 114_n: first to nth image information extraction modules
116: gradient map extraction module 118: gradient feature extraction module
120_1 to 120_n: 1st to nth gradient information extraction modules
122_1 to 122_n: first to nth addition modules
200: enhancement module 202: compression module
204: first 3 × 3 convolution module 206: slicing module
208: connection module 210: second 3 × 3 convolution module
400: processor 402: memory

Claims (10)

an image information extraction branch for extracting image information from a low-resolution image;
a gradient information extraction branch for extracting gradient information from the low-resolution image;
a first adder for adding an output of the image information extraction branch and an output of the gradient information extraction branch;
a deconvolution unit configured to deconvolve an output of the first adder;
an upsampling unit for upsampling the low resolution image; and
A second addition unit for forming a super-resolution image by adding the output of the deconvolution unit and the output of the upsampling unit;
The gradient information extraction branch extracts the gradient information based on intermediate image information of the image information extraction branch and gradient characteristics of the low-resolution image;
The image information extraction branch,
a feature map extraction module for extracting a feature map from the low-resolution image; and
First to n-th image information extraction modules connected in series for receiving the feature map output from the feature map extraction module and outputting the image information;
The intermediate image information of the image information extraction branch is an output of the first to n th imager information extraction modules;
The gradient information extraction branch,
a gradient map extraction module for extracting a gradient map from the low-resolution image;
a gradient feature extraction module for extracting a gradient feature from the gradient map output from the gradient map extraction module;
a first addition module configured to add a gradient feature output from the gradient feature extraction module and an output of the first image information extraction module;
a first gradient information extraction module configured to receive an output of the first addition module and output gradient information;
second to n-th gradient information extraction modules serially connected to the first gradient information extraction module; and
And second to nth addition modules for adding outputs of each of the first to (n-1)th gradient information extraction modules and outputs of each of the second to nth image information extraction modules,
The output of the nth addition module is input to the nth gradient information extraction module,
wherein n is an integer greater than or equal to 2;
The first to nth image information extraction modules and the first to nth gradient information extraction modules, respectively,
an enhancement module that includes two shallow convolutions; and
a compression module coupled to the output of the enhancement module;
The enhancement module,
A first 3×3 convolution module receiving an input signal;
a slicing module for slicing the output of the first 3×3 convolution module into two signals;
a second 3×3 convolution module receiving the first output of the slicing module;
a connection module connecting the second output of the slicing module and the input signal; and
An addition module for adding the output of the second 3 × 3 convolution module and the output of the convolution module,
The compression module includes a 1 × 1 convolution module, a super-resolution image generating device using deep learning-based display tilt information. delete delete delete delete (A) an image information extraction branch extracting image information from a low-resolution image;
(B) a gradient information extraction branch extracting gradient information from the low-resolution image;
(C) adding, by a first adder, an output of the image information extraction branch and an output of the gradient information extraction branch;
(D) deconvoluting the output of the first adder by a deconvolution unit;
(E) upsampling the low resolution image by an upsampling unit; and
(F) forming, by a second adder, a super-resolution image by adding the output of the deconvolution unit and the output of the upsampling unit;
The gradient information extraction branch extracts the gradient information based on intermediate image information of the image information extraction branch and gradient characteristics of the low-resolution image;
The image information extraction branch,
a feature map extraction module for extracting a feature map from the low-resolution image; and
First to n-th image information extraction modules connected in series for receiving the feature map output from the feature map extraction module and outputting the image information;
The intermediate image information of the image information extraction branch is an output of the first to n th imager information extraction modules;
The gradient information extraction branch,
a gradient map extraction module for extracting a gradient map from the low-resolution image;
a gradient feature extraction module for extracting a gradient feature from the gradient map output from the gradient map extraction module;
a first addition module configured to add a gradient feature output from the gradient feature extraction module and an output of the first image information extraction module;
a first gradient information extraction module configured to receive an output of the first addition module and output gradient information;
second to n-th gradient information extraction modules serially connected to the first gradient information extraction module; and
And second to nth addition modules for adding outputs of each of the first to (n-1)th gradient information extraction modules and outputs of each of the second to nth image information extraction modules,
The output of the nth addition module is input to the nth gradient information extraction module,
wherein n is an integer greater than or equal to 2;
The first to nth image information extraction modules and the first to nth gradient information extraction modules, respectively,
an enhancement module that includes two shallow convolutions; and
a compression module coupled to the output of the enhancement module;
The enhancement module,
A first 3×3 convolution module receiving an input signal;
a slicing module for slicing the output of the first 3×3 convolution module into two signals;
a second 3×3 convolution module receiving the first output of the slicing module;
a connection module connecting the second output of the slicing module and the input signal; and
An addition module for adding the output of the second 3 × 3 convolution module and the output of the convolution module,
The compression module includes a 1 × 1 convolution module, a super-resolution image generation method using deep learning-based display tilt information. delete delete delete delete

Images