AU2020100200A4

AU2020100200A4 - Content-guide Residual Network for Image Super-Resolution

Info

Publication number: AU2020100200A4
Application number: AU2020100200A
Authority: AU
Inventors: Shuying Huang; Zheng Wu; Yong Yang
Original assignee: Individual
Current assignee: Individual
Priority date: 2020-02-08
Filing date: 2020-02-08
Publication date: 2020-06-11
Anticipated expiration: 2028-02-08

Abstract

Abstract: Single image super-resolution (SISR) has always received much attention, as one of the basic assignment of computer vision. The development and extensive exploration of deep learning have brought remarkable performance and index improvement to the SISR. However, most of existing CNN-based SISR methods concentrate on using wider or deeper architecture to obtain better results, while ignoring the portability of the network. The huge network structure expend too much computing resources, which constraint its application in mobile equipment. In order to address these problems, this patent advanced a content-based network, which can extract features more efficiently under the premise of ensuring quality and complexity. Experimental results testify that our network achieved better results compared with state-of-the-art SISR methods in terms of both objective indicators and visual quality.

Description

BACKGROUND AND PURPOSE

[0001] Image Super-Resolution, a fundamental and concerned problem, which aims to recover from the observed low-resolution (LR) image to the distinct high-resolution (HR) image. However, this is a sick problem since that there are myriad HR solutions map for a LR input. Therefore, quite some SR methods have been proposed, range from early interpolation-based method and model-based method, as well as the recently popular learning-based method, to emulate the problem.

[0002] In the early years, interpolation-based methods (bilinear and bicubic methods) were used to deal with some simplicity image amplify applications. As a result of its insufficiency flexibility and poor restoration quality, it’s have been displaced by modelbased methods later. The model-based method had achieved many satisfactory results, such as wavelet transform, sparse coding, and total variation model and so on. These methods use the prior information of the image to reestablish the HR image. Although the model-based methods are more flexible to restore a clearer HR image, they still have fatal defects: (1) they take too much time to get a HR image with good quality: (2) in the absence of prior information, the reconstruction quality of the image drops rapidly.

[0003] Deep convolution neural networks (CNNs) have shown great vitality and unprecedented success in many fields because of their powerful ability of feature extraction and expression. The powerful feature expression and end-to-end training of CNNs make they have been extremely referred single image super-resolution (SISR). Recently, a large number of CNNs-based SISR methods have emerged. By the statistical exploration of the image in the dataset, SISR methods based on CNNs shows their power and achieved state-of-the-art results. Most of the structures focus on increasing the width or depth of the network, resulting in huge consumption of computing resources.

[0004] To address these problems, we propose a deep content-guide residual network

2020100200 23 Apr 2020 (CGRN) for more powerful feature expression and feature correlation learning. In particular, we propose a content-guide multi-scale residual module to enhance the feature extraction and expression. Moreover, dilate convolution is presented instead of up and down sampling operation, which can obtain multi-scale feature information without losing high-frequency information easily lost in the up-down sampling operation, it is easy to train and has few parameters. Our method is faster and smaller in model while obtaining better visual quality and recovers more image details compared with stat-of-the-art SR methods.

[0005] In summary, the main contributions of this patent are listed as follows: (1) We propose a deep content-guide residual network (CGRN) for accurate image SR. Experimental results on pubic datasets demonstrate that our CGRN achieves better visual quality and objective indicators compared with other state-of-the-art SR methods.

(2) We propose a multi-scale attention module to extract image features more efficiently than a deeper or wider single-scale network. The multi-scale attention module makes our network pay attention to the structure information of the images as well as the characteristics. (3) We propose to use dilated convolution instead of up and down sampling to obtain more receptive fields. Compared with the high-frequency details of image lost in the up-down sampling operation, the dilated can better retain highfrequency. At the same time, it can avoid the huge increase of computation cost caused by using larger convolution kernel. And our network excellent balances visual quality, speed, and computing resources.

FRAMEWORK OF OUR CONTENT GURIDE RESIDUAL NET

[0001] Our CGRN mainly consists of four parts: shallow features extraction, contentguide residual group (CGRG), multi-scale attention block, and reconstruction part. Define Ilr and Ihr as inputs and outputs of CGRN. We proposed merely one convolution layer to extract the shallow features Fo from the LR images.

2020100200 23 Apr 2020

Equation 1

Where H_SF(·) representative convolution operation. Then the extracted shallow features Fo is used as the input of CGRG to extract the deep and structural features, which thus generate the deep features as.

^Fdf=^Hcgrg(^Fo) Equation 2

Where Fdf represents the deep feature and structure information extracts by CGRG, which is composed if four multi-scale attention blocks. Then the extracted deep feature Fdf is upscaled via the upscale module via.

F_t=H_t(F_flf.) Equation 3

Where /C(·) and F_f are an upscale module and upscaled feature respectively.

There are some processing methods in the reconstruction part, such as transposed convolution, ESPCN. We use pixelshuffle function and a convolution layer to form the upscale part.

Isr ~ ^Fr (F)) — Hcgrg (Jlr ) Equation 4

Where H_R(·), /C(·) and H_CGRG(·) are the reconstruction layer, upscale layer and the function of CGRN, respectively.

[0002] Li and L2 loss functions had been widely used in SISR. To show the effectiveness of our network, we use Li loss as current work. Given a training set {IJ, , which are composed of N LR input and their HR counterparts. The goal of training CGRN is to optimize the loss function of LI.

2020100200 23 Apr 2020 ^N ^£(^Θ) ⁼ ΐυΣ|Κσ™(4)-4?|| Equation 5

Where Θ denotes the parameter set of CGRN. The loss function is optimized by using stochastic gradient descent.

CONTENT GUIDE RESIDUAL GROUP

[0001] Our content-guide residual group (CGRG) contains four multi-scale attention blocks which each has its own convolution kernel with the same size (3 x 3) and different dilated rate to acquire more structure information and reduce parameters. We fused the intermediate information obtained by each multi-scale attention block to enhance the feature extraction ability of CGRG. Give input features F_in, this procedure in the CGRG can be expressed as

F¹ = H¹ ,(F ) Equation 6 msat msat v in /~

F² , = //² (F¹ ,) = //² ,(//¹ ,(F )) Equation 7 msat msat s msat/ msat s msats m//T.

F\=H³ (F² ,) = Η³ ,(. H² ,(//¹ ,(F ))) Equation 8 msat msat s msat/ msat' msat' msat' m / / /T.

FL = CO = Equation 9

Fcgrg = F_in + Re duce_m (Concat(F^_sal, F²sat, F³sal, F*_sat)) Equation 10

Where F‘ (·) denotes that features which obtain by the i -th multi-scale attention block. H‘ ,(·) denotes the function of i -th multi-scale attention. Concat denotes concatenation operation along the channel dimension, and Reduce indicates compression operation along the channel dimension.

MULTI-SCALE ATTENTION BLOCK

[0001] Our multi-scale attention block consists of 5 convolution layer (including Relu layer) and 1 SEblock. We fused the features from the first four layers and through the

2020100200 23 Apr 2020

SEblock. Then, we proposed a convolution layer to extract more high frequency features and use skip-chain to inject more details into the network of later layer. We use SEblock to allocate the parameters of the features learned from the precious convolution layer, so as to improve the efficiency of feature extraction. The formula of multi-scale channel block can be expressed as.

fi.j = ^hi,j ) = ^hij ( · -fii · · ·)) Equation 11

Cat = Z,i + ^hi,s (se(concat(f_il ))) Equation 12

Where . represents the features extracted by the j -th layer of the i -th multiscale attention block and .(·) represents its convolution operation (including ReLU layer). The concatl·) operation represents four convolutions layer. The 5β(·) operation stands for the function of the channel attention module. The analysis shows that the channel attention module enhances the network expression ability in the process of assigning weight parameters to the channels.

Claims

1. The procedures of the proposed single-image super-resolution method as follows:

[0001]The proposed method is introduced in detail. The structure of the proposed method is shown in Figure 1. From Figure 1, we can know that the proposed method consists of four parts.

[0002] The part 1: Use a simple 3x3 convolutional layer to extract shallow features of low-resolution images.

[0003] The part 2: We design the Content-guide Residual Groups (CGRG) to extract and fuse multi-level features of the image.

[0004] The part 3: We propose the Multiscale-attention Block containing convolutional layers with multiple dilations to obtain a larger receptive field and thus more structural features of the image.

[0005] The part 4: The image reconstruction part includes a convolutional layer to integrate the features obtained previously and an upsampling layer to reconstruct the image.

2. The structures of content guide residual group are as follows:

[0001] Our content-guide residual group (CGRG) contains four multi-scale attention blocks which each has its own convolution kernel with the same size (3 x 3) and different dilated rate to acquire more structure information and reduce parameters. We fused the intermediate information obtained by each multi-scale attention block to enhance the feature extraction ability of CGRG. Give input features E_n, this procedure

2020100200 23 Apr 2020 in the CGRG can be expressed as ^FLt=^HLt(^FJ Equation 1 ^FL = (^FL· ) - (^F _in)) Equation 2 ^FLt = ^HLat (^FL) - ^HLat (^HLt (^Fin))) Equation 3 ^FL - HLA^fL ) - C(C(C(C(O) Equation 4 ^Fcgrg = ^Fin + ^{Re dlice}w (^Concat(^FLt ’ ^FL, ’ ^FL, ’ ^FL,)) Equation 5

Where F' (·) denotes that features which obtain by the i -th multi-scale attention block. H' (·) denotes the function of i -th multi-scale attention. Concat denotes concatenation operation along the channel dimension, and Reduce indicates compression operation along the channel dimension.

3. The structures of multi-scale attention block are as follows:

[0001] Our multi-scale attention block consists of 5 convolution layer (including Relu layer) and 1 SEblock. We fused the features from the first four layers and through the SEblock. Then, we proposed a convolution layer to extract more high frequency features and use skip-chain to inject more details into the network of later layer. We use SEblock to allocate the parameters of the features learned from the precious convolution layer, so as to improve the efficiency of feature extraction. The formula of multi-scale channel block can be expressed as.

fij = ^hi,j ) = ^hij (· ·-f,i ·)) Equation 6

Cat = fi.x + C (se(concat(f ₄, f_{i 7}, f_i3, f_A ))) Equation 7

Where f_t. represents the features extracted by the j -th layer of the i -th multiscale attention block and .(·) represents its convolution operation (including

23 Apr 2020

ReLU layer). The co«cat(·) operation represents four convolutions layer. The se(·) operation stands for the function of the channel attention module. The analysis shows that the channel attention module enhances the network expression ability in the process of assigning weight parameters to the channels.