CN114926343A - Image super-resolution method based on pyramid fusion attention network - Google Patents
Image super-resolution method based on pyramid fusion attention network Download PDFInfo
- Publication number
- CN114926343A CN114926343A CN202210639755.0A CN202210639755A CN114926343A CN 114926343 A CN114926343 A CN 114926343A CN 202210639755 A CN202210639755 A CN 202210639755A CN 114926343 A CN114926343 A CN 114926343A
- Authority
- CN
- China
- Prior art keywords
- module
- resolution
- network
- pyramid
- output
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 68
- 238000000034 method Methods 0.000 title claims abstract description 28
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 claims abstract description 17
- 230000000750 progressive effect Effects 0.000 claims abstract description 17
- 238000005070 sampling Methods 0.000 claims abstract description 7
- 238000000605 extraction Methods 0.000 claims description 21
- 238000012549 training Methods 0.000 claims description 14
- 102000005708 Desmoglein 1 Human genes 0.000 claims description 10
- 108010045579 Desmoglein 1 Proteins 0.000 claims description 10
- 238000010586 diagram Methods 0.000 claims description 7
- 230000008569 process Effects 0.000 claims description 4
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 3
- 230000007246 mechanism Effects 0.000 abstract description 17
- 238000013461 design Methods 0.000 description 5
- 230000008901 benefit Effects 0.000 description 4
- 238000013527 convolutional neural network Methods 0.000 description 4
- 238000013135 deep learning Methods 0.000 description 4
- 230000002708 enhancing effect Effects 0.000 description 4
- 238000013459 approach Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 230000015556 catabolic process Effects 0.000 description 2
- 238000006731 degradation reaction Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- 238000011176 pooling Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 101100365548 Caenorhabditis elegans set-14 gene Proteins 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000008034 disappearance Effects 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000004880 explosion Methods 0.000 description 1
- 230000009191 jumping Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000003278 mimic effect Effects 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4053—Super resolution, i.e. output image resolution higher than sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4046—Scaling the whole image or part thereof using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Biophysics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
A pyramid fusion attention network-based image super-resolution method is characterized in that a pyramid fusion attention mechanism is introduced to recover a high-resolution image from a given low-resolution image, a pyramid fusion structure is adopted firstly, each layer of pyramids are stacked by using residual blocks, and a down-sampling operation and a multi-scale fusion strategy are used to ensure a complete receptive field and master more context detail information. In addition, the invention also provides a progressive backward fusion strategy to fully utilize the hierarchical characteristics generated by the intermediate pyramid fusion attention module. The pyramid fusion attention module disclosed by the invention can better enhance the discrimination capability of a network by modeling the relationship between pixels, so that high-frequency information can be better recovered.
Description
Technical Field
The invention belongs to the technical field of deep learning, computer vision and computer image processing, relates to single image super-resolution, and discloses an image super-resolution method based on a pyramid fusion attention network.
Background
Image Super-Resolution (SISR) has recently received much attention as one of the long-standing basic tasks in the field of computer vision. The task is to restore a Low Resolution image (LR) to a High Resolution image (HR) by a certain technical means, so as to obtain clearer image details. In a real scene, images with higher resolution cannot be directly obtained due to the influence of multiple aspects such as imaging equipment capacity, network bandwidth and storage cost. Therefore, reconstructing a high resolution image from a low resolution image is particularly important in these situations. The image super-resolution has very wide application prospect in multiple fields such as military affairs, medicine, security protection, satellite imaging, HDTV and the like, and has very important research value. However, the SISR task is highly ill-posed because of the existence of one-to-many mapping of the solution space from LR images to HR images.
In recent years, deep learning techniques have achieved an effect superior to conventional techniques in various fields. Since AlexNet has made a great deal of splendid in the ImageNet competition in 2010, methods based on deep learning have found wide application in various fields of computer vision. The image super-resolution task is a long-existing underlying vision task in computer vision and is also deeply influenced by the underlying vision task. Dong et al first introduced a Convolutional Neural Network (CNN) into the field in 2015, and proposed a SRCNN. The model first upsamples the LR image to a corresponding scale and then maps the image to the HR image domain using a network built up by 3-layer convolution. Compared with the traditional algorithm, the SRCNN achieves better effect. Following this, CNN-based methods have been a long-standing development in this area.
To further improve the performance of the network, most models use an attention mechanism based on CNN. Attention mechanisms attempt to mimic the ability of the human visual system to capture information, thereby making the network more focused on salient region features while reducing the focus on extraneous features. The traditional attention mechanism is mainly based on two considerations: 1. the correlation is calculated along the channel domain, the weight of each channel is calculated, and the features in the channels are treated equally. 2. And calculating the correlation along the spatial domain, calculating the correlation for each pixel point in the channel, wherein the correlations of the same coordinate position in different channels are the same. On this basis, a number of attention mechanisms have been proposed.
Disclosure of Invention
The invention aims to solve the problems that: most of the existing attention mechanisms only focus on capturing the inherent feature correlation along a channel domain or a space domain, and treat the features in the corresponding dimension equally, so that the capability of the attention mechanism is hindered; the method has the advantages that a plurality of modules exist in the whole network architecture of image hyper-segmentation, the output of the intermediate modules is intermediate features, most of the existing methods cannot fully utilize the intermediate features, and the features have important significance for reconstructing spatial context details, so that the performance is relatively weak.
The technical scheme of the invention is as follows: a super-resolution method of a single image based on a pyramid fusion attention network is characterized in that an image super-resolution network PFAN is constructed to recover a high-resolution image from a given low-resolution image, the image super-resolution network is composed of a shallow feature extraction module, a stacked feature extraction basic group BG, an up-sampling module and a reconstruction module which are sequentially arranged, and the super-resolution method comprises the following procedures:
1) the shallow feature extraction module uses a layer of convolution to pair the low-resolution image I LR Extracting shallow feature F 0 ：
F 0 ＝H 0 (I LR )
Wherein H 0 (. -) represents a convolution function;
2) deep level feature extraction is carried out by using the stacked feature extraction basic group BG:
F D ＝H BG-D (F D-1 )＝H BG-D (H BG-(D-1) (…H BG-1 (F 0 )…))
wherein H BG-D (. denotes the Dth BG module, F D-1 Denotes the (D-1) th) The output of each BG module;
3) using sub-pixel convolution as an upsampling module to improve resolution while fusing deep features F D And shallow feature F 0 ：
F ↑ ＝H ↑ (F D +F 0 )
Wherein H ↑ (. represents an upsampling module, F ↑ Representing the upsampled features;
4) result F of upsampling using a layer of convolution ↑ And (3) carrying out feature reconstruction:
I SR ＝H R (F ↑ )
output result I of final pyramid fusion attention network SR Namely the reconstructed high-resolution image;
the basic group BG of feature extraction is composed of n pyramid fusion attention modules PFAB and a progressive backward fusion module PBFM, the pyramid fusion attention module PFAB is composed of a basic residual block Plain RB and a pyramid fusion attention network PFA, the PFA adopts a pyramid structure, the output of the basic residual block Plain RB is firstly convoluted and pooled after the PFA is input, then the relation between pixels is modeled by using stacked standard residual blocks RB, the pyramid middle layer receives the output from corresponding upper and lower layers RB as input, the output of each layer of the pyramid is up-sampled to the same size and then is cascaded, and is sent to a convolution layer and a sigmoid layer, so that an attention mask is obtained, and the output of the current PFAB is obtained after the attention mask and a characteristic diagram of the current PFAB module are subjected to pixel level product operation; n PFABs are connected in series in sequence, a progressive backward fusion module PBFM is used for cascade operation on a channel domain for the output of two adjacent PFABs, a channel attention CCA module based on contrast is sent to the feature after each cascade operation to further enhance key information, a 1 multiplied by 1 convolutional layer is used for fusing the feature of the channel domain of the feature enhanced by the CCA module, wherein the PFAB n Output result of (B) n And PFAB n-1 Output result of (B) n-1 Directly cascaded, then convolved with 1 × 1 through a CCA module to obtain the corresponding PFAB n-1 Fused output of B' n-1 For PFAB i I-1, …, n-1, adjacent PFABs i Is cascaded to B i And B' i+1 Cascade, represented as:
B′ i ＝H F (concat(B i +B′ i+1 ))
B′ 1 for the final output of the progressive backward fusion module PBFM, the output characteristic of the jth BG module is recorded as F j Then F is j From B' 1 And F j-1 And performing pixel-level addition to obtain the product.
The present invention recovers a high resolution image from a given low resolution image by introducing a novel pyramid fusion attention mechanism. According to the invention, the stacked feature extraction basic group BG can effectively utilize intermediate features, and the pyramid fusion attention module PFAB can better enhance the discrimination capability of a network by modeling the relationship between pixels, so that high-frequency information can be better recovered. In addition, the invention also provides a progressive backward fusion strategy, which guides the shallow feature to perform feature fusion by using the deep feature and better utilizes the network middle layer feature.
The invention provides an image hyper-resolution network PFAN based on a pyramid fusion attention network, which is used for constructing stronger feature representation and enhancing the discrimination capability of the network. The invention provides a pyramid fusion attention network PFA, and a pyramid fusion attention module PFAB is constructed according to the pyramid fusion attention network PFA. PFA adopts a plurality of residual modules RB to extract intermediate features, and simultaneously uses a pyramid structure with down-sampling operation and multi-scale fusion strategy to recalibrate the obtained feature information. Such a design has three main advantages: first, the pyramid bottom structure maintains the size of the feature map to learn the correlation between pixels. Such a design can make the network more flexible in handling different types of information than channel attention or spatial attention; secondly, the higher-level pyramid structure has a larger receptive field, and can acquire more global context information with relatively less expense; third, feature fusion between adjacent pyramid layers would provide information exchange across multiple resolutions in order to better apply multi-scale information. Thus, the pyramid fusion attention module described above can predict a more accurate attention mask. In addition, in order to fully apply the hierarchical features generated by each PFAB, the invention further provides a progressive backward fusion module PBFM to fuse the features of each intermediate layer to generate features with more discriminative ability, and finally, a plurality of PFABs and one PBFM are combined to form a feature extraction basic group BG, and a plurality of BGs and a global jump connection are used to construct a final PFAN. Generally, the larger the parameter amount of the model is, the stronger the expression ability of the model is, and thus the better the model achieves. However, it is very difficult to fully utilize all parameters, and therefore it is important to design a network structure to achieve better performance under the premise of a certain parameter amount. Compared with the prior method, the PFAN provided by the invention achieves superior performance by using less parameter quantity (11.9M).
Compared with the existing method, the single-image super-resolution method based on the pyramid fusion attention network has the advantages that:
1. an image super-resolution network PFAN based on a pyramid fusion attention network is provided, and is used for a large-model single-image super-resolution task. Full experiments are carried out on the public data sets based on different degradation models, and the experimental results show the effectiveness of PFAN and are superior to the existing latest and most advanced method in the aspects of quantitative indexes and visual results.
2. A pixel-level attention mechanism PFA is proposed that adaptively acquires inter-pixel correlations using a pyramid fusion structure, while allowing information exchange across multiple resolutions and providing a larger and more complete field of view. This design enables the network to generate more accurate attention masks, thereby obtaining more pixel-level and global information.
3. A progressive backward fusion module PBFM is provided to fully utilize a plurality of middle layer features so as to acquire more context information for better image recovery.
Drawings
FIG. 1 is a flow chart of the method of the present invention.
Fig. 2 is a schematic diagram of a basic architecture of a single-image super-resolution network based on a pyramid fusion attention network according to the present invention.
Fig. 3 is a schematic structural diagram of the pyramid fusion attention module in the present invention.
Fig. 4 is a schematic structural diagram of a basic residual block in the present invention.
Fig. 5 is a schematic structural diagram of a standard residual block in the present invention.
Fig. 6 is a schematic structural diagram of a feature extraction basic group in the present invention.
Detailed Description
The invention provides a single-image super-resolution method based on a pyramid fusion attention network. Most recent approaches focus on high frequency information using an attention mechanism. However, these approaches only consider interdependencies between channels or spaces, resulting in an equal processing of channel or space characteristics, thereby hindering the ability to focus on mechanisms. The invention provides an image super-resolution network PFAN based on a pyramid fusion attention network, which is used for constructing stronger feature representation and enhancing the discrimination capability of the network. Specifically, the high resolution images HR are recovered from a given low resolution image LR by introducing a novel pyramid fusion attention mechanism PFA. The PFA module is used for better enhancing the discrimination capability of the network by modeling the relation between pixels, so that high-frequency information can be better recovered. PFA adopts a pyramid fusion structure, each layer of pyramid uses a standard residual block introduced with a BN layer for stacking, and a down-sampling operation and a multi-scale fusion strategy are used for ensuring a complete receptive field and mastering more context detail information. In addition to utilizing the feature correlation between pixels, the present invention also provides a progressive backward fusion strategy PBFM to fully utilize the hierarchical features generated by the intermediate pyramid fusion attention module PFAB.
The method comprises the steps of calculating the error between an output image and a high-resolution real image by using an MAE function during PFAN training of the image hyper-division network, calculating the gradient of each parameter of the image hyper-division network according to the error, updating the parameter, and training the network by using an Adam optimizer. The invention is described in further detail below with reference to the following figures and detailed description.
Step 1: and (4) data is expanded. In order to better apply the capability of the common data set, before the model training, the invention firstly carries out data enhancement on the training data set through a strategy of randomly turning horizontally and randomly rotating by 90 degrees, 180 degrees or 270 degrees, and cuts the training data set into 48 multiplied by 48 image blocks, thereby enhancing the generalization capability of the network training.
Step 2: and constructing a single image super-resolution network model and carrying out supervision training. The basic framework of network model construction is shown in fig. 2, and includes four key parts: the device comprises a shallow layer feature extraction module, a stacked feature extraction basic group BGs, an up-sampling module and a reconstruction module. The overall structure can be formulated as the following flow:
(1) the shallow feature extraction module uses a layer of convolution to pair the low-resolution image I LR Extracting shallow feature F 0 ：
F 0 ＝H 0 (I LR )#(1)
Wherein H 0 (. cndot.) represents a convolution function.
(2) And performing deep level feature extraction by using the stacked feature extraction basic group.
F D ＝H BG,D (F D-1 )＝H BG,D (H BG,(D-1) (…H BG,1 (F 0 )…))#(2)
H BG,D (. smallcircle.) denotes the operation of the Dth BG module, F D-1 Represents the output of the (D-1) th BG module, F D The output of the last BG module of the stack.
(3) And using sub-pixel convolution as an up-sampling module to improve the resolution, and fusing the deep layer features and the shallow layer features at the same time through global jumping.
F ↑ ＝H ↑ (F D +F 0 )#(3)
Wherein H ↑ (. represents an upsampling module, F) ↑ Representing the upsampled features.
(4) A final feature reconstruction is performed using a layer of convolution.
I SR ＝H R (F ↑ )#(4)
Output result I of final pyramid fusion attention network SR I.e. the reconstructed high resolution image.
The basic group BG for feature extraction is composed of a plurality of pyramid fusion attention modules PFAB and a progressive backward fusion module PBFM, taking the 2 nd BG module as an example:
F 2,1 ＝H PFAB,1 (F 1 )
F 2,2 ＝H PFAB,2 (F 2,1 ) #（5）
F 2,n ＝H PFAB,n (F 2,(n-1) )
F 2 ＝H PBFM (F 2,1 ,F 2,2 ,…,F 2,n )+F 1 #(6)
wherein H PFAB,i (. o) denotes the ith PFAB module of the 2 nd BG module, n PFAB modules in total, F 2,i Denotes the output of the ith PFAB module of the 2 nd BG module, H PBFM (. represents a PBFM module, F 2 For the output of the 2 nd BG module, the result of the fusion by the PBFM and the output F of the first BG module 1 And carrying out pixel-level addition to obtain the product.
The PFAB is composed of a basic residual block play RB and a pyramid Fusion attention network pfa (pyramid Fusion attention), and the specific structure is shown in fig. 3. The architecture of the basic residual block plain RB is shown in FIG. 4. As discussed in the EDSR (see article Enhanced Deep Residual network for Single Image Super-Resolution b.lim, s.son, h.kim, s.nah and k.m.lee, "Enhanced Deep Residual network for Single Image Super-Resolution,"2017IEEE Conference on Computer Vision and Pattern Recognition Works (CVPRW),2017, pp.1132-1140, doi: 10.1109/prw.2017.151.), the bulk normalization layer BN normalizes the features, which means that the Residual block with the BN layer is not the domain flexibility of the network by normalizing the features, which is required for the beyond Resolution task of the Image. The BN layer is therefore deleted here in the base residual block structure, which is crucial for the underlying tasks like SR.
Previous work on attention mechanisms focused primarily on the inherent feature correlation at the channel-level or spatial-level, resulting in equal treatment of features within these dimensions. Accordingly, the present invention proposes PFA to solve the above-mentioned drawbacks. To make the overall structure lighter, PFA first uses one 1 × 1 convolution layer to reduce the number of channels. After the convolutional layer, PFA uses stacked residual blocks to model the relationship between pixels. Thanks to the definition of convolution operations by the existing deep learning framework, each channel in the feature map considers all input channels after a 2-dimensional convolutional layer process with in _ channel and out _ channel mechanisms. Therefore, PFA can acquire the correlation between the characteristic channels and obtain the complete receptive field by simply using the residual block composed of the 2D convolution layer and the jump connection. Unlike the basic residual block plain RB, the standard residual block RB structure adopted in PFA is shown in fig. 5, into which the batch normalization layer BN is newly introduced. Since EDSR explored the role of BN layers in underlying visual tasks, almost all subsequent approaches removed BN layers in the network. However, the attention module is intended to direct the network to focus more on the salient regions, rather than directly on the computation of the image super-score results, which means that this part of the feature may be high-level. Therefore, reintroducing the BN layer into the attention module does not cause performance degradation, but is beneficial to accelerating convergence of network training, preventing gradient explosion or disappearance, and even further improving performance.
In order to further expand the receptive field and improve the attention mechanism, the PFA of the invention adopts a pyramid structure. The structure of each layer in the pyramid structure is similar, and it is noted that PFA has a maximum pooling layer behind the 1 × 1 convolution layer in each layer of the pyramid structure, the pooling-n in fig. 3 represents the maximum pooling layer that reduces the size of the input feature map by n times, and different layers in the pyramid structure have different n values, which makes the feature maps of different layers have different sizes. Thanks to the above design, the pyramid bottom structure aims at maintaining the size of the feature map to learn the correlation between pixels, which makes the network more flexible when processing different types of information; and a higher-layer structure can obtain a larger receptive field and obtain more global context information at the same time. In addition, the pyramid middle layer will receive as input the output from the corresponding upper and lower layers RB and fuse using one upsampling layer and one 1 x 1 convolutional layer, providing information exchange across multiple resolutions. Finally, the output of each pyramid layer is up-sampled to the same size and then cascaded (concatee layer), and is sent to a convolutional layer and a sigmoid layer, so as to obtain the final attention mask. And after the attention mask and the feature image of the current PFAB module are subjected to pixel level multiplication operation, the output of the current PFAB is obtained.
In addition, for the connection strategy of a plurality of PFAB modules in the feature extraction basic group, the invention provides a progressive Backward Fusion module pbfm (progressive Backward Fusion module), the structure of which is shown in fig. 6. Rather than simply concatenating these PFABs, the PBFM aims to take full advantage of the hierarchical features produced by each PFAB and produce a more discriminative output. Through experimental verification, the output features of the deeper modules help guide the features of the shallow modules, so that the features are better used for image restoration. For the output of two adjacent PFABs, using cascade operation on the channel domain and feeding the cascaded features into a Contrast-based channel attention CCA module to further enhance the critical information, the CCA (Contrast-aware channel attention) module can be seen in the paper "light weight image super-resolution with information multi-resolution network" (arXiv:1909.11856[ eess. IV)]) And then, fusing the characteristics of the channel domain of the characteristics enhanced by the CCA module by using a convolution layer of 1 multiplied by 1. For n PFAB, PFAB n Output result of (B) n And PFAB n-1 Output result B of n-1 Directly cascaded and then convolved with 1 × 1 through a CCA module to obtain the corresponding PFAB n-1 Fused output B 'of' n-1 For PFAB i I-1, …, n-1, adjacent PFABs i Is cascaded as B i And B' i+1 A cascade, represented as:
B′ i ＝H F (concat(B i +B′ i+1 ))
B′ 1 is the output of the final progressive backward fusion module PBFM.
After the above mentioned progressive fusion step, an activation function layer (ReLU) is also used to preserve the non-linearity of the network. Similarly, after all the PFAB fusion operations, skip-connection is used to fully utilize the shallow feature and finally generate the output feature of the current BG module, and the output feature of the jth BG module is recorded as F j Then F is j From B' 1 And F j-1 And carrying out pixel-level addition to obtain the product.
For the supervised training process of this step, the invention uses Adam optimizer with optimizer parameters set to β 1-0.9, β 2-0.99 and ∈ -10 -8 . Initial learning rate was set to 10 -4 And then halved every 200 epochs.
And 3, step 3: and (6) testing. And (3) fixing the parameters of the trained network model, and sending the low-resolution test image into the network model to obtain a corresponding high-resolution image.
The invention provides a brand-new pyramid fusion attention network PFAN. The conventional attention mechanism only calculates the correlation along the channel dimension or the space dimension, and treats the characteristics in the corresponding dimension equally, so that the invention provides a pyramid fusion attention mechanism PFA, which models the correlation under the pixel granularity, ensures the receptive field by using a pyramid structure, acquires global and pixel level information and improves the capability of a network for recovering high-frequency information. In addition, the invention also provides a Progressive Backward Fusion Module (PBFM) which guides the shallow feature to perform feature fusion by using the deep feature and better utilizes the network middle layer feature. The invention achieves the leading performance on the public data sets Set5 and Set14, namely PSNR 38.32 and SSIM 0.9617 on Set5, and PSNR 34.21dB and SSIM 0.9224 on Set 14.
Claims (3)
1. A single-image super-resolution method based on a pyramid fusion attention network is characterized in that an image super-resolution network PFAN is constructed to recover a high-resolution image from a given low-resolution image, the image super-resolution network consists of a shallow feature extraction module, a stacked feature extraction basic group BG, an up-sampling module and a reconstruction module which are sequentially arranged, and the method comprises the following processes:
1) the shallow feature extraction module uses a layer of convolution to pair the low-resolution image I LR Extracting shallow feature F 0 ：
F 0 ＝H 0 (I LR )
Wherein H 0 (. cndot.) represents a convolution function;
2) deep level feature extraction is carried out by using the stacked feature extraction basic group BG:
F D ＝H BG,D (F D-1 )＝H BG,D (H BG,(D-1) (…H BG,1 (F 0 )…))
wherein H BG,D (. represents a Dth BG module, F D-1 Represents the output of the (D-1) th BG module;
3) using sub-pixel convolution as an upsampling module to improve resolution while fusing deep features F D And shallow feature F 0 ：
F ↑ ＝H ↑ (F D +F 0 )
Wherein H ↑ (. represents an upsampling module, F) ↑ Representing the upsampled features;
4) result F of upsampling using a layer of convolution ↑ And (3) carrying out characteristic reconstruction:
I SR ＝H R (F ↑ )
output result I of final pyramid fusion attention network SR Namely, the reconstructed high-resolution image is obtained;
the basic group BG consists of n pyramid fusion attention modules PFAB and a progressive backward fusion module PBFM, and the pyramid fusion attention module PFAB consists of a basic residual errorThe method comprises the steps that a block Plain RB and a pyramid fusion attention network PFA are formed, the PFA adopts a pyramid structure, the output of a basic residual block Plain RB is convoluted and pooled after the PFA is input, then the relation between pixels is modeled by using stacked standard residual blocks RB, the pyramid middle layer receives the output from corresponding upper and lower layers RB as input, the output of each layer of the pyramid is cascaded after being up-sampled to the same size, and is sent to a convolution layer and a sigmoid layer, so that an attention mask is obtained, and the attention mask and a feature diagram of a current PFAB module are subjected to pixel level multiplication operation to obtain the output of the current PFAB; n PFABs are connected in series in sequence, a progressive backward fusion module PBFM is used for cascade operation on a channel domain for the output of two adjacent PFABs, a channel attention CCA module based on contrast is sent to the feature after each cascade operation to further enhance key information, a 1 multiplied by 1 convolutional layer is used for fusing the feature of the channel domain of the feature enhanced by the CCA module, wherein the PFAB n Output result B of n And PFAB n-1 Output result B of n-1 Directly cascaded and then convolved with 1 × 1 through a CCA module to obtain the corresponding PFAB n-1 Fused output of B' n-1 For PFAB i I-1, …, n-1, adjacent PFABs i Is cascaded to B i And B' i+1 Cascade, represented as:
B′ i ＝H F (concat(B i +B′ i+1 ))
B′ 1 for the final output of the progressive backward fusion module PBFM, the output characteristic of the jth BG module is recorded as F j Then F is j From B' 1 And F j-1 And carrying out pixel-level addition to obtain the product.
2. The single-image super-resolution method based on the pyramid fusion attention network as claimed in claim 1, wherein training data is used to train the image super-resolution network, during training, an MAE function is used to calculate errors between output images and high-resolution real images, gradients of parameters of the image super-resolution network are calculated according to the errors, the parameters are updated, and the network is trained by using an Adam optimizer.
3. The single-image super-resolution method based on the pyramid fusion attention network according to claim 2, wherein training data is subjected to data enhancement first, the training data is subjected to data enhancement through a strategy of random horizontal flipping and random rotation by 90 °, 180 ° or 270 °, and each min-batch uses 16 low-resolution RGB image blocks with the size of 48 × 48 as input in the training process.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210639755.0A CN114926343A (en) | 2022-06-08 | 2022-06-08 | Image super-resolution method based on pyramid fusion attention network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210639755.0A CN114926343A (en) | 2022-06-08 | 2022-06-08 | Image super-resolution method based on pyramid fusion attention network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114926343A true CN114926343A (en) | 2022-08-19 |
Family
ID=82813056
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210639755.0A Pending CN114926343A (en) | 2022-06-08 | 2022-06-08 | Image super-resolution method based on pyramid fusion attention network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114926343A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115358931A (en) * | 2022-10-20 | 2022-11-18 | 运易通科技有限公司 | Image reconstruction method and device for warehouse logistics system |
| CN115797789A (en) * | 2023-02-20 | 2023-03-14 | 成都东方天呈智能科技有限公司 | Cascade detector-based rice pest monitoring system and method and storage medium |
| CN116362972A (en) * | 2023-05-22 | 2023-06-30 | 飞狐信息技术（天津）有限公司 | Image processing method, device, electronic equipment and storage medium |
| CN117173024A (en) * | 2023-09-20 | 2023-12-05 | 中国矿业大学 | Mine image super-resolution reconstruction system and method based on overall attention |
-
2022
- 2022-06-08 CN CN202210639755.0A patent/CN114926343A/en active Pending
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115358931A (en) * | 2022-10-20 | 2022-11-18 | 运易通科技有限公司 | Image reconstruction method and device for warehouse logistics system |
| CN115358931B (en) * | 2022-10-20 | 2023-01-03 | 运易通科技有限公司 | Image reconstruction method and device for warehouse logistics system |
| CN115797789A (en) * | 2023-02-20 | 2023-03-14 | 成都东方天呈智能科技有限公司 | Cascade detector-based rice pest monitoring system and method and storage medium |
| CN116362972A (en) * | 2023-05-22 | 2023-06-30 | 飞狐信息技术（天津）有限公司 | Image processing method, device, electronic equipment and storage medium |
| CN116362972B (en) * | 2023-05-22 | 2023-08-08 | 飞狐信息技术（天津）有限公司 | Image processing method, device, electronic equipment and storage medium |
| CN117173024A (en) * | 2023-09-20 | 2023-12-05 | 中国矿业大学 | Mine image super-resolution reconstruction system and method based on overall attention |
| CN117173024B (en) * | 2023-09-20 | 2024-04-16 | 中国矿业大学 | Mine image super-resolution reconstruction system and method based on overall attention |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110570353B (en) | Super-resolution reconstruction method for generating single image of countermeasure network by dense connection | |
| CN111062872B (en) | Image super-resolution reconstruction method and system based on edge detection | |
| CN110119780B (en) | Hyper-spectral image super-resolution reconstruction method based on generation countermeasure network | |
| CN114926343A (en) | Image super-resolution method based on pyramid fusion attention network | |
| CN113240613B (en) | Image restoration method based on edge information reconstruction | |
| CN113592718A (en) | Mine image super-resolution reconstruction method and system based on multi-scale residual error network | |
| CN107154023A (en) | Face super-resolution reconstruction method based on generation confrontation network and sub-pix convolution | |
| CN115222601A (en) | Image super-resolution reconstruction model and method based on residual mixed attention network | |
| CN111861961A (en) | Multi-scale residual error fusion model for single image super-resolution and restoration method thereof | |
| CN113012172A (en) | AS-UNet-based medical image segmentation method and system | |
| CN110363068B (en) | High-resolution pedestrian image generation method based on multiscale circulation generation type countermeasure network | |
| CN109214989A (en) | Single image super resolution ratio reconstruction method based on Orientation Features prediction priori | |
| CN111768340B (en) | Super-resolution image reconstruction method and system based on dense multipath network | |
| CN109949217B (en) | Video super-resolution reconstruction method based on residual learning and implicit motion compensation | |
| KR101977067B1 (en) | Method for reconstructing diagnosis map by deep neural network-based feature extraction and apparatus using the same | |
| CN115484410B (en) | Event camera video reconstruction method based on deep learning | |
| CN109961407A (en) | Facial image restorative procedure based on face similitude | |
| CN112699844A (en) | Image super-resolution method based on multi-scale residual error level dense connection network | |
| CN113469884A (en) | Video super-resolution method, system, equipment and storage medium based on data simulation | |
| CN112949636A (en) | License plate super-resolution identification method and system and computer readable medium | |
| CN112950475A (en) | Light field super-resolution reconstruction method based on residual learning and spatial transformation network | |
| CN115393191A (en) | Method, device and equipment for reconstructing super-resolution of lightweight remote sensing image | |
| CN115293986A (en) | Multi-temporal remote sensing image cloud region reconstruction method | |
| Cheng et al. | Hybrid transformer and cnn attention network for stereo image super-resolution | |
| Bao et al. | S 2 net: Shadow mask-based semantic-aware network for single-image shadow removal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |