CN112991173B - Single-frame image super-resolution reconstruction method based on dual-channel feature migration network - Google Patents
Single-frame image super-resolution reconstruction method based on dual-channel feature migration network Download PDFInfo
- Publication number
- CN112991173B CN112991173B CN202110268450.9A CN202110268450A CN112991173B CN 112991173 B CN112991173 B CN 112991173B CN 202110268450 A CN202110268450 A CN 202110268450A CN 112991173 B CN112991173 B CN 112991173B
- Authority
- CN
- China
- Prior art keywords
- feature
- correction
- fusion
- features
- resolution
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 76
- 238000013508 migration Methods 0.000 title claims abstract description 21
- 230000005012 migration Effects 0.000 title claims abstract description 21
- 238000012937 correction Methods 0.000 claims description 65
- 230000004927 fusion Effects 0.000 claims description 50
- 239000011159 matrix material Substances 0.000 claims description 39
- 239000010410 layer Substances 0.000 claims description 27
- 230000008569 process Effects 0.000 claims description 27
- 238000010586 diagram Methods 0.000 claims description 21
- 230000004913 activation Effects 0.000 claims description 16
- 230000006870 function Effects 0.000 claims description 13
- 238000013507 mapping Methods 0.000 claims description 10
- 230000009466 transformation Effects 0.000 claims description 7
- 230000001174 ascending effect Effects 0.000 claims description 6
- 238000005457 optimization Methods 0.000 claims description 6
- 238000011176 pooling Methods 0.000 claims description 6
- 238000012549 training Methods 0.000 claims description 6
- 230000001131 transforming effect Effects 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 5
- 238000012545 processing Methods 0.000 claims description 5
- NAWXUBYGYWOOIX-SFHVURJKSA-N (2s)-2-[[4-[2-(2,4-diaminoquinazolin-6-yl)ethyl]benzoyl]amino]-4-methylidenepentanedioic acid Chemical compound C1=CC2=NC(N)=NC(N)=C2C=C1CCC1=CC=C(C(=O)N[C@@H](CC(=C)C(O)=O)C(O)=O)C=C1 NAWXUBYGYWOOIX-SFHVURJKSA-N 0.000 claims description 3
- 238000013459 approach Methods 0.000 claims description 3
- 239000002356 single layer Substances 0.000 claims description 3
- 238000005070 sampling Methods 0.000 abstract description 5
- 238000009826 distribution Methods 0.000 abstract description 3
- 230000005540 biological transmission Effects 0.000 abstract description 2
- 230000008034 disappearance Effects 0.000 abstract description 2
- 230000007246 mechanism Effects 0.000 abstract description 2
- 238000004422 calculation algorithm Methods 0.000 description 3
- 101100274274 Secale cereale rsca gene Proteins 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000003909 pattern recognition Methods 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 210000002615 epidermis Anatomy 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000004438 eyesight Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000001575 pathological effect Effects 0.000 description 1
- 210000003462 vein Anatomy 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 230000016776 visual perception Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/15—Correlation function computation including computation of convolution operations
- G06F17/153—Multidimensional correlation or convolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/047—Probabilistic or stochastic networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Evolutionary Computation (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Mathematical Analysis (AREA)
- Health & Medical Sciences (AREA)
- Mathematical Optimization (AREA)
- Molecular Biology (AREA)
- Pure & Applied Mathematics (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Computational Mathematics (AREA)
- Algebra (AREA)
- Databases & Information Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a single-frame image super-resolution reconstruction method based on a dual-channel feature migration network, which aims at the problem that detail information is lost due to the increase of depth of a depth super-resolution image reconstruction network, local receptive fields are not fully utilized for global-local texture similarity, and a network model is designed, and the whole network is based on a residual mechanism, and a post up-sampling module is used for carrying out up-sampling on the image in a spatial scale to output a super-resolution reconstruction result y. The invention can maintain the effective distribution of the spatial repeated texture features, multiplex the out-of-service features by using the residual structure, effectively prevent the disappearance phenomenon of the detail features in the forward transmission process of the network and remarkably improve the super-resolution reconstruction quality of single-frame images.
Description
Technical Field
The invention belongs to the field of super-resolution reconstruction of single-frame images, and particularly relates to a single-frame image super-resolution reconstruction method based on a dual-channel feature migration network.
Background
The purpose of single frame image super-resolution reconstruction (SISR) is to obtain a high resolution image reconstruction output from a low resolution image input and to recover the details and texture information of the image to the maximum extent with increasing its spatial resolution. From the viewpoint of signal entropy, the method is an entropy increasing process, under the condition of no reference and no priori knowledge, the method is a pathological problem of reconstructing an original real signal from a lossy degradation signal, from an early interpolation-based method to a current learning-based method, because human eyes are insensitive to high-frequency details, the algorithms only optimize and reconstruct images from the viewpoint of visual features, but neglect the expression of abundant textures and high-frequency details in a real image, and meanwhile, the non-local similarity of spatial textures is not considered, so that the problems of low reconstruction precision, insufficient data feature utilization and loss of texture details often exist for SISR tasks. The high-precision single-frame image super-resolution reconstruction method has great application value in the fields of biomedicine, security monitoring, image processing, pattern recognition, computer vision and the like.
Aiming at the problems of low reconstruction accuracy, insufficient utilization of characteristic information, loss of texture details and the like of the reconstruction algorithm, along with the continuous development of deep learning in recent years, in order to optimize the SISR problem with high quality, reduce the calculation complexity of reconstruction and improve the scene adaptation performance of the algorithm, a method based on a deep neural network is also introduced into the SISR problem, but a simple neural network can acquire better reconstruction performance under extremely large network depth, and a convolution-based method is only effective for local receptive fields of images, so that only an optimization result conforming to the visual perception of human eyes can be reconstructed by a method for deepening the network, and the performance of the method is often poor for some advanced pattern recognition tasks. Therefore, the network depth is reduced, the characteristic information in the low-resolution image is fully utilized, and based on reasonable image global autocorrelation assumption, a great breakthrough is brought to the SISR problem, and the landing process of the SISR problem in technical production and living development is promoted.
Disclosure of Invention
Therefore, the main objective of the present invention is to provide a single-frame image super-resolution reconstruction method based on a dual-channel feature migration network.
In order to achieve the above purpose, the technical scheme of the invention is realized as follows:
the invention discloses a single-frame image super-resolution reconstruction method based on a dual-channel feature migration network, which comprises the following steps:
by carrying out convolution operation sf (·) on the input original low-resolution image x, shallow features F of the image are obtained 1 ；
The shallow features F are corrected by a non-local correlation correction module h (& gt) 1 Performing characteristic channel correction to obtain correction characteristic F 2 ；
The correction feature F 2 Outputting an intermediate feature map by a dual-channel feature migration module RSCAB, and correcting the feature F 2 Calculating the sum of the signal intensities at the corresponding spatial positions, and outputting an intermediate feature F 3 ；
Said intermediate feature F 3 Outputting the fused characteristic F after fusion through a layer of convolution Conv (& gt) 4 ；
The fusion feature F 4 Outputting corrected fusion characteristic F through a non-local correlation correction module T (& gt) 5 ；
The shallow layer feature F 1 Fusion feature F after correction by means of residual connection 5 Calculating the sum of the signal quantities at the corresponding spatial positions and outputting the deep features F 6 Fusion of shallow features F by means of residual connection 1 Obtaining a characteristic diagram F to be rebuilt 7 ；
The to-be-reconstructed featureSign F 7 And outputting a super-resolution reconstruction result y through upsampling of the reconstruction module.
In the above scheme, the shallow feature F of the image is obtained by performing convolution operation sf (·) on the input original low-resolution image x 1 The method specifically comprises the following steps: performing convolution operation sf (·) on an input original low-resolution image x to acquire shallow features F of the image 1 The number of convolution kernels used is 128, the size is 3×3, the step size is 1, and 0 is added at the edge of the convolution map so that the convolution map is consistent with the original image space size, this process can be expressed as equation (1):
F 1 ＝sf(x)＝Relu(Conv 3×3 (x)) (1)。
in the above scheme, the non-local correlation correction module h (·) corrects the shallow feature F 1 Performing characteristic channel correction to obtain correction characteristic F 2 The method specifically comprises the following steps: for the original uncorrected shallow features F 1 The spatial shape of which is represented by [ C, H, W ]]Respectively represent the channel number C, the longitudinal signal number H and the transverse signal number W, and for the channel number H and the transverse signal number W]The 2X 2 area in the dimension takes out the corresponding signals and reconstructs 4 feature subgraphs, and the 4 recombined shapes are [ C, H/2, W/2 ]]Respectively inputting the feature subgraphs of the (4) sub-graphs into an N-L module to obtain 4 output subgraphs, and recombining the 4 subgraphs in space dimension according to the arrangement before subgraph division and then combining with the original feature graph F 1 The signal quantity at the corresponding space position is summed up to obtain a corrected characteristic diagram F 2 。
In the above scheme, the non-local correlation correction module h (·) corrects the shallow feature F 1 Performing characteristic channel correction to obtain correction characteristic F 2 The method specifically comprises the following steps: for the input shape [ C, H, W ]]Feature map F of (1) 2 Using 3 shapes [ C, C/2, 1]]Is convolved by a convolution kernel to obtain 3 convolutions of the shape [ C/2, H W ]]Is characterized by (a)The elements of the feature map are rearranged to obtain the shape of [ C, H, W ]]Is a two-dimensional feature matrix M of (2) 1 、M 2 、M 3 The process is shown in the formula (2):
for M 2 Transpose the feature matrix with M 1 Matrix multiplication is carried out on the feature matrix to obtain a correlation matrixThe shape is [ H X W, H X W ]]The feature map is then activated using a Softmax activation function, M rel Mapped as M' rel As shown in formula (3), wherein (i, j) represents M' rel A signal value for the (i, j) th position;
for M 3 The matrix is transposed and transformed with M' rel Performing matrix multiplication to obtain a spatial attention correction matrix:
performing dimension ascending by using a convolution kernel space attention correction matrix with the shape of [ C/2, C, 1], and performing point multiplication operation of corresponding space positions on the matrix after dimension ascending and an original input feature map to obtain a corrected feature map:
F 2 ＝F 1 gRelu(Conv 1×1 (M attention-fix )) (5)。
in the above scheme, the correction feature F 2 Outputting an intermediate feature map by a dual-channel feature migration module RSCAB, and correcting the feature F 2 Calculating the sum of the signal intensities at the corresponding spatial positions, and outputting an intermediate feature F 3 The method specifically comprises the following steps: correction feature F for input 2 The shape is [ C, H, W ]]Two shapes are used, C,1]Is convolved by a convolution kernel to obtain two shapesIs [ C, H, W]Is expressed as: s is S f ＝Conv 1×1 (F 2 )，C f ＝Conv 1×1 (F 2 )；
For S f Transforming to obtain the shape of [ C, H, W ]]Is a spatial feature transformation feature map S' f ；
Pair C using global average pooling approach f Processing to obtain average statistical vector V of channel dimension c And pair V using Softmax activation function c Nonlinear activation is carried out, and three layers of full-connection networks are used for V c Mapping results in a correction vector V', which can be expressed as: v'. c ＝fc 2 (Softmax(fc 1 (Softmax(V c ) -a) a; in V' c Correcting input feature map C for weight f Obtaining C' f ＝C f ·V′ c ；
For S' f And C' f The shape of [2C, 1 is used]Fusing features and fusing input features F using residual connections 2 As shown in the formula (6), a high-frequency correction characteristic is obtained:
F 3 ＝F 2 +Conv 1×1 (V′+C′ f )＝F 2 +RSCAB(F 2 ) (6)。
in the above scheme, the intermediate feature F 3 Outputting the fused characteristic F after fusion through a layer of convolution Conv (& gt) 4 The method specifically comprises the following steps: for input intermediate feature F 3 The shape is [ C, H, W ]]Two shapes are used, C,1]Is convolved by a convolution kernel to obtain two shapes [ C, H, W ]]Is expressed as: s is S f ＝Conv 1×1 (F 3 )，C f ＝Conv 1×1 (F 3 )；
For S f Transforming to obtain the shape of [ C, H, W ]]Is a spatial feature transformation feature map S' f ；
For C f Obtaining an average statistical vector V of channel dimensions by a global average pooling method c And pair V using Softmax activation function c Non-linear activation and using three-layer full-connectionNetwork pair V c Mapping is performed to obtain a correction vector V', and the process is shown in a formula (7):
V′ c ＝fc 2 (Softmax(fc 1 (Softmax(V c )))) (7)
in V' c Correcting input feature map C for weight f Obtaining C' f ＝C f ·V′ c ；
For S' f And C' f The shape of [2C, 1 is used]Fusing features and fusing input features F using residual connections 3 Obtaining fusion characteristic F 4 The calculation is shown as a formula (8);
F 4 ＝F 3 +Relu(Conv 1×1 (V′+C′ f ))＝F 3 +RSCAB(F 3 ) (8)。
in the above scheme, the fusion feature F 4 Outputting corrected fusion characteristic F through a non-local correlation correction module T (& gt) 5 The method specifically comprises the following steps: for corrected fusion feature F 4 Fusion of upper layer features F using residual connection 3 Nonlinear mapping is carried out on the fusion characteristic diagram through single-layer convolution to obtain F 5 The number of convolution kernels used is 128, the size is 3×3, the step size is 1, and 0 is added at the edge of the convolution map, so that the convolution map keeps consistent with the spatial dimension of the upper layer input feature, and the process is as shown in formula (9):
F 5 ＝Relu(Conv 3×3 (F 4 )) (9)。
in the above aspect, the shallow feature F 1 Fusion feature F after correction by means of residual connection 5 Calculating the sum of the signal quantities at the corresponding spatial positions and outputting the deep features F 6 Fusion of shallow features F by means of residual connection 1 Obtaining a characteristic diagram F to be rebuilt 7 The method specifically comprises the following steps: for fusion feature F 5 Detecting the similarity of the spatial features and obtaining deep features F 6 Fusion of shallow features F by means of residual connection 1 Obtaining a characteristic diagram F to be rebuilt 7 The process is shown in formula (10):
F 7 ＝F 1 +F 6 ＝F 1 +F 5 gRelu(Conv 1×1 (M attention-fix )) (10)。
in the above scheme, the feature map F is to be reconstructed 7 The super-resolution reconstruction result y is output through upsampling of the reconstruction module, specifically: for the characteristic diagram F to be reconstructed 7 The spatial scale of the feature map is transformed using a reconstruction module comprising an upsampling module in the form of [ C, C/(s x s), 3, and a convolution layer]The spatial scale of the input feature map is increased by s x s times and the shape is used as [ C/(s x s), 3]Is convolved with H ↑ Carrying out fusion optimization on the feature images to obtain super-resolution reconstructed images, wherein the process is shown as a formula (11):
y＝Conv 3×3 (H ↑ (F 7 )) (11)
the model is shown in a formula (12) by using a loss function, wherein y is image data in a real training data set, alpha and delta are self-adaptive adjustment parameters according to a training convergence state, and the process optimizes network parameters in an end-to-end mode:
compared with the prior art, the method has the advantages that the network model design is conducted aiming at the problem that the detail information is lost and the local receptive field is not fully utilized for the global-local texture similarity caused by the increase of the depth super-resolution image reconstruction network, the whole network is based on a residual mechanism, and a post up-sampling module is used for up-sampling the image in a space scale; and a non-local correlation correction module is embedded in the front end and the rear end of the network, so that the number of convolution layers in the process of shallow layer feature extraction and advanced feature fusion is obviously reduced, and the spatial non-local correlation features are obviously corrected. The middle high-frequency characteristic reconstruction part is based on a space and channel dual-attention module, a learner structure is designed for purposefully carrying out re-weighted distribution on channel characteristics according to important new aiming at three-dimensional space characteristics of a middle characteristic layer, non-local similarity is calculated on similar texture characteristics of space dimensions, effective distribution of space repeated texture characteristics is kept, residual error structures are used for multiplexing out-of-service characteristics, the context consistency level of a network is improved, the disappearance phenomenon of detail characteristics in the forward transmission process of the network is effectively prevented, and the super-resolution reconstruction quality of single-frame images can be remarkably improved.
Drawings
FIG. 1 is a single-frame image super-resolution reconstruction network model diagram based on a dual-channel feature migration network;
FIG. 2 is a diagram of a network model of non-locally relevant correction module units according to the present invention;
FIG. 3 is a diagram of a network model of a spatial similarity feature detection module (N-L module) unit in a non-local correlation correction module according to the present invention;
FIG. 4 is a network model diagram of the RSCA module unit of the present invention;
fig. 5 is a graph showing the result of super-resolution reconstruction of an image for the case of downsampling rate of 0.0625 (1/42).
FIG. 6 is an infrared artwork of a scene and an infrared super-resolution reconstructed image processed by the present invention;
fig. 7 is a two-infrared original image of a scene and an infrared super-resolution reconstructed image processed by the method.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
The embodiment of the invention provides a single-frame image super-resolution reconstruction method based on a dual-channel feature migration network, which is realized by the following steps:
step 1: by carrying out convolution operation sf (·) on the input original low-resolution image x, shallow features F of the image are obtained 1 。
Specifically, the shallow layer of the image is obtained by convolving the input original low resolution image x with sf (·) operationFeature F 1 The number of convolution kernels used is 128, the size is 3×3, the step size is 1, and 0 is added at the edge of the convolution map so that the convolution map is consistent with the original image space size, this process can be expressed as equation (1):
F 1 ＝sf(x)＝Relu(Conv 3×3 (x)) (1)
step 2: the shallow features F are corrected by a non-local correlation correction module h (& gt) 1 Performing characteristic channel correction to obtain correction characteristic F 2 。
Specifically, the extracted shallow features F 1 Obtaining corrected correction characteristic F after non-local correlation compression module 2 。
For the original uncorrected feature map F 1 The spatial shape of which is represented by [ C, H, W ]]Respectively represent the channel number C, the longitudinal signal number H and the transverse signal number W, and for the channel number H and the transverse signal number W]The 2×2 regions in the dimension respectively take out the corresponding signals and reconstruct into 4 feature subgraphs, as shown in fig. 2;
the recombined 4 shapes are [ C, H/2, W/2 ]]Respectively inputting the feature subgraphs of the (4) sub-graphs into an N-L module to obtain 4 output subgraphs, and recombining the 4 subgraphs in space dimension according to the arrangement before subgraph division and then combining with the original feature graph F 1 The signal quantity at the corresponding space position is summed up to obtain a corrected characteristic diagram F 2 ；
The N-L module can detect the space similar characteristics of the characteristic subgraphs and correct the similar characteristics according to the optimization target, and specifically comprises the following steps:
for the input shape [ C, H, W ]]Feature map x of (2) 1 Using 3 shapes [ C, C/2, 1]]Is a convolution kernel of (2)(i=1, 2, 3) convolving it to give three shapes [ C/2, h×w]Feature map of->The elements of the feature map are rearranged to obtain the shape of [ C, H, W ]]Is a two-dimensional feature matrix M of (2) 1 、M 2 、M 3 ；
For M 2 Transpose the feature matrix with M 1 Matrix multiplication is carried out on the feature matrix to obtain a correlation matrixThe shape is [ H X W, H X W ]]The feature map is then activated using a Softmax activation function, M rel Mapped as M' rel ＝Softmax(M rel ) The calculation formula is as follows: />In [ i, j ]]Represents M' rel I, j of (a)]A numerical value of the location;
for M 3 The matrix is transposed and transformed with M' rel Performing matrix multiplication to obtain a spatial attention correction matrix
The shape is [ C/2, C, 1]]The convolution kernel space attention correction matrix of (2) is subjected to dimension increasing, and the matrix after dimension increasing and the original input feature map are subjected to point multiplication operation of corresponding space positions to obtain a corrected feature map x '' 1 ＝x 1 gConv 1×1 (M attention-fix )。
For the input shape [ C, H, W ]]Feature map F of (1) 2 Using 3 shapes [ C, C/2, 1]]Is convolved by a convolution kernel to obtain 3 convolutions of the shape [ C/2, H W ]]Is characterized by (a)The elements of the feature map are rearranged to obtain the shape of [ C, H, W ]]Is a two-dimensional feature matrix M of (2) 1 、M 2 、M 3 The process is shown in the formula (2):
for M 2 Transpose transformation of feature matrixChange with M 1 Matrix multiplication is carried out on the feature matrix to obtain a correlation matrixThe shape is [ H X W, H X W ]]The feature map is then activated using a Softmax activation function, M rel Mapped as M' rel As shown in formula (3), wherein (i, j) represents M' rel A signal value for the (i, j) th position;
for M 3 The matrix is transposed and transformed with M' rel Performing matrix multiplication to obtain a spatial attention correction matrix:
performing dimension ascending by using a convolution kernel space attention correction matrix with the shape of [ C/2, C, 1], and performing point multiplication operation of corresponding space positions on the matrix after dimension ascending and an original input feature map to obtain a corrected feature map:
F 2 ＝F 1 gRelu(Conv 1×1 (M attention-fix )) (5)
step 3: the correction feature F 2 Outputting an intermediate feature map by a dual-channel feature migration module RSCAB, and correcting the feature F 2 Calculating the sum of the signal intensities at the corresponding spatial positions, and outputting an intermediate feature F 3 。
In particular, a one-layer convolution is used, the shape of which is [ C, C,3]For the correction characteristic F corrected by the non-local correlation compression module 2 Feature fusion is carried out, and the feature fusion is delivered to an RSCAB module for processing to obtain intermediate features F 3 And delivering correction feature F by means of residual connection 2 ；
Correction of feature F using RSCA module 2 Optimizing the space high-frequency information of (2), and outputting optimized high-frequency detailsHigh frequency correction of features feature F 3 The method specifically comprises the following steps:
correction feature F for input 2 The shape is [ C, H, W ]]Two shapes are used, C,1]Is convolved by a convolution kernel to obtain two shapes [ C, H, W ]]Is expressed as: s is S f ＝Conv 1×1 (F 2 )，C f ＝Conv 1×1 (F 2 )；
For S f Transforming to obtain the shape of [ C, H, W ]]Is a spatial feature transformation feature map S' f ；
Pair C using global average pooling approach f Processing to obtain average statistical vector V of channel dimension c And pair V using Softmax activation function c Nonlinear activation is carried out, and three layers of full-connection networks are used for V c Mapping results in a correction vector V', which can be expressed as: v'. c ＝fc 2 (Softmax(fc 1 (Softmax(V c ) -a) a; in V' c Correcting input feature map C for weight f Obtaining C' f ＝C f ·V′ c ；
For S' f And C' f The shape of [2C, 1 is used]Fusing features and fusing input features F using residual connections 2 As shown in the formula (6), a high-frequency correction characteristic is obtained:
F 3 ＝F 2 +Conv 1×1 (V′+C′ f )＝F 2 +RSCAB(F 2 ) (6)。
step 4: said intermediate feature F 3 Outputting the fused characteristic F after fusion through a layer of convolution Conv (& gt) 4 。
In particular, the spatial-channel dual-attention module is used for the intermediate feature F 3 Extracting space and space scale features, and outputting the fused features F after space-channel correction fusion 4 。
For input features F 3 The shape is [ C, H, W ]]Two shapes are used, C,1]Is convolved by a convolution kernel to obtain two shapes [ C, H, W ]]Is expressed as: s is S f ＝Conv 1×1 (F 3 )，C f ＝Conv 1×1 (F 3 )；
For S f Transforming to obtain the shape of [ C, H, W ]]Is a spatial feature transformation feature map S' f ；
For C f The global average pooling method is used for the first time to obtain the average statistical vector V of the channel dimension c And pair V using Softmax activation function c Nonlinear activation is carried out, and three layers of full-connection networks are used for V c Mapping is performed to obtain a correction vector V', and the process is shown in a formula (7):
V′ c ＝fc 2 (Softmax(fc 1 (Softmax(V c )))) (7)
in V' c Correcting input feature map C for weight f Obtaining C' f ＝C f ·V′ c ；
For S' f And C' f The shape of [2C, 1 is used]Fusing features and connecting intermediate features F using residuals 3 Obtaining fusion characteristic F 4 The calculation is shown as a formula (8);
F 4 ＝F 3 +Relu(Conv 1×1 (V′+C′ f ))＝F 3 +RSCAB(F 3 ) (8)。
step 5: the fusion feature F 4 Outputting corrected fusion characteristic F through a non-local correlation correction module T (& gt) 5 。
Specifically, for corrected fusion feature F 4 Fusing intermediate features F using residual connection 3 Nonlinear mapping is carried out on the fusion feature map through single-layer convolution to obtain fusion feature F 5 The number of convolution kernels used is 128, the size is 3×3, the step size is 1, and 0 is added at the edge of the convolution map, so that the convolution map keeps consistent with the spatial dimension of the upper layer input feature, and the process is as shown in formula (9):
F 5 ＝Relu(Conv 3×3 (F 4 )) (9)。
step 6: the shallow layer feature F 1 Through residual connectionIs combined with corrected fusion feature F 5 Calculating the sum of the signal quantities at the corresponding spatial positions and outputting the deep features F 6 Fusion of shallow features F by means of residual connection 1 Obtaining a characteristic diagram F to be rebuilt 7 。
Specifically, for fusion feature F 5 The non-local correlation correction module in the step 2 is used for carrying out space feature similarity detection on the deep features F 6 Fusion of shallow features F by means of residual connection 1 Obtaining a characteristic diagram F to be rebuilt 7 The process is shown in formula (10):
F 7 ＝F 1 +F 6 ＝F 1 +F 5 gRelu(Conv 1×1 (M attention-fix )) (10)
step 8: the characteristic diagram F to be reconstructed 7 And outputting a super-resolution reconstruction result y through upsampling of the reconstruction module.
Specifically, for the feature map F to be reconstructed 7 The spatial scale of the feature map is transformed using a reconstruction module comprising an upsampling module and a convolution layer. The up-sampling module has the shape of [ C, C/(s×s), 3]The spatial scale of the input feature map is increased by s x s times and the shape is used as [ C/(s x s), 3]Is convolved with H ↑ Carrying out fusion optimization on the feature images to obtain super-resolution reconstructed images, wherein the process is shown as a formula (11):
y＝Conv 3×3 (H ↑ (F 7 )) (11)
the model is shown in a formula (12) by using a loss function, wherein y is image data in a real training data set, alpha and delta are self-adaptive adjustment parameters according to a training convergence state, and network parameter optimization is carried out in an end-to-end mode in the process.
In order to prove the effectiveness of the method, super-resolution reconstruction experiments are respectively carried out on visible light, infrared and terahertz images, and fig. 5, 6 and 7 respectively show the super-resolution reconstruction results of the method at the downsampling rate of 4×4.
Fig. 5 shows the super-resolution reconstruction result of the terahertz image by the method of the invention, and it can be seen that the overall quality of the super-resolution result image is improved compared with the original downsampled image, the vein profile of the leaf is clearer, and the local texture detail of the epidermis is more obvious.
Fig. 6 shows the result of super-resolution reconstruction of an infrared image by the method of the present invention, and it can be seen that the overall quality of the super-resolution image is improved compared with the original downsampled image, and the contour and detail features are effectively enhanced.
Fig. 7 shows the super-resolution reconstruction result of the method for the visible light image, and it can be seen that the overall look and feel and quality of the super-resolution result image are obviously improved compared with the original downsampled image, the outlines and the existing detail information of buildings, vehicles, ground facilities and the like in the figure are restored, and the characters are clearer and more discernable.
The same or similar reference numerals in the drawings of the present embodiment correspond to the same or similar components; in the description of the present invention, it should be understood that the directions or positional relationships indicated by the terms "upper", "lower", "left", "right", "inner", "outer", etc. are based on the directions or positional relationships shown in the drawings, are merely for convenience of describing the present invention and simplifying the description, and are not indicative or implying that the devices or elements being referred to must have specific directions, be constructed and operated in specific directions, so that the terms describing the positional relationships in the drawings are merely for exemplary illustration, are not to be construed as limitations of the present patent, and the specific meanings of the terms described above may be understood by those of ordinary skill in the art according to specific circumstances.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, article or apparatus that comprises the element.
The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the present invention.
Claims (8)
1. A single-frame image super-resolution reconstruction method based on a dual-channel feature migration network is characterized by comprising the following steps:
by carrying out convolution operation sf (·) on the input original low-resolution image x, shallow features F of the image are obtained 1 ；
The shallow features F are corrected by a non-local correlation correction module h (& gt) 1 Performing characteristic channel correction to obtain correction characteristic F 2 ；
The correction feature F 2 Outputting an intermediate feature map by a dual-channel feature migration module RSCAB, and correcting the feature F 2 Calculating the sum of the signal intensities at the corresponding spatial positions, and outputting an intermediate feature F 3 ；
Said intermediate feature F 3 Outputting the fused characteristic F after fusion through a layer of convolution Conv (& gt) 4 ；
The fusion feature F 4 Outputting corrected fusion characteristic F through a non-local correlation correction module T (& gt) 5 ；
The shallow layer feature F 1 Fusion feature F after correction by means of residual connection 5 Calculating the sum of the signal quantities at the corresponding spatial positions and outputting the deep features F 6 Fusion of shallow features F by means of residual connection 1 Obtaining a characteristic diagram F to be rebuilt 7 ；
The characteristic diagram F to be reconstructed 7 Outputting a super-resolution reconstruction result y through upsampling of a reconstruction module;
the correction feature F 2 Outputting an intermediate feature map by a dual-channel feature migration module RSCAB, and correcting the feature F 2 Calculating the sum of the signal intensities at the corresponding spatial positions, and outputting an intermediate feature F 3 The method specifically comprises the following steps: correction feature F for input 2 The shape is [ C, H, W ]]Two shapes are used, C,1]Is convolved by a convolution kernel to obtain two shapes [ C, H, W ]]Is expressed as: s is S f ＝Conv 1×1 (F 2 )，C f ＝Conv 1×1 (F 2 )；
For S f Transforming to obtain the shape of [ C, H, W ]]Is a spatial feature transformation feature map S' f ；
Pair C using global average pooling approach f Processing to obtain average statistical vector V of channel dimension c And pair V using Softmax activation function c Nonlinear activation is carried out, and three layers of full-connection networks are used for V c Mapping results in a correction vector V', which can be expressed as: v (V) c ′＝fc 2 (Softmax(fc 1 (Softmax(V c ) -a) a; in V form c ' weight corrected input feature map C f Obtaining C' f ＝C f ·V c ′；
For S' f And C' f The shape of [2C, 1 is used]Fusing features and fusing input features F using residual connections 2 As shown in the formula (6), a high-frequency correction characteristic is obtained:
F 3 ＝F 2 +Conv 1×1 (V′+C′ f )＝F 2 +RSCAB(F 2 ) (6)。
2. the method for reconstructing a single-frame image super-resolution based on a dual-channel feature migration network according to claim 1, wherein the shallow feature F of the image is obtained by performing a convolution operation sf (·) on the input original low-resolution image x 1 The method specifically comprises the following steps: performing convolution operation sf (·) on an input original low-resolution image x to acquire shallow features F of the image 1 The number of convolution kernels used is 128, the size is 3×3, the step size is 1, and 0 is added at the edge of the convolution map so that the convolution map is consistent with the original image space size, this process can be expressed as equation (1):
F 1 ＝sf(x)＝Relu(Conv 3×3 (x)) (1)。
3. the method for reconstructing super-resolution of single-frame image based on dual-channel feature migration network according to claim 1 or 2, wherein the shallow feature F is corrected by a non-local correlation correction module h () 1 Performing characteristic channel correction to obtain correction characteristic F 2 The method specifically comprises the following steps: for the original uncorrected shallow features F 1 The spatial shape of which is represented by [ C, H, W ]]Respectively represent the channel number C, the longitudinal signal number H and the transverse signal number W, and for the channel number H and the transverse signal number W]The 2X 2 area in the dimension takes out the corresponding signals and reconstructs 4 feature subgraphs, and the 4 recombined shapes are [ C, H/2, W/2 ]]Respectively inputting the feature subgraphs of the (4) sub-graphs into an N-L module to obtain 4 output subgraphs, and recombining the 4 subgraphs in space dimension according to the arrangement before subgraph division and then combining with the original feature graph F 1 The signal quantity at the corresponding space position is summed up to obtain a corrected characteristic diagram F 2 。
4. The method for reconstructing a single-frame image super-resolution based on a dual-channel feature migration network as recited in claim 3, wherein said shallow feature F is corrected by a non-local correlation correction module h () 1 Performing characteristic channel correction to obtain correction characteristic F 2 The method specifically comprises the following steps: for the input shape [ C, H, W ]]Feature map F of (1) 2 Using 3 shapes [ C, C/2, 1]]Is convolved by a convolution kernel to obtain 3 convolutions of the shape [ C/2, H W ]]Is characterized by (a)The elements of the feature map are rearranged to obtain the shape of [ C, H, W ]]Is a two-dimensional feature matrix M of (2) 1 、M 2 、M 3 The process is shown in the formula (2):
for M 2 Transpose the feature matrix with M 1 Matrix multiplication is carried out on the feature matrix to obtain a correlation matrixThe shape is [ H X W, H X W ]]The feature map is then activated using a Softmax activation function, M rel Mapping to M r ′ el As shown in formula (3), wherein (i, j) represents M r ′ el A signal value for the (i, j) th position;
for M 3 Transpose the matrix with M r ′ el Performing matrix multiplication to obtain a spatial attention correction matrix:
performing dimension ascending by using a convolution kernel space attention correction matrix with the shape of [ C/2, C, 1], and performing point multiplication operation of corresponding space positions on the matrix after dimension ascending and an original input feature map to obtain a corrected feature map:
F 2 ＝F 1 gRelu(Conv 1×1 (M attention-fix )) (5)。
5. the method for reconstructing super-resolution of single-frame image based on dual-channel feature migration network of claim 4, wherein the intermediate feature F 3 Outputting the fused characteristic F after fusion through a layer of convolution Conv (& gt) 4 The method specifically comprises the following steps: for input intermediate feature F 3 The shape is [ C, H, W ]]Two shapes are used, C,1,1]Is convolved by a convolution kernel to obtain two shapes [ C, H, W ]]Is characterized in that,
expressed as: s is S f ＝Conv 1×1 (F 3 )，C f ＝Conv 1×1 (F 3 )；
For S f Transforming to obtain the shape of [ C, H, W ]]Is a spatial feature transformation feature map S' f ；
For C f Obtaining an average statistical vector V of channel dimensions by a global average pooling method c And pair V using Softmax activation function c Nonlinear activation is carried out, and three layers of full-connection networks are used for V c Mapping is performed to obtain a correction vector V', and the process is shown in a formula (7):
V c ′＝fc 2 (Softmax(fc 1 (Softmax(V c )))) (7)
in V form c ' weight corrected input feature map C f Obtaining C' f ＝C f ·V c ′；
For S' f And C' f The shape of [2C, 1 is used]Fusing features and fusing input features F using residual connections 3 Obtaining fusion characteristic F 4 The calculation is shown as a formula (8);
F 4 ＝F 3 +Relu(Conv 1×1 (V′+C′ f ))＝F 3 +RSCAB(F 3 ) (8)。
6. the method for reconstructing super-resolution of single-frame image based on dual-channel feature migration network of claim 5, wherein the fusion feature F 4 Outputting corrected fusion characteristic F through a non-local correlation correction module T (& gt) 5 The method specifically comprises the following steps: for corrected fusion feature F 4 Fusion of upper layer features F using residual connection 3 Nonlinear mapping is carried out on the fusion characteristic diagram through single-layer convolution to obtain F 5 The number of convolution kernels used is 128, the size is 3×3, the step size is 1, and 0 is added at the edge of the convolution map, so that the space size of the convolution map and the upper-layer input features is ensuredConsistent, the process is shown in formula (9):
F 5 ＝Relu(Conv 3×3 (F 4 )) (9)。
7. the method for reconstructing super-resolution of single-frame image based on dual-channel feature migration network of claim 6, wherein the shallow feature F 1 Fusion feature F after correction by means of residual connection 5 Calculating the sum of the signal quantities at the corresponding spatial positions and outputting the deep features F 6 Fusion of shallow features F by means of residual connection 1 Obtaining a characteristic diagram F to be rebuilt 7 The method specifically comprises the following steps: for fusion feature F 5 Detecting the similarity of the spatial features and obtaining deep features F 6 Fusion of shallow features F by means of residual connection 1 Obtaining a characteristic diagram F to be rebuilt 7 The process is shown in formula (10):
F 7 ＝F 1 +F 6 ＝F 1 +F 5 gRelu(Conv 1×1 (M attention-fix )) (10)。
8. the method for reconstructing super-resolution of single-frame image based on dual-channel feature migration network of claim 7, wherein the feature map F to be reconstructed is characterized by 7 The super-resolution reconstruction result y is output through upsampling of the reconstruction module, specifically: for the characteristic diagram F to be reconstructed 7 The spatial scale of the feature map is transformed using a reconstruction module comprising an upsampling module in the form of [ C, C/(s x s), 3, and a convolution layer]The spatial scale of the input feature map is increased by s x s times and the shape is used as [ C/(s x s), 3]Is convolved with H ↑ Carrying out fusion optimization on the feature images to obtain super-resolution reconstructed images, wherein the process is shown as a formula (11):
y＝Conv 3×3 (H ↑ (F 7 )) (11)
the loss function is shown in a formula (12), wherein y is image data in a real training data set, alpha and delta are self-adaptive adjustment parameters according to training convergence states, and the process optimizes network parameters in an end-to-end mode:
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110268450.9A CN112991173B (en) | 2021-03-12 | 2021-03-12 | Single-frame image super-resolution reconstruction method based on dual-channel feature migration network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110268450.9A CN112991173B (en) | 2021-03-12 | 2021-03-12 | Single-frame image super-resolution reconstruction method based on dual-channel feature migration network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112991173A CN112991173A (en) | 2021-06-18 |
| CN112991173B true CN112991173B (en) | 2024-04-16 |
Family
ID=76335040
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110268450.9A Active CN112991173B (en) | 2021-03-12 | 2021-03-12 | Single-frame image super-resolution reconstruction method based on dual-channel feature migration network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112991173B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TWI788171B (en) * | 2021-09-01 | 2022-12-21 | 鴻海精密工業股份有限公司 | Image processing device and super resolution processing method |
| CN115861045A (en) * | 2021-09-22 | 2023-03-28 | 深圳市中兴微电子技术有限公司 | Image super-resolution method, device, computer equipment and readable medium |
| CN113963176B (en) * | 2021-10-28 | 2023-07-07 | 北京百度网讯科技有限公司 | Model distillation method and device, electronic equipment and storage medium |
| CN114092330B (en) * | 2021-11-19 | 2024-04-30 | 长春理工大学 | Light-weight multi-scale infrared image super-resolution reconstruction method |
| CN114219836B (en) * | 2021-12-15 | 2022-06-03 | 北京建筑大学 | Unmanned aerial vehicle video vehicle tracking method based on space-time information assistance |
| CN114612309B (en) * | 2022-05-12 | 2022-10-14 | 电子科技大学 | Full-on-chip dynamic reconfigurable super-resolution device |
| CN115186774B (en) * | 2022-09-13 | 2022-12-16 | 徐州飞宇机械科技有限公司 | Intelligent cable stripping equipment and method thereof |
| CN115857614B (en) * | 2022-11-17 | 2023-12-29 | 弘正储能(上海)能源科技有限公司 | Multi-path photovoltaic MPPT staggered BOOST control method and system |
| CN117788477B (en) * | 2024-02-27 | 2024-05-24 | 贵州健易测科技有限公司 | Image reconstruction method and device for automatically quantifying tea leaf curl |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111242846A (en) * | 2020-01-07 | 2020-06-05 | 福州大学 | Fine-grained scale image super-resolution method based on non-local enhancement network |
| WO2020238558A1 (en) * | 2019-05-24 | 2020-12-03 | 鹏城实验室 | Image super-resolution method and system |
-
2021
- 2021-03-12 CN CN202110268450.9A patent/CN112991173B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020238558A1 (en) * | 2019-05-24 | 2020-12-03 | 鹏城实验室 | Image super-resolution method and system |
| CN111242846A (en) * | 2020-01-07 | 2020-06-05 | 福州大学 | Fine-grained scale image super-resolution method based on non-local enhancement network |
Non-Patent Citations (1)
| Title |
|---|
| 贡荣麟 ; 施俊 ; 王骏 ; .面向乳腺超声图像分割的混合监督双通道反馈U-Net.中国图象图形学报.2020,(10),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112991173A (en) | 2021-06-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112991173B (en) | Single-frame image super-resolution reconstruction method based on dual-channel feature migration network | |
| CN110992270A (en) | Multi-scale residual attention network image super-resolution reconstruction method based on attention | |
| CN111784602B (en) | Method for generating countermeasure network for image restoration | |
| CN109035142B (en) | Satellite image super-resolution method combining countermeasure network with aerial image prior | |
| CN114119444B (en) | Multi-source remote sensing image fusion method based on deep neural network | |
| CN111444896A (en) | Method for positioning human meridian key points through far infrared thermal imaging | |
| Wang et al. | Laplacian pyramid adversarial network for face completion | |
| Qu et al. | A dual-branch detail extraction network for hyperspectral pansharpening | |
| CN112507997A (en) | Face super-resolution system based on multi-scale convolution and receptive field feature fusion | |
| CN104835130A (en) | Multi-exposure image fusion method | |
| CN113763299B (en) | Panchromatic and multispectral image fusion method and device and application thereof | |
| CN109102485A (en) | Image interfusion method and device based on NSST and adaptive binary channels PCNN | |
| CN110246084A (en) | A kind of super-resolution image reconstruction method and its system, device, storage medium | |
| CN117252761A (en) | Cross-sensor remote sensing image super-resolution enhancement method | |
| CN114862731B (en) | Multi-hyperspectral image fusion method guided by low-rank priori and spatial spectrum information | |
| CN113112583B (en) | 3D human body reconstruction method based on infrared thermal imaging | |
| CN113139585B (en) | Infrared and visible light image fusion method based on unified multi-scale dense connection network | |
| CN115359372A (en) | Unmanned aerial vehicle video moving object detection method based on optical flow network | |
| CN114926343A (en) | Image super-resolution method based on pyramid fusion attention network | |
| CN114897694A (en) | Image super-resolution reconstruction method based on mixed attention and double-layer supervision | |
| CN112767243A (en) | Hyperspectral image super-resolution implementation method and system | |
| CN117788296B (en) | Infrared remote sensing image super-resolution reconstruction method based on heterogeneous combined depth network | |
| Gao et al. | Bayesian image super-resolution with deep modeling of image statistics | |
| CN116468605A (en) | Video super-resolution reconstruction method based on time-space layered mask attention fusion | |
| CN111428809B (en) | Crowd counting method based on spatial information fusion and convolutional neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |