CN111161150B - Image super-resolution reconstruction method based on multi-scale attention cascade network - Google Patents
Image super-resolution reconstruction method based on multi-scale attention cascade network Download PDFInfo
- Publication number
- CN111161150B CN111161150B CN201911392155.3A CN201911392155A CN111161150B CN 111161150 B CN111161150 B CN 111161150B CN 201911392155 A CN201911392155 A CN 201911392155A CN 111161150 B CN111161150 B CN 111161150B
- Authority
- CN
- China
- Prior art keywords
- convolution
- image
- feature
- output
- scale attention
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 60
- 238000013135 deep learning Methods 0.000 claims abstract description 21
- 238000000605 extraction Methods 0.000 claims abstract description 16
- 238000005070 sampling Methods 0.000 claims abstract description 11
- 230000004927 fusion Effects 0.000 claims description 12
- 238000011176 pooling Methods 0.000 claims description 3
- 230000001953 sensory effect Effects 0.000 abstract description 3
- 230000000694 effects Effects 0.000 description 6
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 3
- 238000013527 convolutional neural network Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 230000002427 irreversible effect Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The invention provides an image super-resolution reconstruction method based on a multi-scale attention cascade network, which comprises the steps of firstly, extracting shallow layer features of a low-resolution image by using convolution operation; then, inputting the shallow features into a feature extraction subnet to obtain cascading features; further, the cascade features pass through a convolution layer with a convolution kernel of 1 to obtain optimized features; inputting the optimized features into an image deep learning up-sampling module to obtain a reconstructed image
At the same time, for low resolution image I LR Obtaining a reconstructed image by adopting a bicubic linear interpolation algorithm
Finally, the image is reconstructed
And
fusion to obtain final high-resolution reconstructed image I SR . The method is suitable for super-resolution reconstruction of the image, and the obtained reconstructed image has high definition, more true texture and good sensory effect.
Description
Technical Field
The invention belongs to the field of image restoration, relates to an image super-resolution reconstruction method, and in particular relates to an image super-resolution reconstruction method based on a multi-scale attention cascade network.
Background
Single image super resolution reconstruction (SISR) has recently received a lot of attention. In general, the purpose of SISR is to produce a visual High Resolution (HR) output from a Low Resolution (LR) input. However, the whole process is completely irreversible, as there are several solutions for the mapping between LR and HR. Therefore, a large number of image super-resolution reconstruction (SR) methods have been proposed, from early interpolation-based methods and model-based methods, to more recently depth-learning-based methods.
Interpolation-based methods are simple and fast, but cannot be applied more widely because of poor image quality. For more flexible SR methods, more advanced model-based methods and sparse matrix methods are proposed by exploiting strong image priors, such as non-local similarity, which, while flexible to produce relatively high quality HR images, still have some drawbacks: 1) Such methods often involve a time-consuming optimization process; 2) Reconstruction performance may rapidly degrade when image statistics are biased from images.
Convolutional Neural Networks (CNNs) have now been shown to provide significant performance in SISR problems. However, the conventional SR model has the following problems: 1) The characteristics are not utilized enough: most of the methods blindly increase the depth of the network to improve the performance of the network, but ignore the image feature characteristics of the LR. As the depth of the network increases, the information gradually disappears during transmission. How to fully utilize these features is critical to the network in reconstructing high quality images. 2) SR image detail loss: using an interpolation-amplified LR image as input will increase computational complexity while not favoring the learning of final image details. Therefore, recent methods are more focused on magnifying LR images. However, the effect of the SR image cannot be improved by merely enlarging the SR image with a single network structure.
In order to solve the problems, the invention provides a novel image super-resolution reconstruction method based on deep learning.
Disclosure of Invention
The invention aims to solve the problems that: in the existing super-resolution reconstruction method based on deep learning, most methods blindly increase the depth of a network to improve the performance of the network, but neglect the characteristics of fully utilizing an LR image; and as the depth of the network increases, the characteristic information gradually disappears in the transmission process; the LR image amplified by adopting the interpolation mode is used as the input of the network, so that the calculation complexity is increased, and the learning of the network on the image details is not facilitated. The novel super-resolution reconstruction method based on the deep learning needs to be provided, and the look and feel and the robustness of the image after super-resolution reconstruction are improved.
In order to solve the above problems, the present invention provides an image super-resolution reconstruction method based on a multi-scale attention cascade network, wherein a multi-scale attention block group with a U-shaped structure is used for extracting features of an LR image, and a combination mode of interpolation reconstruction and network reconstruction based on deep learning is adopted for performing super-resolution reconstruction on the LR image, comprising the following steps:
1) Will low resolution image I LR As input to a multi-scale attention cascade network, pair I LR Performing convolution operation to extract shallow layer feature F 0 ；
2) Will shallow layer feature F 0 Inputting a feature extraction sub-network formed by n multi-scale attention blocks, cascading the features output by each multi-scale attention block in the sub-network to obtain cascading features F c ；
3) Will cascade feature F c The number of parameters is reduced by a convolution layer with a convolution kernel of 1, and an optimized feature vector F is obtained d Optimized feature vector F d The training and the feature extraction of the data can be more effectively and intuitively carried out;
4) Feature vector F to be optimized d In the input image deep learning up-sampling module, a reconstructed image is obtained
5) Pair I LR Obtaining a reconstructed image by adopting an interpolation algorithm
Will->
And->
Fusion is carried out to obtain a final reconstructed image I SR 。
As a further preferred mode, the obtaining cascade features in step 2) is specifically:
2.1 To shallow layer feature F) 0 Inputting into a feature extraction subnet composed of n multi-scale attention blocks to obtain n features F respectively i ，i＝1,2,3,…,n。
For the ith multi-scale attention block, the input is the feature output F of the previous multi-scale attention block i-1 The output is characterized by F i 。
Each multi-scale attention block consists of a U-shaped structure module, a bottleneck layer structure module and a residual error module.
2.1.1 For the U-shaped structure module in the ith multiscale attention block, the U-shaped structure module is formed by non-local mean, 3×3 convolution, 5×5 convolution, 7×7 convolution, attention mechanism, 5×5 convolution, 3×3 convolution, and non-local mean series; furthermore, a Concat layer is included between the two 3 x 3 convolutions; two 5 x 5 convolutions contain a Concat layer between them. The characteristic input being the output characteristic F of the preceding multiscale attention block i-1 After the U-shaped structural module is processed, the characteristic F is obtained i,0 ；
For the characteristic input F of the non-local mean value, firstly, the input characteristic F is respectively input into three parallel convolution layers to obtain three characteristics F x ,F y ,F z Then, carrying out feature fusion on the three features through Concat operation, and inputting the fused features into a subsequent convolution layer to obtain a feature F w Finally, feature F w And adding the non-local mean value with the input characteristic F point by point to obtain the output characteristic of the non-local mean value.
For input feature F of the attention mechanism, first, feature F is input to the global pooling layer extraction channel information descriptor M avg Then, the channel information descriptor M avg And (3) inputting the obtained product into two subsequent convolution layers for further processing to obtain M, and finally multiplying the M with the characteristic F channel by channel to obtain the output characteristic of the attention mechanism.
2.1.2 For the bottleneck layer structure module in the ith multi-scale attention block, the bottleneck layer structure module is composed of two bottleneck layersAnd the two are connected in series. The input of the input is the output characteristic F of the U-shaped structure module in the multi-scale attention block i,0 After the bottleneck layer structure module is processed, the characteristic F is obtained i,2 ；
2.1.3 For the residual block in the ith multi-scale attention block, the residual block adds the output features of the previous multi-scale attention block and the output features of the bottleneck layer structure point by point. The input of which is the output characteristic F of the preceding multiscale attention block i-1 And the output characteristics F of the bottleneck layer structure module in the multi-scale attention block i,2 After processing by a residual error module, obtaining a characteristic F i ；
2.2 Feature F) output for n multi-scale attention blocks i I=1, 2,3, …, n, using the Concat join operation, gives a cascade feature F c ：
F c ＝Concat(F 1 ,F 2 ,...,F n )
Where Concat (-) represents the operation of concatenating the features of the n multi-scale attention block outputs.
As a further preferred mode, step 3) is specifically:
3.1 To cascade feature F) c Inputting into a convolution layer with a convolution kernel of 1, reducing the number of parameters, and obtaining an optimized characteristic F d :
F d ＝Conv 1×1 (F c )
Wherein Conv 1×1 (. Cndot.) represents a convolution operation with a convolution kernel of 1.
As a further preferred mode, step 4) the obtaining of the reconstructed image of the image deep learning upsampling module
The image deep learning up-sampling module consists of a convolution layer with a convolution kernel of 3 and a sub-pixel convolution layer. Optimized feature F d Obtaining a reconstructed image through an image deep learning up-sampling module
The specific process of (2) is as follows:
4.1 Using a convolution layer with a convolution kernel of 3 for the optimized feature F d Rearranging to obtain feature F e ：
F e ＝Conv 3×3 (F d )
Wherein Conv 3×3 (. Cndot.) represents a convolution operation with a convolution kernel of 3.
4.2 To rearranged features F) e Is input into a sub-pixel convolution layer, amplified to a corresponding scale, and a reconstructed image is obtained
Wherein H is Sp (. Cndot.) represents a subpixel convolution operation.
As a further preferred mode, step 5) said obtaining a final reconstructed image I SR The method comprises the following specific steps of:
5.1 For low resolution image I) LR Obtaining an interpolated reconstructed image by using a bicubic linear interpolation algorithm
5.2 A reconstructed image obtained by the image deep learning up-sampling module)
And reconstructed image interpolated by bicubic linear interpolation algorithm +.>
Fusing to obtain a final reconstructed image I SR ：
Although the super-resolution reconstruction method using the interpolation algorithm is high in reconstruction speed, redundant information is added, and the super-resolution reconstruction effect is poor; the image super-resolution reconstruction method based on the deep learning is lack of a reasonable instruction in the reconstruction process, so that part of detail information in the reconstructed image is lost. The reconstruction result of the interpolation algorithm is used as the guidance of the reconstruction process of the image super-resolution reconstruction method based on the deep learning, so that the effect of image super-resolution reconstruction can be improved, and redundant information generated by the interpolation algorithm can be removed.
The invention provides an image super-resolution reconstruction method of a multi-scale attention cascade network, which comprises the steps of firstly extracting shallow layer features of a low-resolution image by using convolution operation; then, inputting the shallow features into a feature extraction subnet to obtain cascading features; further, the cascade features pass through a convolution layer with a convolution kernel of 1 to obtain an optimized feature vector; inputting the optimized feature vector into an image deep learning up-sampling module to obtain a reconstructed image
At the same time, for low resolution image I LR Obtaining a reconstructed image by means of an interpolation algorithm>
Finally, the reconstructed image->
And->
Fusion to obtain final high-resolution reconstructed image I SR . By the method, the problem of low detail definition of the reconstructed image by the existing image super-resolution reconstruction method is solved, and the appearance is improved; the method also solves the problem that the existing image super-resolution reconstruction algorithm based on deep learning cannot fully extract the low-resolution image features. The invention is suitable for super-resolution reconstruction of images and uses the inventionThe super-resolution reconstruction of the image is performed obviously, the obtained reconstructed image has high definition, more true texture and good perception effect.
Advantageous effects
Firstly, the invention adopts a group of multi-scale attention blocks to extract the characteristics of the low-resolution image, and can fully utilize the detail information of the low-resolution image; and secondly, realizing super-resolution reconstruction by adopting a mode of combining interpolation reconstruction and network reconstruction based on deep learning, and improving the reconstruction effect of the reconstructed image.
Drawings
FIG. 1 is a flow chart of an image super-resolution reconstruction method based on a multi-scale attention cascade network;
FIG. 2 is a network structure diagram of an image super-resolution method based on a multi-scale attention cascade network of the invention;
FIG. 3 is a block diagram of a multi-scale attention module of the present invention;
Detailed Description
The invention provides an image super-resolution reconstruction method of a multi-scale attention cascade network, which comprises the steps of firstly extracting shallow layer features of a low-resolution image by using convolution operation; then, inputting the shallow features into a feature extraction subnet to obtain cascading features; further, the cascade features pass through a convolution layer with a convolution kernel of 1 to obtain an optimized feature vector; inputting the optimized feature vector into an image deep learning up-sampling module to obtain a reconstructed image
At the same time, for low resolution image I LR Obtaining a reconstructed image by means of an interpolation algorithm>
Finally, the reconstructed image->
And->
Fusion to obtain the final productHigh resolution reconstructed image I SR . The method is suitable for super-resolution reconstruction of the image, and the obtained high-resolution image has high definition, more real texture and good sensory effect by using the method to reconstruct the super-resolution.
As shown in fig. 1, the present invention includes the steps of:
1) Will low resolution image I LR As input to the multiscale attention cascade network, a convolution operation is used from the low resolution image I LR Extracting shallow layer characteristic F 0 ：
F 0 ＝H sf (I LR )
Wherein H is sf () Representing a convolution operation.
2) Will shallow layer feature F 0 Inputting a feature extraction subnet composed of a group of multi-scale attention blocks, cascading the features output by each multi-scale attention block in the subnet by using Cancat operation to obtain cascading features F c ；
2.1 To shallow layer feature F) 0 Inputting into a feature extraction subnet composed of n multi-scale attention blocks to obtain n features F respectively i ，i＝1,2,3,…,n。
For the ith multi-scale attention block, the input is the feature output F of the previous multi-scale attention block i-1 The output is characterized by F i 。
Each multi-scale attention block consists of a U-shaped structure module, a bottleneck layer structure module and a residual error module.
2.1.1 For the U-shaped structure module in the ith multiscale attention block, the module is formed by non-local mean, 3×3 convolution, 5×5 convolution, 7×7 convolution, attention mechanism, 5×5 convolution, 3×3 convolution, non-local mean series; furthermore, a Concat layer is included between the two 3 x 3 convolutions; two 5 x 5 convolutions contain a Concat layer between them. The input of which is the output characteristic F of the preceding multiscale attention block i-1 The output of the U-shaped structural module being characteristic F i,0 ；
Will feature F i-1 Input to the U-shaped structure module to obtain the characteristic F i,0 ：
F i,0 ＝H u (F i-1 )
Wherein H is u Representing the extraction of feature operations using U-shaped structural modules.
2.1.2 For the bottleneck layer structure module in the ith multi-scale attention block, the bottleneck layer structure module is formed by connecting two bottleneck layers in series. The input of the input is the output characteristic F of the U-shaped structure module in the multi-scale attention block i,0 The output of the bottleneck layer structure module is characterized by F i,2 ；
Each bottleneck layer structure module consists of two bottleneck layers. The feature input of the first bottleneck layer is the feature output F of the previous multi-scale attention block i-1 The characteristic output is F i,1 ：
F i,1 ＝H b (F i-1 )
Wherein H is b (. Cndot.) represents the first bottleneck layer operation.
Further extracting the characteristic F from the U-shaped structure module i,0 And the output F of the first bottleneck layer i,1 Inputting to a second bottleneck layer for detail fusion to obtain a feature F i,2 ：
F i,2 ＝H c (F i,0 ,F i,1 )
Wherein H is c (. Cndot.) represents a second bottleneck layer operation.
2.1.3 For the residual block in the ith multi-scale attention block, the block adds the output features of the previous multi-scale attention block and the output features of the bottleneck layer structure point by point. The input of which is the output characteristic F of the preceding multiscale attention block i-1 And the output characteristics F of the bottleneck layer structure module in the multi-scale attention block i,2 The output of the residual module is characteristic F i ；
Output feature F of previous multiscale attention block i-1 And feature F i,2 Input to a residual error module to obtain a characteristic F i ：
F i ＝F i-1 +F i,2
2.2 Feature F) output for n multi-scale attention blocks i I=1, 2,3, …, n, using a Concat connectionOperation to obtain cascade characteristic F c ：
F c ＝Concat(F 1 ,F 2 ,...,F n )
Where Concat (-) represents the operation of concatenating the features of the n multi-scale attention block outputs.
3) Will cascade feature F c The number of parameters is reduced by a convolution layer with a convolution kernel of 1, and an optimized feature vector F is obtained d Optimized feature vector F d The training and the feature extraction of the data can be more effectively and intuitively carried out;
3.1 To cascade feature F) c Inputting into a convolution layer with a convolution kernel of 1, reducing the number of parameters, and obtaining an optimized characteristic F d :
F d ＝Conv 1×1 (F c )
Wherein Conv 1×1 (. Cndot.) represents a convolution operation with a convolution kernel of 1.
4) Feature vector F to be optimized d In the input image deep learning up-sampling module, a reconstructed image is obtained
4.1 Using a convolution layer with a convolution kernel of 3 for the optimized feature F d Rearranging to obtain feature F e ：
F e ＝Conv 3×3 (F d )
Wherein Conv 3×3 (. Cndot.) represents a convolution operation with a convolution kernel of 3.
4.2 To rearranged features F) e Is input into a sub-pixel convolution layer, amplified to a corresponding scale, and a reconstructed image is obtained
Wherein H is Sp (. Cndot.) represents a subpixel convolutional layer operation.
5) Pair I LR Obtaining a reconstructed image by adopting an interpolation algorithm
Will->
And->
Fusion is carried out to obtain a final reconstructed image I SR 。
5.1 For low resolution image I) LR Obtaining an interpolated reconstructed image by using a bicubic linear interpolation algorithm
5.2 A reconstructed image obtained by the image deep learning up-sampling module)
And reconstructed image interpolated by bicubic linear interpolation algorithm +.>
Fusing to obtain a final reconstructed image I SR ：
Wherein I is SR And (5) obtaining a final image super-resolution reconstruction result.
The invention has wide application in the field of image restoration, such as throwing large-size photo billboards, reducing image transmission pressure, enlarging thumbnail and the like. The present invention will be described in detail below with reference to the accompanying drawings.
1) Will low resolution image I LR As input to a multi-scale attention cascade network, pair I LR Performing convolution operationExtracted shallow features F 0 ；
2) Will shallow layer feature F 0 Inputting a feature extraction sub-network formed by a group of multi-scale attention blocks, cascading the features output by each multi-scale attention block in the sub-network to obtain cascading features F c ；
3) Will cascade feature F c The number of parameters is reduced by a convolution layer with a convolution kernel of 1, and an optimized feature vector F is obtained d ；
4) Feature vector F to be optimized d In the input image deep learning up-sampling module, a reconstructed image is obtained
5) For low resolution image I LR Obtaining a reconstructed image by adopting a bicubic linear interpolation algorithm
Will->
And->
Fusion is carried out to obtain a final reconstructed image I SR 。
The method was implemented based on the PyTorch deep learning framework under NVIDIA GeForce GTX 1080Ti and Ubuntu16.04 bit operating systems.
The invention provides an image super-resolution reconstruction method based on a multi-scale attention cascade network. The method is suitable for super-resolution reconstruction of the image, the super-resolution reconstruction is carried out by using the method, the obtained reconstructed image has high definition, more true texture and good sensory effect.
Claims (8)
1. The image super-resolution reconstruction method based on the multi-scale attention cascade network is characterized by comprising the following steps of:
step 1) image I of low resolution LR As a plurality ofInput of scale attention cascade network, for I LR Performing a convolution operation to extract shallow features F 0 ；
Step 2) shallow layer feature F 0 Inputting a feature extraction subnet composed of n multi-scale attention blocks, cascading the features output by each multi-scale attention block in the subnet to obtain cascading features F c The method comprises the steps of carrying out a first treatment on the surface of the Each multi-scale attention block consists of a U-shaped structure module, a bottleneck layer structure module and a residual error module;
step 3) feature F will cascade c The optimized feature vector F is obtained by a convolution layer with a convolution kernel of 1 d Optimized feature vector F d The training and feature extraction of the data can be more effectively and intuitively carried out, and the method is specifically shown as follows:
F d ＝Conv 1×1 (F c )
wherein Conv 1×1 (g) A convolution operation representing a convolution kernel of 1;
step 4) optimizing feature vector F d In the input image deep learning up-sampling module, a reconstructed image is obtained
Step 5) for low resolution image I LR Obtaining a reconstructed image by adopting a bicubic linear interpolation algorithm
Will->
And->
Fusion is carried out to obtain a final reconstructed image I SR The method is characterized by comprising the following steps:
2. the image super-resolution reconstruction method based on the multi-scale attention cascade network according to claim 1, wherein the method comprises the following steps of: the feature extraction sub-network described in the step 2 is composed of n multi-scale attention blocks, wherein the ith multi-scale attention block has the feature output F input as the previous multi-scale attention block i-1 The output is characterized by F i ；
For the U-shaped structural module in the ith multi-scale attention block, the input is the output characteristic F of the previous multi-scale attention block i-1 After the U-shaped structural module is processed, the characteristic F is obtained i,0 ；
For the bottleneck layer structure module in the ith multi-scale attention block, the bottleneck layer structure module is formed by connecting two bottleneck layers in series; output feature F of the previous multiscale attention block i-1 Inputting a first bottleneck layer, inputting the output of the first bottleneck layer into a second bottleneck layer, and simultaneously receiving the output characteristics F of the U-shaped structure module in the multi-scale attention block by the second bottleneck layer i,0 The output characteristic F of the bottleneck layer structure module is obtained through the process i,2 ；
The residual modules in the ith multi-scale attention block are specifically: output feature F of the previous multi-scale attention block i-1 And output features F of bottleneck layer structure i,2 Adding point by point to obtain feature F i 。
3. The image super-resolution reconstruction method based on the multi-scale attention cascade network according to claim 2, wherein the method comprises the following steps of:
the U-shaped structure module is formed by serial connection of a non-local mean value, a 3X 3 convolution, a 5X 5 convolution, a 7X 7 convolution, an attention mechanism, a 5X 5 convolution, a 3X 3 convolution and a non-local mean value; wherein a Concat layer is contained between the two 3X 3 convolutions; a Concat layer is arranged between the two 5X 5 convolution convolutions; the first 5 x 5 convolved input is the sum of the first non-local mean output and the first 3 x 3 convolved output, the 7 x 7 convolved input is the sum of the first 5 x 5 convolved output and the first non-local mean output, the second 5 x 5 convolved input is the Concat fusion of the first 5 x 5 convolved output and the attention mechanism output, and the second 3 x 3 convolved input is the Concat fusion of the first 3 x 3 convolved output and the second 5 x 5 convolved output.
4. A method for reconstructing an image super-resolution based on a multi-scale attention cascade network according to claim 3, wherein: the non-local mean value is formed by a convolution with three parallel convolution kernels of 1 and a convolution series for feature fusion;
for the characteristic input F of the non-local mean value, firstly, the input characteristic F is respectively input into three parallel convolution layers to obtain three characteristics F x ,F y ,F z Then, carrying out feature fusion on the three features through Concat operation, and inputting the fused features into a subsequent convolution layer to obtain a feature F w Finally, feature F w And adding the non-local mean value with the input characteristic F point by point to obtain the output characteristic of the non-local mean value.
5. A method for reconstructing an image super-resolution based on a multi-scale attention cascade network according to claim 3, wherein: the attention mechanism is formed by sequentially connecting a global pooling layer and two convolutions;
for input feature F of the attention mechanism, first, feature F is input to the global pooling layer extraction channel information descriptor M avg Then, the channel information descriptor M avg And (3) inputting the obtained product into two subsequent convolution layers for further processing to obtain M, and finally multiplying the M with the characteristic F channel by channel to obtain the output characteristic of the attention mechanism.
6. The image super-resolution reconstruction method based on the multi-scale attention cascade network according to claim 2, wherein the method comprises the following steps of: the bottleneck layer is formed by connecting two convolution layers in series.
7. The image super-resolution reconstruction method based on the multi-scale attention cascade network according to claim 1, wherein the method comprises the following steps of: the cascade operation described in step 2 is specifically represented as follows:
F c ＝Concat(F 1 ,F 2 ,...,F n )
wherein Concat (g) represents an operation of concatenating the characteristics of the n multi-scale attention block outputs, F i I=1, 2,3, …, n denotes the characteristics of the n multi-scale attention block outputs.
8. The method for reconstructing an image super-resolution based on a multi-scale attention cascade network as recited in claim 1, wherein said obtaining a reconstructed image of step 4) is performed by
The specific steps of (a) are as follows:
4.1 Using a convolution layer with a convolution kernel of 3 for the optimized feature F d Rearranging to obtain feature F e ：
F e ＝Conv 3×3 (F d )
Wherein Conv 3×3 (g) A convolution operation representing a convolution kernel of 3;
4.2 To rearranged features F) e Is input into a sub-pixel convolution layer, amplified to a corresponding scale, and a reconstructed image is obtained
Wherein H is Sp (g) Representing a sub-pixel convolution operation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911392155.3A CN111161150B (en) | 2019-12-30 | 2019-12-30 | Image super-resolution reconstruction method based on multi-scale attention cascade network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911392155.3A CN111161150B (en) | 2019-12-30 | 2019-12-30 | Image super-resolution reconstruction method based on multi-scale attention cascade network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111161150A CN111161150A (en) | 2020-05-15 |
| CN111161150B true CN111161150B (en) | 2023-06-23 |
Family
ID=70558951
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911392155.3A Active CN111161150B (en) | 2019-12-30 | 2019-12-30 | Image super-resolution reconstruction method based on multi-scale attention cascade network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111161150B (en) |
Families Citing this family (25)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111368849B (en) * | 2020-05-28 | 2020-08-28 | 腾讯科技（深圳）有限公司 | Image processing method, image processing device, electronic equipment and storage medium |
| CN111915481B (en) * | 2020-06-08 | 2024-03-29 | 北京大米未来科技有限公司 | Image processing method, device, electronic equipment and medium |
| CN111724308A (en) * | 2020-06-28 | 2020-09-29 | 深圳壹账通智能科技有限公司 | Blurred image processing method and system |
| CN111951164B (en) * | 2020-08-11 | 2023-06-16 | 哈尔滨理工大学 | Image super-resolution reconstruction network structure and image reconstruction effect analysis method |
| CN111970513A (en) * | 2020-08-14 | 2020-11-20 | 成都数字天空科技有限公司 | Image processing method and device, electronic equipment and storage medium |
| CN112070669B (en) * | 2020-08-28 | 2024-04-16 | 西安科技大学 | Super-resolution image reconstruction method for arbitrary fuzzy core |
| CN112116527B (en) * | 2020-09-09 | 2024-02-23 | 北京航空航天大学杭州创新研究院 | Image super-resolution method based on cascade network frame and cascade network |
| CN112215755B (en) * | 2020-10-28 | 2023-06-23 | 南京信息工程大学 | Image super-resolution reconstruction method based on back projection attention network |
| CN112381839B (en) * | 2020-11-14 | 2022-08-02 | 四川大学华西医院 | Breast cancer pathological image HE cancer nest segmentation method based on deep learning |
| CN113160047B (en) * | 2020-11-23 | 2023-05-23 | 南京邮电大学 | Single image super-resolution method based on multi-scale channel attention mechanism |
| CN112580473B (en) * | 2020-12-11 | 2024-05-28 | 北京工业大学 | Video super-resolution reconstruction method integrating motion characteristics |
| CN112801868B (en) * | 2021-01-04 | 2022-11-11 | 青岛信芯微电子科技股份有限公司 | Method for image super-resolution reconstruction, electronic device and storage medium |
| CN112767247A (en) * | 2021-01-13 | 2021-05-07 | 京东方科技集团股份有限公司 | Image super-resolution reconstruction method, model distillation method, device and storage medium |
| CN113012046B (en) * | 2021-03-22 | 2022-12-16 | 华南理工大学 | Image super-resolution reconstruction method based on dynamic packet convolution |
| CN113222822B (en) * | 2021-06-02 | 2023-01-24 | 西安电子科技大学 | Hyperspectral image super-resolution reconstruction method based on multi-scale transformation |
| CN113674156B (en) * | 2021-09-06 | 2022-12-30 | 苏州大学 | Method and system for reconstructing image super-resolution |
| CN113688783B (en) * | 2021-09-10 | 2022-06-28 | 一脉通(深圳)智能科技有限公司 | Face feature extraction method, low-resolution face recognition method and equipment |
| CN113538250A (en) * | 2021-09-14 | 2021-10-22 | 苏州微清医疗器械有限公司 | Fundus shooting system with function of rapidly processing images |
| CN113763251B (en) * | 2021-09-14 | 2023-06-16 | 浙江师范大学 | Image super-resolution amplification model and method thereof |
| CN114025200B (en) * | 2021-09-15 | 2022-09-16 | 湖南广播影视集团有限公司 | Ultra-high definition post-production solution based on cloud technology |
| CN113837946B (en) * | 2021-10-13 | 2022-12-06 | 中国电子技术标准化研究院 | Lightweight image super-resolution reconstruction method based on progressive distillation network |
| WO2024007160A1 (en) * | 2022-07-05 | 2024-01-11 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | Convolutional neural network (cnn) filter for super-resolution with reference picture resampling (rpr) functionality |
| CN115564653A (en) * | 2022-09-30 | 2023-01-03 | 江苏济远医疗科技有限公司 | Multi-factor fusion image super-resolution method |
| CN115546032B (en) * | 2022-12-01 | 2023-04-21 | 泉州市蓝领物联科技有限公司 | Single-frame image super-resolution method based on feature fusion and attention mechanism |
| CN116797456A (en) * | 2023-05-12 | 2023-09-22 | 苏州大学 | Image super-resolution reconstruction method, system, device and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106709875A (en) * | 2016-12-30 | 2017-05-24 | 北京工业大学 | Compressed low-resolution image restoration method based on combined deep network |
| CN107240066A (en) * | 2017-04-28 | 2017-10-10 | 天津大学 | Image super-resolution rebuilding algorithm based on shallow-layer and deep layer convolutional neural networks |
| CN109886871A (en) * | 2019-01-07 | 2019-06-14 | 国家新闻出版广电总局广播科学研究院 | The image super-resolution method merged based on channel attention mechanism and multilayer feature |
| CN109903228A (en) * | 2019-02-28 | 2019-06-18 | 合肥工业大学 | A kind of image super-resolution rebuilding method based on convolutional neural networks |
| CN110136060A (en) * | 2019-04-24 | 2019-08-16 | 西安电子科技大学 | The image super-resolution rebuilding method of network is intensively connected based on shallow-layer |
| CN110276721A (en) * | 2019-04-28 | 2019-09-24 | 天津大学 | Image super-resolution rebuilding method based on cascade residual error convolutional neural networks |
-
2019
- 2019-12-30 CN CN201911392155.3A patent/CN111161150B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106709875A (en) * | 2016-12-30 | 2017-05-24 | 北京工业大学 | Compressed low-resolution image restoration method based on combined deep network |
| CN107240066A (en) * | 2017-04-28 | 2017-10-10 | 天津大学 | Image super-resolution rebuilding algorithm based on shallow-layer and deep layer convolutional neural networks |
| CN109886871A (en) * | 2019-01-07 | 2019-06-14 | 国家新闻出版广电总局广播科学研究院 | The image super-resolution method merged based on channel attention mechanism and multilayer feature |
| CN109903228A (en) * | 2019-02-28 | 2019-06-18 | 合肥工业大学 | A kind of image super-resolution rebuilding method based on convolutional neural networks |
| CN110136060A (en) * | 2019-04-24 | 2019-08-16 | 西安电子科技大学 | The image super-resolution rebuilding method of network is intensively connected based on shallow-layer |
| CN110276721A (en) * | 2019-04-28 | 2019-09-24 | 天津大学 | Image super-resolution rebuilding method based on cascade residual error convolutional neural networks |
Non-Patent Citations (2)
| Title |
|---|
| Image Super-Resolution Using Very Deep Residual Channel Attention Networks;Yulun Zhang et al.;ECCV 2018;1-16 * |
| 基于运动特征融合的快速视频超分辨率重构方法;付利华 等;模式识别与人工智能;第32卷(第11期);1022-1031 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111161150A (en) | 2020-05-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111161150B (en) | Image super-resolution reconstruction method based on multi-scale attention cascade network | |
| CN110033410B (en) | Image reconstruction model training method, image super-resolution reconstruction method and device | |
| CN110136062B (en) | Super-resolution reconstruction method combining semantic segmentation | |
| CN111861961B (en) | Single image super-resolution multi-scale residual error fusion model and restoration method thereof | |
| CN109035146B (en) | Low-quality image super-resolution method based on deep learning | |
| CN111932461A (en) | Convolutional neural network-based self-learning image super-resolution reconstruction method and system | |
| CN110634105A (en) | Video high-space-time resolution signal processing method combining optical flow method and deep network | |
| CN113837946B (en) | Lightweight image super-resolution reconstruction method based on progressive distillation network | |
| CN112419152B (en) | Image super-resolution method, device, terminal equipment and storage medium | |
| CN113222818A (en) | Method for reconstructing super-resolution image by using lightweight multi-channel aggregation network | |
| CN111654621B (en) | Dual-focus camera continuous digital zooming method based on convolutional neural network model | |
| Li et al. | Dlgsanet: lightweight dynamic local and global self-attention networks for image super-resolution | |
| CN115358932A (en) | Multi-scale feature fusion face super-resolution reconstruction method and system | |
| CN115953294A (en) | Single-image super-resolution reconstruction method based on shallow channel separation and aggregation | |
| CN115797176A (en) | Image super-resolution reconstruction method | |
| CN115619645A (en) | Image super-resolution reconstruction method based on multi-stage residual jump connection network | |
| Zhou et al. | Image super-resolution based on adaptive cascading attention network | |
| CN114897704A (en) | Single-image super-resolution algorithm based on feedback mechanism | |
| Wu et al. | Lightweight asymmetric convolutional distillation network for single image super-resolution | |
| CN114049251A (en) | Fuzzy image super-resolution reconstruction method and device for AI video analysis | |
| CN113240584A (en) | Multitask gesture picture super-resolution method based on picture edge information | |
| CN113362239A (en) | Deep learning image restoration method based on feature interaction | |
| CN116485654A (en) | Lightweight single-image super-resolution reconstruction method combining convolutional neural network and transducer | |
| CN116797456A (en) | Image super-resolution reconstruction method, system, device and storage medium | |
| CN111402140A (en) | Single image super-resolution reconstruction system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |