CN110047038B - Single-image super-resolution reconstruction method based on hierarchical progressive network - Google Patents
Single-image super-resolution reconstruction method based on hierarchical progressive network Download PDFInfo
- Publication number
- CN110047038B CN110047038B CN201910146330.4A CN201910146330A CN110047038B CN 110047038 B CN110047038 B CN 110047038B CN 201910146330 A CN201910146330 A CN 201910146330A CN 110047038 B CN110047038 B CN 110047038B
- Authority
- CN
- China
- Prior art keywords
- layer
- resolution
- super
- convolution
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 24
- 230000000750 progressive effect Effects 0.000 title claims abstract description 21
- 238000000605 extraction Methods 0.000 claims abstract description 15
- 238000012549 training Methods 0.000 claims abstract description 6
- 239000010410 layer Substances 0.000 claims description 57
- 238000013507 mapping Methods 0.000 claims description 15
- 238000005070 sampling Methods 0.000 claims description 10
- 230000004913 activation Effects 0.000 claims description 8
- 230000007704 transition Effects 0.000 claims description 6
- 238000010606 normalization Methods 0.000 claims description 5
- 239000002356 single layer Substances 0.000 claims description 4
- 238000012545 processing Methods 0.000 abstract description 4
- 230000006870 function Effects 0.000 description 11
- 238000010586 diagram Methods 0.000 description 7
- 230000008034 disappearance Effects 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 101100365548 Caenorhabditis elegans set-14 gene Proteins 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000002059 diagnostic imaging Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
- Compression Of Band Width Or Redundancy In Fax (AREA)
Abstract
The invention provides a single image super-resolution reconstruction method based on a hierarchical progressive network, which mainly comprises the following steps: the method comprises a characteristic extraction branch, an image reconstruction branch, a hierarchical progressive network structure and a loss function. The method decomposes a high-power super-resolution task into a plurality of subtasks, each subtask can be independently completed by one super-resolution unit network, and the plurality of super-resolution unit networks are cascaded to form the whole network. The invention can use the same training model to carry out super-resolution reconstruction processing on images with multiple times.
Description
Technical Field
The invention relates to the technical field of image super-resolution, in particular to a single image super-resolution reconstruction method based on a hierarchical progressive network.
Background
The image super-resolution is a technology for restoring a corresponding high-resolution image with more detail information by using a single low-resolution image or a low-resolution image sequence with sub-pixel offset on the basis of the condition of original hardware equipment. The restored image can express potential details and hidden structures, and the visual effect of the image is enhanced. The image super-resolution technology plays an important role in the fields of medical imaging, safety monitoring, audio-video entertainment, satellite remote sensing and the like.
At present, a single-image super-resolution reconstruction technology which is applied more is a learning-based method, and the method comprises a method based on ultra-complete sparse dictionary learning, a method based on a deep convolutional neural network and the like. The method can obtain a better reconstruction effect under a specific scene and a low-power super-resolution task. However, satisfactory results are still not obtained in the context of distributed generalization, especially for high-power super-resolution tasks. On the other hand, for the multi-scale super-resolution task, most of the existing methods adopt a strategy of learning for multiple times to obtain the mapping from the low-resolution images with different scales to the high-resolution images, which greatly increases the learning cost.
Disclosure of Invention
The invention aims to provide a method for single-image super-resolution reconstruction, which solves the problems that the existing method has poor reconstruction effect under a high-power super-resolution task and is difficult to complete a multi-scale super-resolution task at one time.
The technical solution for realizing the purpose of the invention is as follows: a single image super-resolution reconstruction method based on a hierarchical progressive network is characterized by comprising the following steps:
the method comprises the following steps of (I) a feature extraction branch, wherein the feature extraction branch comprises a feature extraction convolutional layer, a nonlinear mapping module and an upsampling layer, the size of a convolutional kernel is 3 multiplied by 3, the number of output feature maps is 160, the convolution step length is 1, and the upsampling layer is obtained by transposing the convolutional layer: the convolution kernel size is 3 x 3, and the upsampling factor is set to 2;
(II) an image reconstruction branch, the image reconstruction branch comprising a local residual structure, N recursive block structures and a residual prediction convolutional layer, the local residual structure being represented by the following formula:
the input-to-output mapping relationship can be expressed by equation (1):
y＝F(x，{W i })+x
where x is the input, y is the desired output, F (x, { W) i }) represents potential mappings that need to be learned;
the recursive block structure comprises dense connection blocks and transition layers, wherein each layer in the dense connection blocks obtains input from all layers in front of the dense connection blocks, and simultaneously transmits output of the dense connection blocks to all subsequent layers, the size of a convolution kernel of each transition layer is 1 multiplied by 1, the number of output characteristic graphs is 160, and the convolution step length is 1;
the convolution kernel size of the residual prediction convolution layer is 3 multiplied by 3, the number of output characteristic graphs is 160, and the convolution step length is 1;
(III) hierarchical progressive network structure;
(iv) a loss function represented by:
wherein, I r For residual images obtained by mapping F, I SR For super-resolution reconstruction of images, θ is a parameter in the network, N is the number of pictures of a training batch, U (I) L ) Is an interpolated upsampling operation on the low resolution image, I H -U(I L ) The residual error of the true value is obtained, and rho is a Charbonier function and is defined as
Epsilon is taken as an empirical value of 10 -3 。
Furthermore, the network structure of the single layer in the dense connection block comprises a convolution layer I, a batch normalization layer, a nonlinear activation function and a convolution layer II,
the convolution kernel size of the convolution layer I is 1 multiplied by 1, the convolution step length is 1, and the padding is 1; the batch normalization layer is represented by the following formula:
wherein E (-) and var (-) respectively represent an absolute value taking operation and a variance taking operation;
the nonlinear activation function adopts a ReLU activation function and is represented by the following formula:
f(z)＝max(0，z)；
the convolution kernel size of convolution layer II is 1 × 1, the convolution step size is 1, and padding is 1.
Further, the up-sampling layer performs interpolation up-sampling on the input low-resolution image using a bicubic interpolation method.
Furthermore, the super-resolution tasks of different multiples in the hierarchical progressive network structure reuse the same feature extraction branch structure.
Furthermore, the hierarchical progressive network structure has the input of the feature extraction branch being the output of the up-sampling feature map of the previous level under the 4 x and 8 x super-resolution tasks.
Has the beneficial effects that: the invention provides a hierarchical progressive network structure for single image super-resolution reconstruction, which decomposes a high-power super-resolution task into a plurality of subtasks, wherein each subtask can be independently completed by one super-resolution unit network, and the plurality of super-resolution unit networks are cascaded to form the whole network. The invention adopts structures with skip connection property, such as local residual error, dense connection and the like, so as to improve the information flow transmission efficiency and avoid gradient disappearance.
Drawings
FIG. 1 is a diagram of a hierarchical progressive network architecture for single image super resolution according to the present invention;
FIG. 2 is a detailed structure diagram of 2 times image super-resolution task in the network structure of the present invention;
FIG. 3 is a diagram of recursive blocks in the network architecture of the present invention;
FIG. 4 is a single-layer structure diagram of the dense connection structure of the present invention.
Detailed Description
The invention is further illustrated by the following examples in conjunction with the accompanying drawings.
Fig. 1 is a diagram illustrating a hierarchical progressive network structure for single image super resolution according to the present invention. The hierarchical progressive neural network provided by the invention can perform super-resolution processing with the reconstruction multiple of s on a single picture (wherein s =2 or 4 or 8). The network consists of a group of cascaded up-sampling units, and each unit is used for performing 2-time super-resolution processing on an image. The sampling unit structure on each level mainly comprises a characteristic extraction branch and an image reconstruction branch.
Wherein the feature extraction branch in the sampling unit structure at each level is shown in fig. 2. The device comprises a feature extraction convolution layer, a nonlinear mapping module and an upper sampling layer.
The characteristic extraction layer specifically comprises: the feature extraction layer of the 2-time super-resolution task is a convolution layer, the size of a convolution kernel is 3 multiplied by 3, the number of output feature maps is 160, and the convolution step length is 1; and the 4-time and 8-time super-resolution task feature extraction layers are output of the super-resolution unit at the upper stage.
The nonlinear mapping module comprises a local residual error structure, N recursive block structures and a residual error prediction convolutional layer.
The local residual structure specifically comprises: with x as input and y as desired output, the input to output mapping can be expressed by equation (1):
y＝F(x，{W i })+x (1)
wherein F (x, { W) i }) represent complex potential mappings that need to be learned.
The recursive block structure is shown in fig. 3, and includes a dense connection block and a transition layer, specifically:
dense-connected blocks, where each layer in the structure takes input from all layers before it, and passes its output to all subsequent layers, i.e. the input of each layer is the superposition of the information streams output by all the previous layers. The network structure of the single layer is shown in fig. 4:
(1) A convolutional layer I: the size of the convolution kernel is 1 multiplied by 1, the convolution step is 1, and the padding is 1;
(2) The Batch Normalization (BN) layer can be specifically represented by formula (2):
wherein E (-) and var (-) respectively represent an absolute value operation and a variance operation;
(3) Nonlinear activation function: a ReLU activation function is adopted, and the specific function is as follows (3):
f(z)＝max(0，z) (3)
(4) And (3) convolutional layer II: the convolution kernel size is 1 × 1, the convolution step size is 1, padding is 1
The transition layer is specifically: the convolution kernel size is 1 × 1, the number of output feature maps is 160, and the convolution step size is 1.
And the residual prediction convolutional layer is used for integrating the characteristic diagrams extracted by the dense connection structure into a residual diagram so as to be convenient for subsequent superposition with input data. Specifically, the convolution layer has a convolution kernel size of 3 × 3, the number of output feature maps of 160, and a convolution step size of 1.
The upsampling layer uses a transposed convolutional layer: the convolution kernel size is 3 x 3 and the upsampling factor is set to 2.
The image reconstruction branch performs interpolation up-sampling on the input low-resolution image by using a bicubic interpolation method. And aiming at reconstruction tasks of different multiples, corresponding up-sampling proportions are adopted. Images of different super-resolution tasks obtain input images from original low-resolution images, and interpolation images with the same size as the prediction residual images are obtained through 2 x, 4 x and 8 x bicubic interpolation respectively. And finally, performing pixel-by-pixel addition on the images of the two branches to obtain a final super-resolution reconstructed image.
The purpose of super-resolution of the image is to find the mapping function F, so that the low-resolution image I L Super obtained after mappingResolution image F (I) L ) And the original high resolution image I H As consistent as possible. Defining the residual image obtained by mapping F as I r The super-resolution reconstructed image is I SR The parameter in the network is θ. The loss function used in the present invention can be represented by equation (4):
where N is the number of pictures of a training batch, U (I) L ) Is an interpolated upsampling operation on the low resolution image, I H -U(I L ) The result is a residual error of the true value. ρ is a Charbonier function, defined as
E is to take an empirical value of 10 -3 。
Compared with the traditional image super-resolution depth neural network algorithm, the hierarchical progressive structure provided by the invention can generate a multi-scale predicted image in one-time forward propagation. And meanwhile, information flow is integrated by using local residual errors and dense connection, so that the problem of gradient disappearance is avoided, and a deeper network is obtained. The present embodiment uses peak signal-to-noise ratio (PSNR) and Structural Similarity (SSIM) to evaluate the image super-resolution method of the present invention. The higher PSNR and SSIM values are generally considered to be the better the quality of the image. The PSNR value of the obtained image super-resolution reconstruction result on each test set is 0.25 dB-2.35dB higher, and the SSIM value is 0.003-0.053 higher when the set5, the set14 and the BSD100 data sets are used for testing the invention.
The invention provides a hierarchical progressive network structure for single-image super-resolution reconstruction. The structure decomposes a high-power super-resolution task into a plurality of subtasks, each subtask can be independently completed by one super-resolution unit network, and a plurality of super-resolution unit networks are cascaded to form the whole network. In the network training stage, 2 x, 4 x and 8 x upsampling factors can be trained simultaneously, and in the testing stage, the same training model can be used for performing super-resolution reconstruction processing on images by three times. Meanwhile, the invention adopts structures with skip connection property, such as local residual error, dense connection and the like, so as to improve the information flow transmission efficiency and avoid gradient disappearance.
Claims (3)
1. A single image super-resolution reconstruction method based on a hierarchical progressive network is characterized by comprising the following steps:
the method comprises the following steps of (A) extracting features, wherein the extracting features comprise a feature extraction convolutional layer, a nonlinear mapping module and an upsampling layer, the size of a convolutional kernel is 3 multiplied by 3, the number of output feature graphs is 160, the convolution step size is 1, and the upsampling layer is obtained by transposing the convolutional layer: the convolution kernel size is 3 x 3, and the upsampling factor is set to 2;
(II) an image reconstruction branch, the image reconstruction branch comprising a local residual structure, N recursive block structures and a residual prediction convolutional layer, the local residual structure being represented by the following formula:
the input-to-output mapping is represented by:
y＝F(x，{W i })+x
where x is the input, y is the desired output, F (x, { W) i }) represents potential mappings that need to be learned;
the recursive block structure comprises a dense connection block and transition layers, wherein each layer in the dense connection block obtains input from all the layers in front of the dense connection block and simultaneously transmits output of the dense connection block to all the subsequent layers, the convolution kernel size of the transition layers is 1 multiplied by 1, the number of output characteristic graphs is 160, and the convolution step length is 1;
the convolution kernel size of the residual prediction convolution layer is 3 multiplied by 3, the number of output characteristic graphs is 160, and the convolution step length is 1;
(III) hierarchical progressive network structure;
(iv) a loss function represented by:
wherein, I r For residual images obtained by mapping F, I SR For super-resolution reconstruction of images, θ is a parameter in the network, N is the number of pictures of a training batch, U (I) L ) Is an interpolated upsampling operation on the low resolution image, I H -U(I L ) The residual error of the true value is obtained, and rho is a Charbonier function and is defined as
E is to take an empirical value of 10 -3 ；
The super-resolution tasks of different multiples in the hierarchical progressive network structure reuse the same feature extraction branch structure;
under the 4 x and 8 x super-resolution tasks of the hierarchical progressive network structure, the input of the feature extraction branch is the output of the up-sampling feature map of the previous level.
2. The single image super-resolution reconstruction method based on hierarchical progressive network as claimed in claim 1, wherein the network structure of single layer in the dense connection block comprises convolutional layer I, batch normalization layer, nonlinear activation function and convolutional layer II,
the convolution kernel size of the convolution layer I is 1 multiplied by 1, the convolution step length is 1, and the padding is 1; the batch normalization layer is represented by the following formula:
wherein E (-) and var (-) respectively represent an absolute value taking operation and a variance taking operation;
the nonlinear activation function adopts a ReLU activation function and is represented by the following formula:
f(z)＝max(0，z)；
the convolution kernel size of convolution layer II is 1 × 1, the convolution step size is 1, and padding is 1.
3. The single-image super-resolution reconstruction method based on hierarchical progressive network according to claim 1, wherein the upsampling layer interpolates the input low-resolution image by using a bicubic interpolation method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910146330.4A CN110047038B (en) | 2019-02-27 | 2019-02-27 | Single-image super-resolution reconstruction method based on hierarchical progressive network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910146330.4A CN110047038B (en) | 2019-02-27 | 2019-02-27 | Single-image super-resolution reconstruction method based on hierarchical progressive network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110047038A CN110047038A (en) | 2019-07-23 |
| CN110047038B true CN110047038B (en) | 2022-11-04 |
Family
ID=67274284
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910146330.4A Active CN110047038B (en) | 2019-02-27 | 2019-02-27 | Single-image super-resolution reconstruction method based on hierarchical progressive network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110047038B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111260552B (en) * | 2020-01-09 | 2023-05-30 | 复旦大学 | Progressive learning-based image super-resolution method |
| CN113538307B (en) * | 2021-06-21 | 2023-06-20 | 陕西师范大学 | Synthetic aperture imaging method based on multi-view super-resolution depth network |
| CN113610706A (en) * | 2021-07-19 | 2021-11-05 | 河南大学 | Fuzzy monitoring image super-resolution reconstruction method based on convolutional neural network |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107657586A (en) * | 2017-10-13 | 2018-02-02 | 深圳市唯特视科技有限公司 | A kind of single photo super-resolution Enhancement Method based on depth residual error network |
| CN109118432A (en) * | 2018-09-26 | 2019-01-01 | 福建帝视信息科技有限公司 | A kind of image super-resolution rebuilding method based on Rapid Circulation convolutional network |
-
2019
- 2019-02-27 CN CN201910146330.4A patent/CN110047038B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107657586A (en) * | 2017-10-13 | 2018-02-02 | 深圳市唯特视科技有限公司 | A kind of single photo super-resolution Enhancement Method based on depth residual error network |
| CN109118432A (en) * | 2018-09-26 | 2019-01-01 | 福建帝视信息科技有限公司 | A kind of image super-resolution rebuilding method based on Rapid Circulation convolutional network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110047038A (en) | 2019-07-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109886871B (en) | Image super-resolution method based on channel attention mechanism and multi-layer feature fusion | |
| CN109903228B (en) | Image super-resolution reconstruction method based on convolutional neural network | |
| CN106910161B (en) | Single image super-resolution reconstruction method based on deep convolutional neural network | |
| CN110163801B (en) | Image super-resolution and coloring method, system and electronic equipment | |
| CN111861884B (en) | Satellite cloud image super-resolution reconstruction method based on deep learning | |
| CN110047038B (en) | Single-image super-resolution reconstruction method based on hierarchical progressive network | |
| CN110675321A (en) | Super-resolution image reconstruction method based on progressive depth residual error network | |
| CN110288524B (en) | Deep learning super-resolution method based on enhanced upsampling and discrimination fusion mechanism | |
| Chen et al. | Single image super-resolution using deep CNN with dense skip connections and inception-resnet | |
| CN113096017A (en) | Image super-resolution reconstruction method based on depth coordinate attention network model | |
| CN111932461A (en) | Convolutional neural network-based self-learning image super-resolution reconstruction method and system | |
| CN111986092B (en) | Dual-network-based image super-resolution reconstruction method and system | |
| CN111784582A (en) | DEC-SE-based low-illumination image super-resolution reconstruction method | |
| CN111861886B (en) | Image super-resolution reconstruction method based on multi-scale feedback network | |
| CN112001843A (en) | Infrared image super-resolution reconstruction method based on deep learning | |
| CN114926336A (en) | Video super-resolution reconstruction method and device, computer equipment and storage medium | |
| CN115496663A (en) | Video super-resolution reconstruction method based on D3D convolution intra-group fusion network | |
| CN113379606B (en) | Face super-resolution method based on pre-training generation model | |
| CN115713462A (en) | Super-resolution model training method, image recognition method, device and equipment | |
| CN117196959B (en) | Self-attention-based infrared image super-resolution method, device and readable medium | |
| CN112150356A (en) | Single compressed image super-resolution reconstruction method based on cascade framework | |
| CN106981046B (en) | Single image super resolution ratio reconstruction method based on multi-gradient constrained regression | |
| CN117455770A (en) | Lightweight image super-resolution method based on layer-by-layer context information aggregation network | |
| CN116797541A (en) | Transformer-based lung CT image super-resolution reconstruction method | |
| CN116862765A (en) | Medical image super-resolution reconstruction method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |