CN111353938A - Image super-resolution learning method based on network feedback - Google Patents
Image super-resolution learning method based on network feedback Download PDFInfo
- Publication number
- CN111353938A CN111353938A CN202010132826.9A CN202010132826A CN111353938A CN 111353938 A CN111353938 A CN 111353938A CN 202010132826 A CN202010132826 A CN 202010132826A CN 111353938 A CN111353938 A CN 111353938A
- Authority
- CN
- China
- Prior art keywords
- image
- resolution
- convolution
- feedback
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 25
- 238000005070 sampling Methods 0.000 claims abstract description 12
- 238000012545 processing Methods 0.000 claims abstract description 7
- 238000010586 diagram Methods 0.000 claims description 6
- 230000000694 effects Effects 0.000 claims description 4
- 238000007670 refining Methods 0.000 claims description 3
- 238000012512 characterization method Methods 0.000 claims description 2
- 238000013135 deep learning Methods 0.000 abstract description 9
- 230000008713 feedback mechanism Effects 0.000 abstract description 6
- 230000000007 visual effect Effects 0.000 abstract description 4
- 238000001514 detection method Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000003042 antagnostic effect Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000002059 diagnostic imaging Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Processing (AREA)
Abstract
The invention provides an image super-resolution learning method based on network feedback. The method comprises the steps of firstly processing a low-resolution image by using a convolution network to obtain a shallow feature, then connecting the shallow feature of the obtained low-resolution image and a high-level feature output by a feedback network at a previous moment through feedback to be used as the input of the feedback network, and correcting the high-level feature as one of low-level features through the feedback network to obtain a higher-level feature. And finally, deconvoluting the output of the feedback network, then performing convolution to obtain a residual image, and adding the image obtained by up-sampling the original image through bilinear interpolation to the residual image to obtain a super-resolution image. The invention solves the problems that the existing deep learning method does not utilize a feedback mechanism commonly existing in a human visual system, a plurality of high-resolution images correspond to the same low-resolution image, and the details of the image cannot be repaired due to the increase of a super-resolution scaling factor.
Description
Technical Field
The invention relates to the technical field of image and video processing, in particular to an image super-resolution learning method based on network feedback.
Background
The super resolution reconstruction (SR) is an important image processing technology in the fields of computer vision and image processing, and because the image super resolution method can correct the damage of the image caused by equipment or environment to a certain extent, the method has important application values in many fields, such as target detection, medical imaging, security monitoring, satellite remote sensing, and the like.
In recent years, with the rapid development of deep learning, the deep learning is applied to various artificial intelligence tasks such as image classification and target detection, and has made breakthrough progress, researchers have also actively explored the use of deep learning to solve the super-resolution problem, and a more effective method, namely, a super-resolution method based on deep learning, is widely used to solve the super-resolution problem of images. By training an end-to-end network model, the mapping relation between the low-resolution image and the high-resolution image is directly learned.
With the various super-resolution methods based on deep learning, the method from the early convolutional neural network to the later antagonistic network generation shows good performance. However, the feedback mechanism ubiquitous in the human visual system has not been fully exploited in existing deep learning. The super-resolution problem is very challenging and not applicable since there are always multiple high-resolution images corresponding to the same low-resolution image, and furthermore, as the super-resolution scaling factor increases, the recovery of the missing details of the image becomes more complicated. Therefore, the information feedback is realized by using a hidden state in an RNN in a network feedback-based mode, high-level information is obtained through the feedback, and the final high-resolution image is generated iteratively.
Disclosure of Invention
Aiming at the problems that the existing deep learning method does not utilize a feedback mechanism commonly existing in a human visual system, a plurality of high-resolution images correspond to the same low-resolution image, and the details of the image cannot be repaired due to the increase of a super-resolution scaling factor, the invention provides an image super-resolution learning method based on network feedback.
An image super-resolution learning method based on network feedback comprises the following steps:
and (1) processing the low-resolution image by using a convolution network to obtain shallow features.
Image of low resolutionInputting the data into a convolution network with 4 × m convolution kernels with the size of 3 × 3 and m convolution kernels with the size of 1 × 1 to obtain shallow layer characteristic outputAs input to the feedback network. Low resolution image at next momentByIs obtained by down-sampling due toAnd (3) for the super-resolution image of the initial reconstruction, taking the image obtained by sampling the reconstructed image as the input of the iteration at the next moment, and forming feedback to improve the later reconstruction effect.
And (2) taking the shallow feature of the obtained low-resolution image and the high-level feature output by the feedback network at the previous moment as the input of the feedback network through a feedback connection, and correcting the high-level feature as one of the low-level features through the feedback network to obtain a higher-level feature. The method comprises the following specific steps:
(2.1) shallow characterizationAnd advanced featuresAs a feedback network input, m convolutional layers with convolutional kernel size of 1 × 1 are usedAndconcatenating and compressing, feeding back informationRefining input features produces refined input featuresWhereinIs composed ofWhere the size of the convolution kernel k depends on the scaling factor.
(2.2) input featuresGenerating 1 high resolution feature by upsampling 1 deconvolution layer with m convolution kernels of size k × k and number m of convolution kernelsCharacterizing high resolutionConvolution with 1 convolution kernel size of k × kThe convolution layer with m cores is subjected to up-sampling to obtain refined high-grade characteristics
(2.3) the refined high-level features are obtainedAs input features, 1 convolution layer with convolution kernel size of 1 × 1 and convolution kernel number of m is added before convolution and deconvolution in step (2.2), the same operation as in step (2.2) is carried out, G times of repetition is carried out to obtain high-level feature vector group generated by low-level featuresAnd corresponding refined feature vector groups
(2.4) in order that the feedback input at the next moment contains better characteristics, fusing the refined characteristics through 1-layer convolution layers with the convolution kernel size of 1 × 1 and the convolution kernel number of m to generate the output of the feedback network
And (3) deconvoluting the output of the feedback network, then obtaining a residual image by convolution, and adding the image obtained by up-sampling the original image through bilinear interpolation to the residual image to obtain a super-resolution image. The method comprises the following specific steps:
(3.1) feeding back the output of the networkAnd generating a high-resolution feature map by using the deconvolution layer with the convolution kernel size of k in m layers. Drawing the originalObtaining a high resolution map by bilinear interpolation
(3.2) obtaining a residual image by passing the high-resolution feature map through a convolution layer with convolution kernel size of 3 × 3 of n layersFinally, the large resolution map is processedAnd residual imageAdding to obtain super-resolution imageWhen the image is a gray scale image, n is 1, and when the image is a color image, n is 3.
The invention has the following beneficial effects:
the invention solves the problems that the existing deep learning method does not utilize a feedback mechanism commonly existing in a human visual system, a plurality of high-resolution images correspond to the same low-resolution image, and the details of the image cannot be repaired due to the increase of a super-resolution scaling factor.
Drawings
FIG. 1 is a flow chart of the method of the present invention;
FIG. 2 is a detailed view of a model of the method of the present invention;
FIG. 3 is a block diagram of the feedback network of the present invention;
fig. 4 is a supplementary explanatory diagram of the distribution-instruction feedback mechanism.
Detailed Description
The objects and effects of the present invention will become more apparent from the following detailed description of the present invention with reference to the accompanying drawings.
Fig. 1 is a flowchart of a method for learning super-resolution images based on network feedback according to the present invention, fig. 2 is a structural diagram of a method model, fig. 3 is a structural diagram of a feedback network, and fig. 4 is a supplementary explanatory diagram of a feedback mechanism.
The specific steps are as follows as shown in figure 1:
and (1) processing the low-resolution image by using a convolution network to obtain shallow layer characteristics.
Step (1.1) Low resolution image extraction according to the shallow feature extraction Module operation of FIG. 1Inputting the data into a convolution network with 4 × m convolution kernels with the size of 3 × 3 and m convolution kernels with the size of 1 × 1 to obtain shallow layer characteristic outputAs input to the feedback network.
Step (1.2) Low resolution image at the next momentSuper-resolution image for initial reconstructionAnd the down sampling is obtained, and the reconstruction result is taken as a feedback to be introduced into the input, so that the subsequent reconstruction effect is improved. Specifically, the operation (a) and the down-sampling operation in the left part of the model fig. 2 are described.
And (2) inputting the shallow feature of the obtained low-resolution image and the high-level feature output by the feedback network at the previous moment into the feedback network to generate the refined high-level feature.
Step (2.1) shallow featureAnd advanced featuresAs a feedback network input, m convolutional layers with convolutional kernel size of 1 × 1 are usedAndconcatenating and compressing, feeding back informationRefining input features produces refined input featuresWhereinIs composed ofAs shown in part a of fig. 3.
Step (2.2) input characteristics after thinning1 high-resolution feature is generated by up-sampling 1 deconvolution layer with convolution kernel size k × k and convolution kernel number mByUpsampling 1 convolution layer with convolution kernel size of k × k and convolution kernel number of m to obtain refined high-level featuresAs shown in part b of fig. 3.
Step (2.3) is to obtain the refined advanced featuresAdding 1 convolution layer with convolution kernel size of 1 × 1 and convolution kernel number of m before convolution and deconvolution in step (2.2) as input features, performing the same operation in step (2.2) for G times to obtain high-level feature vector groupAnd corresponding refined high-level feature vector groupAs shown in part c of fig. 3
And (2.4) finally fusing the refined characteristics through 1 layer of convolutional layers with the convolutional kernel size of 1 × 1 and the convolutional kernels with the number of m to generate the output of the feedback networkAs shown in part d of fig. 3
And (3) deconvoluting the output of the feedback network, then obtaining a residual image by convolution, and adding the image obtained by up-sampling the original image through bilinear interpolation to the residual image to obtain a super-resolution image.
Step (3.1) outputting the output obtained by the feedback module output moduleAnd generating a high-resolution feature map by using the deconvolution layer with the convolution kernel size of k in m layers. Drawing the originalObtaining a high resolution map by bilinear interpolation
Step (3.2) of obtaining a residual image by passing the high-resolution feature map through a convolution layer with the convolution kernel size of 3 × 3 (n layers)Finally, the resolution map is largeAnd residual imageAdding to obtain super-resolution imageWhen the image is a gray scale image, n is 1, and when the image is a color image, n is 3.
Claims (1)
1. An image super-resolution learning method based on network feedback is characterized by comprising the following steps:
processing a low-resolution image by using a convolution network to obtain shallow layer characteristics;
image of low resolutionInputting the data into a convolution network with 4 × m convolution kernels with the size of 3 × 3 and m convolution kernels with the size of 1 × 1 to obtain shallow layer characteristic outputAs an input to a feedback network; low resolution image at next momentByIs obtained by down-sampling due toFor the super-resolution image reconstructed at the initial stage, taking the image obtained by sampling the reconstructed image as the input of iteration at the next moment to form feedback and improve the later reconstruction effect;
step (2), the shallow feature of the obtained low-resolution image and the high-level feature output by the feedback network at the previous moment are used as the input of the feedback network through feedback connection, and the high-level feature is used as one kind of low-level feature to be corrected through the feedback network to obtain a higher-level feature; the method comprises the following specific steps:
(2.1) shallow characterizationAnd advanced featuresAs a feedback network input, m convolutional layers with convolutional kernel size of 1 × 1 are usedAndconcatenating and compressing, feeding back informationRefining input features produces refined input featuresWhereinIs composed ofWherein the size of the convolution kernel k depends on the scaling factor;
(2.2) input featuresGenerating 1 high resolution feature by upsampling 1 deconvolution layer with m convolution kernels of size k × k and number m of convolution kernelsCharacterizing high resolutionUpsampling by 1 convolutional layer with convolutional kernel size of k × k and convolutional kernel number of mTo obtain refined high-grade features
(2.3) the refined high-level features are obtainedAs input features, 1 convolution layer with convolution kernel size of 1 × 1 and convolution kernel number of m is added before convolution and deconvolution in step (2.2), the same operation as in step (2.2) is carried out, G times of repetition is carried out to obtain high-level feature vector group generated by low-level featuresAnd corresponding refined feature vector groups
(2.4) in order that the feedback input at the next moment contains better characteristics, fusing the refined characteristics through 1-layer convolution layers with the convolution kernel size of 1 × 1 and the convolution kernel number of m to generate the output of the feedback network
Step (3), deconvoluting the output of the feedback network, then obtaining a residual image by convolution, and adding an image obtained by up-sampling the original image through bilinear interpolation to the residual image to obtain a super-resolution image; the method comprises the following specific steps:
(3.1) feeding back the output of the networkGenerating a high-resolution characteristic diagram through m deconvolution layers with convolution kernel size of k; drawing the originalBy bilinear interpolation to obtain oneA large resolution map
(3.2) obtaining a residual image by passing the high-resolution feature map through a convolution layer with convolution kernel size of 3 × 3 of n layersFinally, the large resolution map is processedAnd residual imageAdding to obtain super-resolution imageWhen the image is a gray scale image, n is 1, and when the image is a color image, n is 3.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010132826.9A CN111353938A (en) | 2020-02-29 | 2020-02-29 | Image super-resolution learning method based on network feedback |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010132826.9A CN111353938A (en) | 2020-02-29 | 2020-02-29 | Image super-resolution learning method based on network feedback |
Publications (1)
Publication Number | Publication Date |
---|---|
CN111353938A true CN111353938A (en) | 2020-06-30 |
Family
ID=71197315
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202010132826.9A Pending CN111353938A (en) | 2020-02-29 | 2020-02-29 | Image super-resolution learning method based on network feedback |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN111353938A (en) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111986085A (en) * | 2020-07-31 | 2020-11-24 | 南京航空航天大学 | Image super-resolution method based on depth feedback attention network system |
CN112084908A (en) * | 2020-08-28 | 2020-12-15 | 广州汽车集团股份有限公司 | Image processing method and system and storage medium |
CN112508794A (en) * | 2021-02-03 | 2021-03-16 | 中南大学 | Medical image super-resolution reconstruction method and system |
CN113781304A (en) * | 2021-09-08 | 2021-12-10 | 福州大学 | Lightweight network model based on single image super-resolution and processing method |
WO2023010831A1 (en) * | 2021-08-03 | 2023-02-09 | 长沙理工大学 | Method, system and apparatus for improving image resolution, and storage medium |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180293706A1 (en) * | 2017-04-05 | 2018-10-11 | Here Global B.V. | Deep convolutional image up-sampling |
CN109741260A (en) * | 2018-12-29 | 2019-05-10 | 天津大学 | A kind of efficient super-resolution method based on depth back projection network |
CN110276721A (en) * | 2019-04-28 | 2019-09-24 | 天津大学 | Image super-resolution rebuilding method based on cascade residual error convolutional neural networks |
CN110322400A (en) * | 2018-03-30 | 2019-10-11 | 京东方科技集团股份有限公司 | Image processing method and device, image processing system and its training method |
KR20190131205A (en) * | 2018-05-16 | 2019-11-26 | 한국과학기술원 | Super-resolution network processing method and system |
-
2020
- 2020-02-29 CN CN202010132826.9A patent/CN111353938A/en active Pending
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180293706A1 (en) * | 2017-04-05 | 2018-10-11 | Here Global B.V. | Deep convolutional image up-sampling |
CN110322400A (en) * | 2018-03-30 | 2019-10-11 | 京东方科技集团股份有限公司 | Image processing method and device, image processing system and its training method |
KR20190131205A (en) * | 2018-05-16 | 2019-11-26 | 한국과학기술원 | Super-resolution network processing method and system |
CN109741260A (en) * | 2018-12-29 | 2019-05-10 | 天津大学 | A kind of efficient super-resolution method based on depth back projection network |
CN110276721A (en) * | 2019-04-28 | 2019-09-24 | 天津大学 | Image super-resolution rebuilding method based on cascade residual error convolutional neural networks |
Non-Patent Citations (1)
Title |
---|
ZHEN LI, JINGLEI YANG, ZHENG LIU, XIAOMIN YANG, GWANGGIL JEON, WEI WU: "Feedback Network for Image Super-Resolution", 2019 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), pages 1 - 5 * |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111986085A (en) * | 2020-07-31 | 2020-11-24 | 南京航空航天大学 | Image super-resolution method based on depth feedback attention network system |
CN112084908A (en) * | 2020-08-28 | 2020-12-15 | 广州汽车集团股份有限公司 | Image processing method and system and storage medium |
CN112508794A (en) * | 2021-02-03 | 2021-03-16 | 中南大学 | Medical image super-resolution reconstruction method and system |
WO2023010831A1 (en) * | 2021-08-03 | 2023-02-09 | 长沙理工大学 | Method, system and apparatus for improving image resolution, and storage medium |
CN113781304A (en) * | 2021-09-08 | 2021-12-10 | 福州大学 | Lightweight network model based on single image super-resolution and processing method |
CN113781304B (en) * | 2021-09-08 | 2023-10-13 | 福州大学 | Lightweight network model based on single image super-resolution and processing method |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN111353938A (en) | Image super-resolution learning method based on network feedback | |
CN109903228B (en) | Image super-resolution reconstruction method based on convolutional neural network | |
CN111243066A (en) | Facial expression migration method based on self-supervision learning and confrontation generation mechanism | |
CN109544448B (en) | Group network super-resolution image reconstruction method of Laplacian pyramid structure | |
CN110136062B (en) | Super-resolution reconstruction method combining semantic segmentation | |
CN113096017B (en) | Image super-resolution reconstruction method based on depth coordinate attention network model | |
CN109685716B (en) | Image super-resolution reconstruction method for generating countermeasure network based on Gaussian coding feedback | |
CN112215755B (en) | Image super-resolution reconstruction method based on back projection attention network | |
CN111784582B (en) | DEC-SE-based low-illumination image super-resolution reconstruction method | |
CN111861886B (en) | Image super-resolution reconstruction method based on multi-scale feedback network | |
CN111353940A (en) | Image super-resolution reconstruction method based on deep learning iterative up-down sampling | |
CN111696038A (en) | Image super-resolution method, device, equipment and computer-readable storage medium | |
Guan et al. | Srdgan: learning the noise prior for super resolution with dual generative adversarial networks | |
CN112529776A (en) | Training method of image processing model, image processing method and device | |
CN113592715A (en) | Super-resolution image reconstruction method for small sample image set | |
CN115713462A (en) | Super-resolution model training method, image recognition method, device and equipment | |
Gao et al. | Bayesian image super-resolution with deep modeling of image statistics | |
CN109993701B (en) | Depth map super-resolution reconstruction method based on pyramid structure | |
CN113379606B (en) | Face super-resolution method based on pre-training generation model | |
CN114332625A (en) | Remote sensing image colorizing and super-resolution method and system based on neural network | |
CN112184552A (en) | Sub-pixel convolution image super-resolution method based on high-frequency feature learning | |
CN108765297B (en) | Super-resolution reconstruction method based on cyclic training | |
CN116797456A (en) | Image super-resolution reconstruction method, system, device and storage medium | |
CN116228576A (en) | Image defogging method based on attention mechanism and feature enhancement | |
CN115205527A (en) | Remote sensing image bidirectional semantic segmentation method based on domain adaptation and super-resolution |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |