CN115131210B - Alternating optimized image blind super-resolution reconstruction method based on accurate kernel estimation - Google Patents
Alternating optimized image blind super-resolution reconstruction method based on accurate kernel estimation Download PDFInfo
- Publication number
- CN115131210B CN115131210B CN202210744814.0A CN202210744814A CN115131210B CN 115131210 B CN115131210 B CN 115131210B CN 202210744814 A CN202210744814 A CN 202210744814A CN 115131210 B CN115131210 B CN 115131210B
- Authority
- CN
- China
- Prior art keywords
- resolution image
- degradation
- low
- convolution
- resolution
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 55
- 230000015556 catabolic process Effects 0.000 claims abstract description 73
- 238000006731 degradation reaction Methods 0.000 claims abstract description 73
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims description 38
- 230000008569 process Effects 0.000 claims description 20
- 230000008447 perception Effects 0.000 claims description 14
- 238000012549 training Methods 0.000 claims description 12
- 230000004927 fusion Effects 0.000 claims description 9
- 238000012937 correction Methods 0.000 claims description 5
- 238000013507 mapping Methods 0.000 claims description 5
- 238000011176 pooling Methods 0.000 claims description 5
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 claims description 3
- 239000000284 extract Substances 0.000 claims description 3
- 230000000452 restraining effect Effects 0.000 claims 1
- 238000010586 diagram Methods 0.000 description 12
- 238000012545 processing Methods 0.000 description 10
- 238000004590 computer program Methods 0.000 description 7
- 230000000694 effects Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 230000008901 benefit Effects 0.000 description 3
- 238000005457 optimization Methods 0.000 description 3
- 238000003860 storage Methods 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 101100365548 Caenorhabditis elegans set-14 gene Proteins 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4053—Super resolution, i.e. output image resolution higher than sensor resolution
-
- G06T3/02—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Abstract
The invention provides an alternate optimized image blind super-resolution reconstruction method based on accurate kernel estimation. By using the method, the image with higher quality can be generated under the condition that the degradation mode of the image is unknown.
Description
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to an alternating optimized image blind super-resolution reconstruction method based on accurate kernel estimation.
Background
The super-resolution reconstruction of an image is a process of estimating an image from a Low resolution image (LR) to obtain a High resolution image (HR), and is a classical uncertainty problem in computer vision, and in recent researches, a convolutional neural network has achieved a remarkable effect on the super-resolution reconstruction of an image [1-4] . But these methods use what is predefined as bicubic interpolation for the downsampling of the image. Although in this way the network has achieved very good results, in real scenes the downsampling of low resolution images is much more complex than this. With the ideal hypothesis training network, the robustness is poor, the effect on the low-resolution image of other downsampling modes is severely reduced, and the requirements of real scenes are hardly met. It is therefore becoming a growing concern to have a trained network meet the need to estimate high resolution images in a variety of degradation modes. In order for the network to better estimate high resolution images in multiple degradation modes, SRMD [5] Stretching the degradation information to the same dimension as the picture is proposed to splice for use to the degradation information guiding network to reconstruct the image, but this approach has the disadvantage that it relies on additional input of the degradation information and therefore only when there is an accurate kernel estimation of the low resolution image can the effect of guaranteeing the reconstructed high resolution image. And if the wrong blur kernel is used to guide the network to reconstruct the image, it may lead to a severe performance slip. It is therefore necessary to take into account how to estimate more accurate blur kernels to direct the reconstructed image.
However, the estimation of the blur kernel is also an uncomfortable problem, and the information of the blur kernel is estimated accurately only by the low resolution image without the high resolution image, and the difference in the information of the blur kernel also causes the non-ideal performance of the network. To alleviate this problem, IKC [6] The method comprises the steps of providing that two sub-networks of fuzzy core estimation and reconstructed image work cooperatively through the thought of iterative correction, integrating the two sub-problems into a unified framework, firstly simulating degradation information through an estimation module of a fuzzy core, fusing the degradation information into a corrector, carrying out iterative correction on the solved fuzzy core and the reconstructed image in the corrector, and generating a more accurate fuzzy core through an iterative correction mode, so that a better effect is obtained. However, this approach of IKC is still a two-step split training for networks, multiple sub-networks are prone to falling into a locally optimal solution, and may limit their network performance because the relationship between high resolution images and low resolution is not considered when estimating the blur kernel. Although all are based on the idea of iterative solution, unlike IKC, DAN [7] The method adopts an end-to-end optimization mode for integrating two modules together, so that two sub-networks are mutually cooperated to obtain better performance, and simultaneously, a kernel estimator fully excavates the relation between a high-resolution image and a low-resolution image and fits more accurate fuzzy kernel informationAnd (5) extinguishing. The advantage of the DAN is that the DAN guarantees a close relation with the prior information of its respective module by using the subnetworks of the restorer and estimator modules, but the existing algorithm does not consider the difference between the degradation information and the image characteristics when estimating the blur kernel and estimating the high resolution image, and adopts the same fusion strategy.
Prior art documents to which reference may be made include:
[1]Dong C,Loy C C,He K,et al.Image super-resolution using deep convolutional networks[J].IEEE transactions on pattern analysis and machine intelligence,2015,38(2):295-307.
[2]Kim J,Lee J K,Lee K M.Accurate image super-resolution using very deep convolutional networks[C].Proceedings of the IEEE conference on computer vision and pattern recognition.2016:1646-1654.
[3]Shi W,Caballero J,Huszár F,et al.Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C].Proceedings of the IEEE conference on computer vision and pattern recognition.2016:1874-1883.
[4]Dong C,Loy C C,Tang X.Accelerating the super-resolution convolutional neural network[C].Proceedings of the European conference on computer vision.2016:391-407.
[5]Zhang K,Zuo W,Zhang L.Learning a single convolutional super-resolution network for multiple degradations[C].Proceedings of the IEEE conference on computer vision and pattern recognition.2018:3262-3271.
[6]Gu J,Lu H,Zuo W,et al.Blind super-resolution with iterative kernel correction[C].Proceedings of the IEEE conference on computer vision and pattern recognition.2019:1604-1613.
[7]Luo Z,Huang Y,Li S,et al.End-to-end Alternating Optimization for Blind Super Resolution[J].arXiv preprint arXiv:2105.06878,2021.
[8]Liang J,Sun G,Zhang K,et al.Mutual affine network for spatially variant kernel estimation in blind image super-resolution[C].Proceedings of the IEEE conference on computer vision and pattern recognition.2021:4096-4105.
[9]Levin A,Weiss Y,Durand F,et al.Understanding and evaluating blind deconvolution algorithms[C].Proceedings of the IEEE conference on computer vision and pattern recognition.2009:1964-1971.
[10]Wang L,Wang Y,Dong X,et al.Unsupervised degradation representation learning for blind super-resolution[C].Proceedings of the IEEE conference on computer vision and pattern recognition.2021:10581-10590.
[11]Agustsson E,Timofte R.Ntire 2017 challenge on single image super-resolution:Dataset and study[C].Proceedings of the IEEE conference on computer vision and pattern recognition workshops.2017:126-135.
[12]Lim B,Son S,Kim H,et al.Enhanced deep residual networks for single image super-resolution[C].Proceedings of the IEEE conference on computer vision and pattern recognition workshops.2017:136-144.
[13]Bevilacqua M,Roumy A,Guillemot C,et al.Low-complexity single-image super-resolution based on nonnegative neighbor embedding[C].British Machine Vision Conference,2012.
[14]Zeyde R,Elad M,Protter M.On single image scale-up using sparse-representations[C].International conference on curves and surfaces,2010:711-730.
[15]Huang J B,Singh A,Ahuja N.Single image super-resolution from transformed self-exemplars[C].Proceedings of the IEEE conference on computer vision and pattern recognition.2015:5197-5206.
[16]Arbelaez P,Maire M,Fowlkes C,et al.Contour detection and hierarchical image segmentation[J].IEEE transactions on pattern analysis and machine intelligence,2010,33(5):898-916.
[17]Fujimoto A,Ogawa T,Yamamoto K,et al.Manga109 dataset and creation of metadata[C].Proceedings of the 1st international workshop on comics analysis,processing and understanding.2016:1-5.。
disclosure of Invention
In order to further improve the prior art scheme, the invention aims to provide an alternate optimized image blind super-resolution reconstruction method based on accurate kernel estimation. The method and the system are beneficial to generating the image with higher quality under the condition that the degradation mode of the image is unknown.
The method processes the two sub problems of kernel estimation and non-blind super-resolution reconstruction through two network modules of an estimator and a restorer, wherein the estimator module estimates a fuzzy kernel by utilizing a high-resolution image obtained by the restorer, the restorer module reconstructs an image by utilizing the fuzzy kernel obtained by the estimator, and the two modules repeatedly and alternately optimize to form an end-to-end network. The affine module and the degradation perception module are utilized to enable the two sub-networks to more fully utilize information to each other to complete respective tasks. Experiments prove that the method can effectively promote the network to generate a better blind super-resolution reconstruction effect.
In order to achieve the above purpose, the invention adopts the following technical scheme:
the alternating optimized image blind super-resolution reconstruction method based on the accurate kernel estimation is characterized in that the adopted super-resolution reconstruction network consists of a fuzzy kernel estimation module, an estimator and a restorer; the method comprises the steps of adopting a fuzzy kernel estimation module at the head of a network, then alternately iterating an estimator and a restorer to be used as a reasoning process, inputting a low-resolution image X into the network, and outputting a corresponding high-resolution image by the network
Further, firstly inputting the low-resolution image X into the fuzzy kernel estimation module to estimate fuzzy kernels of the degradation modes corresponding to the low-resolution image; the following two steps are then alternately performed as a process of reasoning: introducing the estimated blur kernel and the low-resolution image X as inputs of a restorer to generate a high-resolution image; and generating a blur kernel using the high resolution image and the low resolution image X as inputs to the estimator to achieve a stepwise correction result.
Further, the fuzzy core estimation module extracts shallow layer features from one convolution layer, then passes through three residual blocks formed by the affine convolution layers of the MANet, takes one stride convolution as a downsampling layer and takes deconvolution as an upsampling layer, predicts information of fuzzy cores at the fuzzy core reconstruction module formed by one convolution layer and a pooling layer, and uses Softmax to restrict the sum of the fuzzy cores to 1.
The fuzzy core estimation module is similar to U-Net, a convolution layer and a pooling layer are applied to the final fuzzy core reconstruction module to predict fuzzy core information, and a Softmax layer is added to restrict the generated fuzzy core, so that the algorithm obtains a better convergence effect.
Further, the restorer adopts a double-path degradation sensing block to divide the two branches to process degradation information and low-resolution images in the restorer; for the input of two branches of a module, basic input and conditional input are respectively defined as the input of a low-resolution image and the input of degradation information, the degradation information is connected with the characteristics of the low-resolution image through degradation perception convolution, and the fusion characteristics generated by the degradation perception convolution are subjected to characteristic mapping through a subsequent convolution module.
Further, the degradation perception convolution firstly carries out convolution treatment on a condition input branch so that different characteristic information can be used for guiding image reconstruction on the basis input branch with different depths, then the generated degradation information is connected with low-resolution image characteristics through the degradation perception convolution, and the fusion characteristics generated by the degradation perception convolution are subjected to characteristic mapping through a subsequent convolution module; in the process of stacking the two-way degradation sensing blocks, the restorer adopts a residual nesting mode, and adds a long jump connection after a plurality of two-way degradation sensing blocks to form a two-way degradation sensing group so as to further help network stability training.
Further, the restorer adopts 5 two-way degradation sensing groups, and 10 two-way degradation sensing blocks are stacked in each two-way degradation sensing group.
Further, the estimator adopts a two-way affine block to define basic input and conditional input as input of low-resolution image information and input of high-resolution image information respectively; for high resolution image features, affine modules are used to fuse with low resolution image features.
Further, the two-way affine block is divided into two branches to respectively process the high-resolution image characteristic and the low-resolution image characteristic in the estimator; in the estimator, a long jump connection is added after a number of two-way affine blocks to form a two-way affine group.
Further, the estimator employs 1 two-way affine group, and stacks 10 two-way affine blocks in the two-way affine group.
Specifically, in the initial stage of the network, the low resolution image is first estimated by the estimation module of the blur kernel to generate initialized blur kernel information. The fuzzy kernel estimation module adopts a similar MANet [8] As an initial fuzzy core estimation network. For initial fuzzy kernel estimation, because of the lack of high-resolution image information, a MANet network is adopted to estimate fuzzy kernel information only from a low-resolution image, and the network adopts a U-Net structure and consists of a convolution layer, three residual blocks, a fuzzy kernel reconstruction module, a downsampling layer and an upsampling layer. Using a convolution layer and a pooling layer to predict information of fuzzy core in final fuzzy core reconstruction module and according to literature [9] When the sum of the fuzzy cores is constrained to be 1, the algorithm of the fuzzy cores can be better converged. The blur kernel is therefore constrained at the end using the Softmax layer to generate the final blur kernel.
In the restorer, the invention adopts a double-path degradation sensing block as a basic module of the module, and two paths are utilized for processing respectively in order to utilize the information input additionally in the sub-network. First, for the input of two branches of the module, the basic input and the conditional input are defined as the input of the low resolution image and the input of the degradation information, respectively. For conditions ofThe input branch is firstly subjected to convolution processing to enable the input branch to have different characteristic information at the basic input branch with different depths to guide image reconstruction, and then the generated degradation information is subjected to degradation perception convolution [10] To correlate the degradation information with the low resolution image features and to map the fusion features generated by the degradation-aware convolution to features via a subsequent convolution module. Since the two branches are independent of each other in processing features, convolution kernels of different sizes can be flexibly selected to process features of different dimensions.
In the estimator, the invention adopts a double-path affine block as a basic module of the module, and in the estimator module, basic input and conditional input are respectively defined as input of low-resolution image information and input of high-resolution image information. In order to maintain the same spatial dimension as the low resolution image, the high resolution image needs to undergo step convolution to reduce the spatial dimension, and as a conditional input, estimation due to the blur kernel is also an ill-posed problem. The correlation between the low resolution image and the high resolution image is thus important information that the network can use to predict the blur kernel. For the estimator, a two-way affine block is adopted as a main module of the estimator, and for the high-resolution image characteristics, the affine block and the low-resolution image characteristics are fused, so that the interaction capability between channels is enhanced. Similar to the restorer, a long jump join is added in the estimator after several two-way affine block blocks to compose a two-way affine group.
Compared with the prior art, the invention has the following beneficial effects: and precisely estimating the fuzzy core by a fuzzy core estimation module to serve as an initialized fuzzy core, and utilizing the two-path degradation sensing block and the two-path affine block to better adapt to condition input. The method can realize the function of blind super-resolution reconstruction, thereby obtaining high-quality high-resolution images.
Drawings
The invention is described in further detail below with reference to the attached drawings and detailed description:
fig. 1 is a schematic structural diagram of a fuzzy core estimation module according to an embodiment of the present invention.
Fig. 2 is a schematic structural diagram of a two-way degradation sensing block according to an embodiment of the present invention.
FIG. 3 is a schematic diagram of the structure of a degradation-aware convolution in an embodiment of the present invention.
Fig. 4 is a schematic diagram of the structure of a two-way affine block according to an embodiment of the invention.
Fig. 5 is a schematic view of affine block structure according to an embodiment of the invention.
Fig. 6 is a schematic diagram of the overall structure of a network in an embodiment of the present invention.
FIG. 7 is a schematic diagram of alternative optimization of the estimator and the restorer in an embodiment of the present invention.
FIG. 8 is a graph comparing the reconstruction results of the present method with the other three methods in the example of the present invention.
Detailed Description
In order to make the features and advantages of the present patent more comprehensible, embodiments accompanied with figures are described in detail below:
it should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application. As used herein, the singular is also intended to include the plural unless the context clearly indicates otherwise, and furthermore, it is to be understood that the terms "comprises" and/or "comprising" when used in this specification are taken to specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof.
The embodiment provides an alternate optimized image blind super-resolution reconstruction method based on accurate kernel estimation, which adopts a fuzzy kernel estimation module at the network head, then takes alternate iteration of an estimator and a restorer as an reasoning process, inputs a low-resolution image X into a network, and outputs a corresponding high-resolution image by the network
As shown in fig. 1-7, the fuzzy core estimation module of the present embodiment extracts shallow features from a convolution layer, and then passes through a MANet [8] And finally, predicting the information of the fuzzy core by using a convolution layer and a pooling layer, and restricting the sum of the fuzzy cores to 1 by using Softmax to obtain a preliminary fuzzy core of a degradation mode corresponding to the low-resolution image so as to further help a restorer and an estimator to generate a more accurate high-resolution image and the fuzzy core.
As shown in fig. 3, the two-way degradation sensing block of the present embodiment is divided into two branches to process degradation information and low resolution images in the restorer, and first, for the inputs of the two branches of the module, basic inputs and conditional inputs are defined as the input of the low resolution images and the input of the degradation information, respectively. Where the degraded perceived block convolution DA (Degradation-aware) Conv is denoted as degraded perceived convolution as shown in fig. 4. Firstly, a condition input branch is subjected to convolution processing so that different characteristic information can be provided for basic input branches with different depths to guide image reconstruction, then the generated degradation information is connected with low-resolution image characteristics through degradation perception convolution, and fusion characteristics generated by the degradation perception convolution are subjected to characteristic mapping through a subsequent convolution module. Since the two branches are independent of each other in processing features, convolution kernels of different sizes can be flexibly selected to process features of different dimensions. And finally, in the process of stacking the two-way degradation sensing blocks, a residual nesting mode is adopted, and a long jump connection is added after a plurality of two-way degradation sensing blocks to form a two-way degradation sensing group, so that the network stability training is further assisted. Wherein the two-way degradation perception group is also an important component of the restorer network module. In this embodiment, the designed network employs 5 two-way degradation sensing groups, and 10 two-way degradation sensing blocks are stacked in each two-way degradation sensing group.
As shown in fig. 5, the two-way affine block provided in this embodiment is divided into two branches to process the high-resolution image feature and the low-resolution image feature in the estimator module, and in order to maintain the same spatial dimension as the low-resolution image, the high-resolution image needs to undergo step convolution to reduce the spatial dimension as a conditional input, and the estimation of the blur kernel is also an ill-posed problem. The correlation between the low resolution image and the high resolution image is thus important information that the network can use to predict the blur kernel. Thus, for the estimator, a two-way Affine block is used as the main block of the estimator, wherein Affine blocks represented by Affine are shown in fig. 6, and for high resolution image features, affine blocks are used for fusion with low resolution image features. Compared with the mode of directly splicing the high-resolution image features and the low-resolution image features, the affine module is utilized for feature fusion, so that the calculated amount is greatly saved, and the interaction capability among channels is enhanced. Similar to the two-way degradation perceptual set, a long jump connection is added after several two-way affine blocks in the estimator to compose a two-way affine set. For the estimator, in the present embodiment, the designed network takes 1 two-way affine group, and 10 two-way affine blocks are stacked in the two-way affine group. The reasoning process of the estimator and the restorer skillfully forms a closed loop, and can be iterated for any number of times, and the iteration mode can effectively promote more accurate prediction of the fuzzy core and the SR image.
The embodiment is from DIV2K [11] And Flickr2K [12] 3450 pictures are collected as a training set. The size of the blur kernel in the degradation set is set to 21 and during training, is respectively from [0.2,4.0]And uniformly sampling the nuclear width of training data of the x4 network, and dynamically synthesizing the high-resolution image to generate a blurred picture. At the same time, use Set5 [13] 、Set14 [14] 、Urban100 [15] 、BSD100 [16] And Manga109 [17] As a test set, since all methods need to be fixed fuzzy core parameters for fair comparison when testing the network, the same is adopted [1.80,3.20 ]]Section unification8 blur kernels of fixed kernel width are selected to generate 8 low resolution images of different degradation modes. Batch size (batch size) was set to 40 during training and all passed through 5.5X10 5 The training is completed with a plurality of iterations. Supervised learning using L1 as a loss function only at the end of the network and optimizing parameters using Adam optimizer, where β 1 And beta 2 Set to 0.9 and 0.99, respectively. Set to 4×10 for initial learning rate -4 Take 1X 10 per pass 5 Is a training strategy in which the learning rate decays by half. The network takes 4 days to complete training using 5 Nvidia Titan XP GPU.
In order to fully research the effectiveness of the method of the embodiment, the invention is matched with the blind super-resolution reconstruction network of the current mainstream and uses Bicubic [18] Quantitative comparisons were made as a baseline method. The five main stream data sets and the downsampling arrangement set up mentioned above were tested in case of x4, respectively, and the quality of the SR image was measured using peak signal to noise ratio (PSNR) and Structural Similarity (SSIM), the result of the test being the average of the results generated in case of eight different degradation modes selected according to the above method. As shown in Table 1, it is contemplated that the present invention is based on the improvement of DANv2 and is therefore designated DANv3. It can be seen that the best results are shown in most data sets compared to other methods, with better generalization.
TABLE 1
In terms of reconstructing visual effects, as shown in fig. 8, the present invention and the other three main stream methods visually compare the image generation result of x4 downsampling under different blur kernel degradation modes with the main stream methods described above, since the pictures in the dataset Urban100 have more texture features than the other datasets. Compared with other methods, the texture features generated by the method can effectively relieve the blurring artifacts and generate images with better visual quality. This benefits from a better way of estimating the degradation information and a more efficient use of the degradation information in the method of the present embodiment.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The patent is not limited to the best mode, any person can obtain other various alternative optimization image blind super-resolution reconstruction methods based on accurate kernel estimation under the teaching of the patent, and all equivalent changes and modifications made according to the scope of the patent application are covered by the patent.
Claims (2)
1. The alternating optimized image blind super-resolution reconstruction method based on the accurate kernel estimation is characterized in that the adopted super-resolution reconstruction network consists of a fuzzy kernel estimation module, an estimator and a restorer; the method comprises the steps of adopting a fuzzy kernel estimation module at the head of a network, then alternately iterating an estimator and a restorer to be used as a reasoning process, inputting a low-resolution image X into the network, and outputting a corresponding high-resolution image by the network
Firstly, inputting a low-resolution image X into the fuzzy kernel estimation module to estimate fuzzy kernels of a degradation mode corresponding to the low-resolution image; the following two steps are then alternately performed as a process of reasoning: introducing the estimated blur kernel and the low-resolution image X as inputs of a restorer to generate a high-resolution image; and generating a blur kernel with the high resolution image and the low resolution image X as inputs of an estimator to achieve a stepwise correction result;
the fuzzy core estimation module extracts shallow layer characteristics by a convolution layer, then three residual blocks formed by a MANet affine convolution layer are processed, a stride convolution is used as a downsampling layer and a deconvolution is used as an upsampling layer, and finally, a fuzzy core reconstruction module formed by a convolution layer and a pooling layer predicts information of fuzzy cores, and Softmax is used for restraining the sum of the fuzzy cores to be 1;
the restorer adopts a double-path degradation sensing block to divide the two branches to process degradation information and low-resolution images in the restorer; for the input of two branches of a module, respectively defining basic input and conditional input as input of a low-resolution image and input of degradation information, associating the degradation information with low-resolution image features through degradation perception convolution of the degradation information, and carrying out feature mapping on fusion features generated by the degradation perception convolution through a subsequent convolution module;
firstly, carrying out convolution treatment on a condition input branch so that different characteristic information can be used for guiding image reconstruction on basic input branches with different depths, then, linking the generated degradation information with low-resolution image characteristics through the degradation perception convolution, and carrying out characteristic mapping on fusion characteristics generated by the degradation perception convolution through a subsequent convolution module; in the process of stacking the two-way degradation sensing blocks, the restorer adopts a residual nesting mode, and adds a long jump connection after a plurality of two-way degradation sensing blocks to form a two-way degradation sensing group so as to further help the network stability training;
the restorer adopts 5 double-path degradation sensing groups, and 10 double-path degradation sensing blocks are stacked in each double-path degradation sensing group;
the estimator adopts a double-path affine block, and basic input and conditional input are respectively defined as input of low-resolution image information and input of high-resolution image information; for the high-resolution image features, fusing the affine module with the low-resolution image features;
the two-way affine block is divided into two branches to respectively process the high-resolution image characteristic and the low-resolution image characteristic in the estimator; in the estimator, a long jump connection is added after a number of two-way affine blocks to form a two-way affine group.
2. The method for alternately optimizing blind super-resolution reconstruction of an image based on accurate kernel estimation according to claim 1, wherein the method comprises the following steps of; the estimator takes 1 two-way affine group and stacks 10 two-way affine blocks in the two-way affine group.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210744814.0A CN115131210B (en) | 2022-06-28 | 2022-06-28 | Alternating optimized image blind super-resolution reconstruction method based on accurate kernel estimation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210744814.0A CN115131210B (en) | 2022-06-28 | 2022-06-28 | Alternating optimized image blind super-resolution reconstruction method based on accurate kernel estimation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115131210A CN115131210A (en) | 2022-09-30 |
| CN115131210B true CN115131210B (en) | 2023-06-27 |
Family
ID=83379597
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210744814.0A Active CN115131210B (en) | 2022-06-28 | 2022-06-28 | Alternating optimized image blind super-resolution reconstruction method based on accurate kernel estimation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115131210B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106251297A (en) * | 2016-07-19 | 2016-12-21 | 四川大学 | A kind of estimation based on multiple image fuzzy core the rebuilding blind super-resolution algorithm of improvement |
| AU2020100275A4 (en) * | 2020-02-25 | 2020-03-26 | Huang, Shuying DR | Remote sensing image super-resolution based on multi-dictionary sparse representation with fractal classification |
| CN114170087A (en) * | 2021-12-15 | 2022-03-11 | 北京工业大学 | Cross-scale low-rank constraint-based image blind super-resolution method |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112541877B (en) * | 2020-12-24 | 2024-03-19 | 广东宜教通教育有限公司 | Defuzzification method, system, equipment and medium for generating countermeasure network based on condition |
| CN114202459B (en) * | 2021-10-26 | 2024-03-22 | 北京工业大学 | Blind image super-resolution method based on depth priori |
| CN114494015A (en) * | 2022-01-25 | 2022-05-13 | 西安电子科技大学 | Image reconstruction method based on blind super-resolution network |
-
2022
- 2022-06-28 CN CN202210744814.0A patent/CN115131210B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106251297A (en) * | 2016-07-19 | 2016-12-21 | 四川大学 | A kind of estimation based on multiple image fuzzy core the rebuilding blind super-resolution algorithm of improvement |
| AU2020100275A4 (en) * | 2020-02-25 | 2020-03-26 | Huang, Shuying DR | Remote sensing image super-resolution based on multi-dictionary sparse representation with fractal classification |
| CN114170087A (en) * | 2021-12-15 | 2022-03-11 | 北京工业大学 | Cross-scale low-rank constraint-based image blind super-resolution method |
Non-Patent Citations (1)
| Title |
|---|
| Deep Progressive Convolutional Neural Network for Blind Super-Resolution With Multiple Degradations;Jun Xiao 等;《2019 IEEE Internatinal Conference on Lmage Processing》;1-16 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115131210A (en) | 2022-09-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Huang et al. | Unfolding the alternating optimization for blind super resolution | |
| CN111028150B (en) | Rapid space-time residual attention video super-resolution reconstruction method | |
| CN113177882B (en) | Single-frame image super-resolution processing method based on diffusion model | |
| CN110363068B (en) | High-resolution pedestrian image generation method based on multiscale circulation generation type countermeasure network | |
| Soh et al. | Deep universal blind image denoising | |
| Zhang et al. | Convolutional neural network-based video super-resolution for action recognition | |
| Xu et al. | AutoSegNet: An automated neural network for image segmentation | |
| Ahn et al. | Neural architecture search for image super-resolution using densely constructed search space: DeCoNAS | |
| Ye et al. | Depth super-resolution via deep controllable slicing network | |
| Li et al. | D3c2-net: Dual-domain deep convolutional coding network for compressive sensing | |
| CN113096032B (en) | Non-uniform blurring removal method based on image region division | |
| CN108765517B (en) | Multi-volume visual data filling method based on convex optimization | |
| Thimonier et al. | Learning long term style preserving blind video temporal consistency | |
| Kim et al. | Task-agnostic vision transformer for distributed learning of image processing | |
| CN115131210B (en) | Alternating optimized image blind super-resolution reconstruction method based on accurate kernel estimation | |
| KR102319643B1 (en) | Method for processing microscopy image using artificial neural network with point spread function layer and apparatus therefore | |
| CN109272450A (en) | A kind of image oversubscription method based on convolutional neural networks | |
| CN116862795A (en) | Multistage motion blur removing method based on pixel-by-pixel degradation prediction network | |
| Yue et al. | Deep variational network toward blind image restoration | |
| CN109087247A (en) | The method that a kind of pair of stereo-picture carries out oversubscription | |
| CN105894481B (en) | Histogram equalization method and device for video monitoring equipment | |
| Yang | Super resolution using dual path connections | |
| Wong et al. | Perceptual image super-resolution with progressive adversarial network | |
| Tian et al. | A modeling method for face image deblurring | |
| Rezayi et al. | Huber Markov random field for joint super resolution |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |