CN114494015A - Image reconstruction method based on blind super-resolution network - Google Patents

Image reconstruction method based on blind super-resolution network Download PDF

Info

Publication number
CN114494015A
CN114494015A CN202210083251.5A CN202210083251A CN114494015A CN 114494015 A CN114494015 A CN 114494015A CN 202210083251 A CN202210083251 A CN 202210083251A CN 114494015 A CN114494015 A CN 114494015A
Authority
CN
China
Prior art keywords
image
convolution layer
convolution
layer
fuzzy
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210083251.5A
Other languages
Chinese (zh)
Inventor
路文
胡健
孙晓鹏
张立泽
高新波
何立火
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Xidian University
Original Assignee
Xidian University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Xidian University filed Critical Xidian University
Priority to CN202210083251.5A priority Critical patent/CN114494015A/en
Publication of CN114494015A publication Critical patent/CN114494015A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformation in the plane of the image
    • G06T3/40—Scaling the whole image or part thereof
    • G06T3/4053—Super resolution, i.e. output image resolution higher than sensor resolution
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Processing (AREA)

Abstract

The invention provides an image reconstruction method based on a blind super-resolution network, which comprises the following steps: (1) acquiring a training sample set and a test sample set; (2) constructing an image reconstruction model O based on a blind super-resolution network; (3) carrying out iterative training on the blind hyper-resolution image reconstruction network model O; (4) and acquiring an image reconstruction result. The blind super-resolution image reconstruction model constructed by the invention can perform fuzzy kernel estimation and correct fuzzy kernels according to different degraded images in a self-adaptive manner, so that the estimated fuzzy kernels are more accurate, the technical problems of fuzzy texture and structural distortion of reconstructed images caused by inaccurate fuzzy kernel estimation in the prior art are solved, and the quality of the reconstructed images is effectively improved on the premise of ensuring the resolution of the reconstructed images.

Description

Image reconstruction method based on blind super-resolution network
Technical Field
The invention belongs to the technical field of image processing, relates to an image reconstruction method, and particularly relates to an RGB image reconstruction method based on a blind super-resolution network, which can be used in the fields of video monitoring, remote sensing imaging and the like.
Background
During the imaging process, due to the influence of various factors of the imaging system, the obtained image may not be a perfect image of the real scene. The process by which an image degrades in quality during formation, propagation, and storage is called image degradation. The image reconstruction is the process of reconstructing a degraded image to restore the original appearance of a scene to the maximum extent. The image reconstruction can only make the image close to the original image as much as possible, but the accurate restoration is difficult due to factors such as noise interference. Due to the limitation of an imaging system or transmission bandwidth, the resolution of the obtained image is often low, and the image super-resolution reconstruction is to reconstruct an image with higher resolution by using the existing image. In the fields of video monitoring, remote sensing imaging and the like with strict requirements on imaging quality, the image is required to have higher resolution, and the image structure should not have structural distortion and wrong texture, so that the target is clear and is easy to identify. And measuring the quality index of the reconstructed image by adopting the peak signal-to-noise ratio and the structural similarity. In practical situations, image degradation modes are various, the degradation process is assumed to be known in general image super-resolution reconstruction, but the image degradation process in the real world is unknown, and different blur kernels can be estimated by blind super-resolution of images according to different degradation modes of different images, so that the possibility is provided for super-resolution reconstruction of images in the real world.
Most of the existing learning-based RGB image super-resolution methods assume that a blur kernel is fixed and known, image degradation modes in the real world are various, and when the methods are applied to an actual scene, the difference of image domains causes serious performance degradation. Most of the existing blind super-scoring methods are based on models, and the implicit fuzzy core is predicted by using the self-similarity of natural images. The fuzzy kernel estimated by the prediction mode is easily influenced by input noise, so that the estimated fuzzy kernel is inaccurate, and the reconstructed high-resolution image also has the problems of texture blurring, structural distortion and the like.
In the patent document "an image blind super-resolution method and system" (patent application No. CN202110471824.7, application publication No. CN 113139904A), which is filed by the university of mansion, an image blind super-resolution reconstruction method is proposed. The method comprises the following steps: acquiring a trained fuzzy core generation network and a spectrogram of a low-resolution image; inputting the spectrogram into the trained fuzzy kernel generation network to obtain a fuzzy kernel corresponding to the low-resolution image; determining a degradation characteristic map corresponding to the low-resolution image according to a fuzzy kernel corresponding to the low-resolution image; splicing the low-resolution image and the corresponding degradation characteristic map to obtain a spliced image; and inputting the mosaic into a trained convolutional neural network to obtain a high-resolution image. Although the method improves the resolution of the image, the step of estimating the blur kernel is separated from the step of restoring the high-resolution image based on the blur kernel, and only limited information in the low-resolution image can be utilized when the blur kernel is estimated, so that the accuracy of estimating the blur kernel is low, and finally the reconstructed high-resolution image has the phenomena of structural distortion, edge distortion and other geometric distortions.
Disclosure of Invention
The invention aims to provide an image reconstruction method based on a blind super-resolution network aiming at overcoming the defects in the prior art, and aims to solve the technical problems of blurred reconstructed image texture and structural distortion caused by inaccurate estimation of a blur kernel in the prior art on the premise of ensuring the image resolution.
In order to achieve the purpose, the technical scheme adopted by the invention comprises the following steps:
(1) acquiring a training sample set and a testing sample set:
(1a) selecting K RGB images to perform Gaussian blur processing, and performing 1/4 downsampling on each RGB image subjected to Gaussian blur processing to obtain K downsampled RGB images;
(1b) cutting each RGB image into image blocks with the size of H multiplied by H, and simultaneously cutting the down-sampled RGB image corresponding to the RGB image into image blocks with the size of H multiplied by H
Figure BDA0003486764700000021
And taking the image block clipped from each RGB image as the label of the corresponding clipped image block after down sampling, and then randomly selectingMore than half of the down-sampled image blocks and labels thereof are taken to form a training sample set R1Combining the residual down-sampled image blocks and labels thereof into a test sample set E1Wherein K is more than or equal to 2000, and H is more than or equal to 256;
(2) constructing an image reconstruction model O based on a blind super-resolution network:
constructing a blind hyper-resolution image reconstruction network model O arranged by D image restoration networks and D-1 fuzzy kernel estimation networks at intervals, wherein the loss function L of the O is represented by an L1 norm loss function L1And structural similarity loss function LSSIMComposition, L ═ L1+LSSIMWherein D is more than or equal to 2;
the image restoration network comprises a first convolution layer, R residual distillation modules and an up-sampling module level which are sequentially cascaded; the residual distillation module comprises Q cascaded residual distillation units and a second convolution layer, the residual distillation units comprise a third convolution layer, a fourth convolution layer, a first nonlinear activation layer and a fifth convolution layer which are cascaded in sequence, the input end of the fifth convolution layer is simultaneously connected with the output ends of the third convolution layer and the fourth convolution layer, the output end of the fifth convolution layer is cascaded with the output end of the nonlinear activation layer, Q is more than or equal to 2, and R is more than or equal to 2;
the fuzzy kernel estimation network comprises a convolution module, an image quality regression network and a first pooling layer which are sequentially cascaded; the convolution module comprises a sixth convolution layer and a seventh convolution layer which are arranged in parallel; the image quality regression network comprises U cascaded image quality regression modules, each image quality regression module comprises an eighth convolutional layer, a second nonlinear activation layer, a ninth convolutional layer and a channel attention module which are sequentially cascaded, each channel attention module comprises a second pooling layer, a tenth convolutional layer, a third nonlinear activation layer, an eleventh convolutional layer and a fourth nonlinear activation layer which are sequentially cascaded, and the output of the channel attention module is added with the input of the image attention module to serve as the output of the standard image attention module; wherein U is more than or equal to 2;
(3) carrying out iterative training on the blind hyper-resolution image reconstruction network model O:
(3a) randomly initializing fuzzy kernel to Q, initializing fuzzy kernel pool to M, and initializingThe initialized fuzzy kernel characteristic vector is T-M.Q, the initialized iteration number is S, the maximum iteration number is S, S is more than or equal to 10000, and the current reconstructed network model is Os，OsThe weight parameter and the fuzzy kernel feature vector are respectively omegas、TsAnd let s equal to 0, Os＝O，Ts＝T，Qs＝Q；
(3b) Will be derived from the training sample set R1Selecting N training samples and fuzzy kernel characteristic vectors T at random and with replacementsImage reconstruction model OsInput of (2), image restoration network to training samples and fuzzy kernel feature vector TsCarrying out image reconstruction to obtain an intermediate image, carrying out fuzzy kernel estimation on the intermediate image and the training sample by a fuzzy kernel estimation network to obtain a fuzzy kernel characteristic vector QsAnd using a fuzzy kernel loss function, passing through a fuzzy kernel QsCalculating the loss value LregularizationThen, calculate LregularizationPartial derivation of the blur kernel parameter b
Figure BDA0003486764700000031
Then, updating the fuzzy kernel parameter b by adopting a gradient descent method, and finally mapping the updated fuzzy kernel back to the fuzzy kernel characteristic vector TsThe updated fuzzy kernel feature vector TsInputting a training sample into a next image restoration network for image reconstruction, and performing alternate reciprocating in such a way, wherein the last image restoration network outputs a reconstructed image of the iteration;
(3c) adopting the loss function of O, and calculating O through the reconstructed image and the corresponding training sample labelsLoss value L ofsCalculating LsFor weight parameter omegasPartial derivatives of
Figure BDA0003486764700000032
Then adopting a gradient descent method to perform
Figure BDA0003486764700000033
At OsThe weight parameter omega is subjected to counter propagationsUpdating is carried out;
(3d) judgment ofWhether S is more than or equal to S is established, if so, obtaining a trained blind over-fraction image reconstruction network model O*And fuzzy kernel feature vector T*Otherwise, let s be s +1, and execute step (3 b);
(4) acquiring an image reconstruction result:
test sample set and fuzzy kernel feature vector T*As a trained image reconstruction model O*The input of (a) is propagated forward to obtain a reconstructed image.
Compared with the prior art, the invention has the following advantages:
the image reconstruction model constructed by the invention comprises a blind super-resolution network consisting of an image restoration network and a fuzzy self-kernel estimation network, and can repeatedly estimate a fuzzy kernel and correct the fuzzy kernel according to a reconstructed image result in the process of training the model and acquiring an image reconstruction result, so that the estimation of the fuzzy kernel is more accurate, and the final reconstructed image structure and texture are more accurate.
Drawings
FIG. 1 is a flow chart of an implementation of the present invention;
FIG. 2 is a schematic diagram of an image restoration network according to the present invention;
FIG. 3 is a schematic diagram of a fuzzy core estimation network according to the present invention;
fig. 4 is a schematic structural diagram of an image reconstruction model of the blind super-resolution network of the invention.
Detailed Description
The invention is described in further detail below with reference to the following figures and specific examples:
referring to fig. 1, the present invention includes the steps of:
step 1) obtaining a training sample set R1And test sample set E1：
Step 1a) acquiring K RGB images from DIV2K and Flickr2K data sets, wherein K is larger than or equal to 2000. In the present embodiment, K — 3450;
step 1b) in order to simulate the real world down-sampling process and facilitate the mutual comparison among different experiments, different parameters are randomly adopted to carry out Gaussian blur processing on each RGB image, and 1/4 down-sampling is carried out on each RGB image after the Gaussian blur processing, and the implementation steps are as follows: setting the size of a Gaussian blur kernel to be 21, randomly selecting sigma in a [0.2,4.0] interval, performing template convolution on each RGB image, and performing 1/4 bicubic downsampling on each RGB image subjected to Gaussian blur processing;
step 1c) cutting each RGB image into image blocks with the size of H multiplied by H, and simultaneously cutting the corresponding down-sampled RGB image of the RGB image into image blocks with the size of H multiplied by H
Figure BDA0003486764700000041
The image blocks cut from each RGB image are used as labels of the corresponding cut image blocks after down-sampling, and then more than half of the cut image blocks after down-sampling and the labels thereof are randomly selected to form a training sample set R1Combining the residual down-sampled image blocks and labels thereof into a test sample set E1Wherein H is more than or equal to 256; h-256 in this embodiment;
step 2) constructing an image reconstruction model O based on a blind super-resolution network, wherein the structure of the image reconstruction model O is shown in FIG. 4;
constructing a blind hyper-resolution image reconstruction network model O arranged by D image restoration networks and D-1 fuzzy kernel estimation networks at intervals, wherein the loss function L of the O is represented by an L1 norm loss function L1And structural similarity loss function LSSIMComposition, L ═ L1+LSSIMWherein D is not less than 2, and in the embodiment, D is 3;
wherein the image restoration network structure is shown in FIG. 2;
the image restoration network comprises a first convolution layer, R residual distillation modules and an up-sampling module which are sequentially cascaded; the residual distillation module comprises V cascaded residual distillation units and a second convolution layer, the residual distillation units comprise a third convolution layer, a fourth convolution layer, a first nonlinear activation layer and a fifth convolution layer which are cascaded in sequence, the input end of the fifth convolution layer is simultaneously connected with the output ends of the third convolution layer and the fourth convolution layer, the output end of the fifth convolution layer is cascaded with the output end of the nonlinear activation layer, V is larger than or equal to 2, and R is larger than or equal to 2; in this embodiment, V ═ 3, R ═ 10;
the image restoration network parameters are set as follows: the sizes of convolution kernels of a first convolution layer, a third convolution layer and a fifth convolution layer are set to be 1 x 1, the sizes of convolution kernels of a second convolution layer and a fourth convolution layer are set to be 3 x 3, the step lengths of the convolution kernels of the first convolution layer, the second convolution layer, the third convolution layer, the fourth convolution layer and the fifth convolution layer are all set to be 1, the number of convolution kernels of the first convolution layer is set to be 64, the number of convolution kernels of the second convolution layer and the third convolution layer is set to be 72, the number of convolution kernels of the fourth convolution layer and the fifth convolution layer is set to be 48, the first nonlinear active layer is realized by a LeakyReLU function, the up-sampling module is realized by PixelShuffle, and the amplification parameter is set to be 4;
wherein, the fuzzy core estimation network has the structure as shown in FIG. 3;
the fuzzy kernel estimation network comprises a convolution module, an image quality regression module and a first pooling layer which are sequentially cascaded; the convolution module comprises a sixth convolution layer and a seventh convolution layer which are arranged in parallel; the image quality regression module comprises U cascaded standard image quality regression modules, the image quality regression module comprises an eighth convolutional layer, a second nonlinear activation layer, a ninth convolutional layer and a channel attention module which are sequentially cascaded, the channel attention module comprises a second pooling layer, a tenth convolutional layer, a third nonlinear activation layer, an eleventh convolutional layer and a fourth nonlinear activation layer which are sequentially cascaded, and the output of the channel attention module is added with the input of the image attention module to serve as the output of the standard image attention module; wherein U is more than or equal to 2; in the embodiment, U is 5;
the fuzzy core estimation network parameters are set as follows: the sizes of convolution kernels of a seventh convolution layer, an eighth convolution layer and a ninth convolution layer are 3 x 3, convolution step sizes of a sixth convolution layer and a tenth convolution layer are all set to be 1 x 1, convolution step sizes of the eleventh convolution layer and the ninth convolution layer are set to be 1, convolution step sizes of the seventh convolution layer and the tenth convolution layer are set to be 4, the number of convolution kernels of the sixth convolution layer, the seventh convolution layer and the ninth convolution layer is set to be 32, the number of convolution kernels of the eighth convolution layer and the eleventh convolution layer is set to be 64, the number of convolution kernels of the tenth convolution layer is set to be 4, the second nonlinear active layer and the third nonlinear active layer are realized by a LeakyReLU function, the fourth nonlinear active layer is realized by a Sigmoid function, and the pooling layer is set to be average pooling;
step 3) carrying out iterative training on the image reconstruction model O based on the blind hyper-separation network;
step 3a) randomly initializing a fuzzy core to be Q, initializing a fuzzy core pool to be M, initializing a fuzzy core characteristic vector to be T as M.Q, initializing the iteration frequency to be S, maximizing the iteration frequency to be S, wherein S is more than or equal to 10000, and the current reconstructed network model is Os，OsThe weight parameter and the fuzzy kernel feature vector are respectively omegas、TsAnd let s equal to 0, Os＝O，Ts＝T，QsQ; in this embodiment, S ═ 200000;
step 3b) from the training sample set R1Selecting N training samples and fuzzy kernel characteristic vectors T at random and with replacementsImage reconstruction model OsThe input of (1);
step 3c) the first convolution layer in the image restoration network convolves the input image to obtain a characteristic diagram Y of the input image1Wherein f ismRepresenting a first feature map after convolution of the mth training sample;
step 3d) reconstructing the fuzzy core characteristic vector TsIs a four-dimensional tensor Ts 1Wherein the first dimension and the second dimension are unchanged, and the third dimension and the fourth dimension are both 1; for the reconstructed fuzzy kernel characteristic vector Ts 1Performing tensor expansion operation to make the fuzzy kernel eigenvector Ts 1The third dimension and the fourth dimension become
Figure BDA0003486764700000061
Obtaining a fuzzy nucleus characteristic tensor Ts 2；
Step 3e) the first characteristic diagram Y1And the fuzzy nuclear feature tensor Ts 2Performing channel splicing and serving as the input of the residual distillation modules, performing feature fusion and feature distillation on input features by the R residual distillation modules, and fusing the first feature diagram Y1And the fuzzy nuclear feature tensor Ts 2And performing characteristic distillation reductionData quantity to obtain a depth feature map Y2The second convolution layer will be depth feature map Y2The channel of (2) is projected to 48, so that the characteristic diagram is reconstructed by an up-sampling layer to obtain an intermediate image I1The up-sampling layer can ensure that the resolution of the reconstructed image is 4 times of that of the input image;
step 3f) seventh convolution layer pair intermediate image I in fuzzy kernel estimation network1Extracting the features to obtain a feature map Y of the intermediate imageI1The eighth convolution layer performs feature extraction on the input sample to obtain a feature map Y of the input sampleRThe step size of the seventh convolution layer is 4, which ensures the feature map Y of the intermediate imageI1And the feature map Y of the input sampleRFeature pattern Y of intermediate image with uniform sizeI1And the feature map Y of the input sampleRIncluding degradation information of the intermediate image to the input samples. Intermediate image feature map YI1And inputting a sample feature map YRObtaining a splicing characteristic diagram Y after channel splicing3Image quality regression module pair mosaic feature map Y3Extracting a degradation feature map Y4The channel attention module in the image quality regression module can assign different weights to different channel feature maps, and fuzzy kernel information contained in the feature maps can be more accurate. Degradation profile Y4Pooling the feature map to a one-dimensional vector by a pooling layer to obtain a fuzzy kernel feature vector T1Updating the fuzzy kernel parameter by adopting a fuzzy kernel loss function, and correcting the fuzzy kernel parameter, wherein the method comprises the following implementation steps:
step 3f1) calculating fuzzy kernel feature vector TsCorresponding fuzzy kernel Qs：
Qs＝Ts·inv(M)
Where inv (·) represents inverting the matrix;
step 3f2) calculating a fuzzy kernel QsLoss L ofregularization：
Figure BDA0003486764700000071
WhereinΣ denotes a summation operation, m denotes a constant mask of weightsi,jThe value of the constant mask representing the weight at position (i, j), bi,jRepresenting a fuzzy kernel QsThe value of the parameter at (i, j), (x)0,y0) A center index representing a blur kernel parameter b; in the present embodiment
g1＝1.373E-4 g2＝4.120E-4 g3＝-1.328E-3 g4＝-5.081E-3
g5＝1.236E-3 g6＝-3.983E-3 g7＝-1.524E-2
g8＝1.283E-2 g9＝4.912E-2 g10＝0.188
Figure BDA0003486764700000072
Figure BDA0003486764700000073
Figure BDA0003486764700000081
Step 3f3) fuzzy core updating formula is as follows:
Figure BDA0003486764700000082
wherein eta represents LregularizationThe learning rate of (a) is determined,
Figure BDA0003486764700000083
representing a derivation operation, bi′,jRepresenting the post-update fuzzy kernel QsThe parameters of (a);
step 3f4) mapping the updated fuzzy kernel back to the fuzzy kernel feature vector to obtain the updated fuzzy kernel feature vector Ts′：
Ts′＝M·Q′s
Wherein Q'sIs the updated blur kernel.
Step 3g) updating the fuzzy core characteristic vector TsInputting the training samples into the next image recovery network for image recovery, and repeating the steps alternately, wherein the last image recovery network outputs the recovered image of the iteration;
step 3H) adopting the loss function of H, and calculating O through the reconstructed image and the corresponding training sample labelsLoss value L ofsCalculating LsFor weight parameter omegasPartial derivatives of
Figure BDA0003486764700000084
Then adopting a gradient descent method to perform
Figure BDA0003486764700000085
At OsThe weight parameter omega is subjected to counter propagationsUpdating is carried out;
calculating OsFor the weight parameter omegasUpdating, wherein the calculation and the updating are respectively as follows:
Ls＝L1+LSSIM
Figure BDA0003486764700000086
Figure BDA0003486764700000087
where, Σ denotes a summing operation,
Figure BDA0003486764700000088
and I represents a reconstructed image block output by a blind super-resolution network, and I represents a label of a sample in a training sample set. SSIM (. cndot.,) represents the computational structural similarity, L1Represents the L1 norm loss function, LSSIMRepresenting a structural similarity loss function;
step 3i) of determining whether S is greater than or equal to SIf yes, obtaining a trained blind super-resolution network image reconstruction model O*And fuzzy kernel feature vector T*Otherwise, let s be s +1, and execute step (3 b);
step 4), obtaining an image reconstruction result:
test sample set and fuzzy kernel feature vector T*As a trained image reconstruction model O*The input of (a) is propagated forward to obtain a reconstructed image.
The image reconstruction model of the blind super-resolution network, which is constructed by the invention, comprises an image recovery network which can extract degradation information of a fuzzy core, and can be fused with multilayer features in an input image to reconstruct the image, a fuzzy core self-estimation network realizes self-supervision updating of the fuzzy core through regular constraint by means of the feature difference of the image, can realize self-adaption fuzzy core estimation according to different images with the same degradation and the same images with different degradation, avoids the problems of fuzzy texture blurring and structural distortion in reconstructed images caused by inaccurate estimation of the fuzzy core in the prior art, and effectively improves the quality of RGB image super-resolution.
The technical effects of the present invention are further described below with reference to simulation experiments:
1. simulation conditions and contents:
the hardware platform used in the simulation experiment is a CPU
Figure BDA0003486764700000091
CoreTMi9-9980XE, master frequency 3GHz, 128G RAM. The software platforms are Python3.7 and Pycharm 2019.3.3 x 64. The operating system is Ubuntu 18.04 LTS x 64.
The RGB image dataset used in the simulation experiment was the DIV2K dataset, the Filck2K dataset. 3450 RGB images are selected to form a sub-data set in a simulation experiment, and more than 80% of the RGB images corresponding to each target category are selected to form a data set R in the sub-data set0R after pretreatment0Form a training sample set R1While removing R0The other RGB images constitute the test sample set E1。
Compared with the prior art, the image blind super-resolution reconstruction method in the image blind super-resolution method and the image blind super-resolution reconstruction system carry out comparison simulation of peak signal-to-noise ratio and structural similarity, and the result is shown in Table 1
Referring to Table 1, the present invention is testing sample set E1The peak signal-to-noise ratio of (1) is 29.29dB, the structural similarity is 0.8431, and the prior art tests a sample set E1The peak signal-to-noise ratio of (3) is 28.95dB and the structural similarity is 0.8375. Compared with the prior art, the peak signal-to-noise ratio and the structural similarity are improved.
The invention Prior Art
PSNR 29.29dB 28.95dB
SSIM 0.8431 0.8375
TABLE 1
By combining the result analysis in the simulation experiment, the method provided by the invention can effectively solve the problem that the fuzzy kernel estimation of the traditional blind super-resolution image reconstruction method is inaccurate, and further solve the problems of fuzzy texture and structural distortion of the reconstructed image.

Claims (4)

1. An image reconstruction method based on a blind super-resolution network is characterized by comprising the following steps:
(1) acquiring a training sample set and a testing sample set:
(1a) selecting K RGB images to perform Gaussian blur processing, and performing 1/4 downsampling on each RGB image subjected to Gaussian blur processing to obtain K downsampled RGB images;
(1b) cutting each RGB image into image blocks with the size of H multiplied by H, and simultaneously cutting the down-sampled RGB image corresponding to the RGB image into image blocks with the size of H multiplied by H
Figure FDA0003486764690000011
The image blocks cut from each RGB image are used as labels of the corresponding cut image blocks after down-sampling, and then more than half of the cut image blocks after down-sampling and the labels thereof are randomly selected to form a training sample set R1Combining the residual down-sampled image blocks and labels thereof into a test sample set E1Wherein K is more than or equal to 2000, and H is more than or equal to 256;
(2) constructing an image reconstruction model O based on a blind super-resolution network:
constructing a blind hyper-resolution image reconstruction network model O arranged by D image restoration networks and D-1 fuzzy kernel estimation networks at intervals, wherein the loss function L of the O is represented by an L1 norm loss function L1And structural similarity loss function LSSIMComposition, L ═ L1+LSSIMWherein D is more than or equal to 2;
the image restoration network comprises a first convolution layer, R residual distillation modules and an up-sampling module level which are sequentially cascaded; the residual distillation module comprises V cascaded residual distillation units and a second convolution layer, the residual distillation units comprise a third convolution layer, a fourth convolution layer, a first nonlinear activation layer and a fifth convolution layer which are cascaded in sequence, the input end of the fifth convolution layer is simultaneously connected with the output ends of the third convolution layer and the fourth convolution layer, the output end of the fifth convolution layer is cascaded with the output end of the nonlinear activation layer, V is larger than or equal to 2, and R is larger than or equal to 2;
the fuzzy kernel estimation network comprises a convolution module, an image quality regression network and a first pooling layer which are sequentially cascaded; the convolution module comprises a sixth convolution layer and a seventh convolution layer which are arranged in parallel; the image quality regression network comprises U cascaded image quality regression modules, each image quality regression module comprises an eighth convolutional layer, a second nonlinear activation layer, a ninth convolutional layer and a channel attention module which are sequentially cascaded, each channel attention module comprises a second pooling layer, a tenth convolutional layer, a third nonlinear activation layer, an eleventh convolutional layer and a fourth nonlinear activation layer which are sequentially cascaded, and the output of the channel attention module is added with the input of the image attention module to serve as the output of the standard image attention module; wherein U is more than or equal to 2;
(3) carrying out iterative training on the blind hyper-resolution image reconstruction network model O:
(3a) randomly initializing a fuzzy kernel to be Q, initializing a fuzzy kernel pool to be M, initializing a fuzzy kernel characteristic vector to be T as M.Q, initializing the iteration frequency to be S, maximizing the iteration frequency to be S, wherein S is more than or equal to 10000, and the current reconstructed network model is Os，OsThe weight parameter and the fuzzy kernel feature vector are respectively omegas、TsAnd let s equal to 0, Os＝O，Ts＝T，Qs＝Q；
(3b) Will be derived from the training sample set R1Selecting N training samples and fuzzy kernel characteristic vectors T at random and with replacementsImage reconstruction model OsInput of (2), image restoration network to training samples and fuzzy kernel feature vector TsCarrying out image reconstruction to obtain an intermediate image, carrying out fuzzy kernel estimation on the intermediate image and the training sample by a fuzzy kernel estimation network to obtain a fuzzy kernel characteristic vector QsAnd using a fuzzy kernel loss function, passing through a fuzzy kernel QsCalculating the loss value LregularizationThen, calculate LregularizationPartial derivation of the blur kernel parameter b
Figure FDA0003486764690000021
Then, updating the fuzzy kernel parameter b by adopting a gradient descent method, and finally mapping the updated fuzzy kernel back to the fuzzy kernel characteristic vector TsThe updated fuzzy core feature vector T is useds' inputting training sample into next image recovery networkReconstructing the line images, and alternately reciprocating in such a way, wherein the last image restoration network outputs the reconstructed image of the iteration;
(3c) adopting the loss function of O, and calculating O through the reconstructed image and the corresponding training sample labelsLoss value L ofsCalculating LsFor weight parameter omegasPartial derivatives of
Figure FDA0003486764690000022
Then adopting a gradient descent method to perform
Figure FDA0003486764690000023
At OsThe weight parameter omega is subjected to counter propagationsUpdating is carried out;
(3d) judging whether S is more than or equal to S, if so, obtaining a trained blind hyper-resolution image reconstruction network model O*And fuzzy kernel feature vector T*Otherwise, let s be s +1, and execute step (3 b);
(4) acquiring an image reconstruction result:
test sample set and fuzzy kernel feature vector T*As a trained image reconstruction model O*The input of (a) is propagated forward to obtain a reconstructed image.
2. The blind super-resolution network-based image reconstruction method according to claim 1, wherein the image reconstruction model O in step (2) is obtained, wherein:
the image restoration network parameters are set as follows: the sizes of convolution kernels of the first convolution layer, the third convolution layer and the fifth convolution layer are set to be 1 multiplied by 1, the sizes of convolution kernels of the second convolution layer and the fourth convolution layer are set to be 3 multiplied by 3, the step lengths of the first convolution layer, the second convolution layer, the third convolution layer, the fourth convolution layer and the fifth convolution layer are all set to be 1, the number of convolution kernels of the first convolution layer is set to be 64, the number of convolution kernels of the second convolution layer and the third convolution layer is set to be 72, the number of convolution kernels of the fourth convolution layer and the fifth convolution layer is set to be 48, and the first nonlinear active layer is realized by a LeakyReLU function;
the fuzzy core estimation network parameters are set as follows: the sizes of convolution kernels of the seventh convolution layer, the eighth convolution layer and the ninth convolution layer are 3 x 3, convolution steps of the sixth convolution layer and the tenth convolution layer are all set to be 1 x 1, convolution kernel sizes of the eleventh convolution layer are set to be 1 x 1, the numbers of convolution kernels of the sixth convolution layer, the eighth convolution layer, the ninth convolution layer, the tenth convolution layer and the eleventh convolution step are all set to be 1, convolution step of the seventh convolution layer is set to be 4, the numbers of convolution kernels of the sixth convolution layer, the seventh convolution layer and the ninth convolution layer are set to be 32, the numbers of convolution kernels of the eighth convolution layer and the eleventh convolution kernel are set to be 64, the number of convolution kernels of the tenth convolution kernels is set to be 4, the second nonlinear active layer and the third nonlinear active layer are realized by a LeakyReLU function, the fourth nonlinear active layer is realized by a Sigmoid function, and the pooling layers are set to be average pooling.
3. The image reconstruction method based on the blind super-resolution network of claim 1, wherein the step (3b) of updating the blur kernel parameters is implemented by the steps of:
(3b1) computing a fuzzy kernel feature vector TsCorresponding fuzzy kernel Qs：
Qs＝Ts·inv(M)
Where inv (·) represents inverting the matrix;
(3b2) computing a fuzzy kernel QsLoss L ofregularization：
Figure FDA0003486764690000031
Where Σ denotes the summation operation, m denotes a constant mask of weights, mi,jThe value of the constant mask representing the weight at position (i, j), bi,jRepresenting a fuzzy kernel QsThe value of the parameter at (i, j), (x)0,y0) A center index representing a blur kernel parameter b;
(3b3) the fuzzy core updating formula is as follows:
Figure FDA0003486764690000041
wherein eta represents LregularizationThe learning rate of (a) is determined,
Figure FDA0003486764690000042
denotes derivation operation, b'i,jRepresenting post-update fuzzy kernel Qs' of;
(3b4) mapping the updated fuzzy kernel back to the fuzzy kernel feature vector to obtain an updated fuzzy kernel feature vector Ts′：
Ts′＝M·Q′s
Wherein Q'sIs the updated blur kernel.
4. The blind super-resolution network-based image reconstruction method according to claim 1, wherein the loss function of O in step (2), wherein L1 norm loss function L1And a structural similarity loss function LSSIMThe expressions are respectively:
Figure FDA0003486764690000043
Figure FDA0003486764690000044
where, Σ denotes a summing operation,
Figure FDA0003486764690000045
and the reconstructed image block is output through a blind super-resolution network, I represents a label of a sample in a training sample set, and SSIM (·,) represents the calculation of structural similarity.
CN202210083251.5A 2022-01-25 2022-01-25 Image reconstruction method based on blind super-resolution network Pending CN114494015A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210083251.5A CN114494015A (en) 2022-01-25 2022-01-25 Image reconstruction method based on blind super-resolution network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210083251.5A CN114494015A (en) 2022-01-25 2022-01-25 Image reconstruction method based on blind super-resolution network

Publications (1)

Publication Number Publication Date
CN114494015A true CN114494015A (en) 2022-05-13

Family

ID=81475422

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210083251.5A Pending CN114494015A (en) 2022-01-25 2022-01-25 Image reconstruction method based on blind super-resolution network

Country Status (1)

Country Link
CN (1) CN114494015A (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115131210A (en) * 2022-06-28 2022-09-30 闽江学院 Alternating optimization image blind super-resolution reconstruction method based on precise kernel estimation
CN117522687A (en) * 2023-11-03 2024-02-06 西安电子科技大学 Super-resolution reconstruction method of hyperspectral image based on particle dynamics
TWI832787B (en) * 2022-07-22 2024-02-11 聯發科技股份有限公司 Generating a high resolution image from a low resolution image for blind super-resolution
CN117522687B (en) * 2023-11-03 2024-05-14 西安电子科技大学 Super-resolution reconstruction method of hyperspectral image based on particle dynamics

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115131210A (en) * 2022-06-28 2022-09-30 闽江学院 Alternating optimization image blind super-resolution reconstruction method based on precise kernel estimation
TWI832787B (en) * 2022-07-22 2024-02-11 聯發科技股份有限公司 Generating a high resolution image from a low resolution image for blind super-resolution
CN117522687A (en) * 2023-11-03 2024-02-06 西安电子科技大学 Super-resolution reconstruction method of hyperspectral image based on particle dynamics
CN117522687B (en) * 2023-11-03 2024-05-14 西安电子科技大学 Super-resolution reconstruction method of hyperspectral image based on particle dynamics

Similar Documents

Publication Publication Date Title
CN110232394B (en) Multi-scale image semantic segmentation method
CN106952228B (en) Super-resolution reconstruction method of single image based on image non-local self-similarity
CN108122197B (en) Image super-resolution reconstruction method based on deep learning
CN111709895A (en) Image blind deblurring method and system based on attention mechanism
CN111598778B (en) Super-resolution reconstruction method for insulator image
CN111861884B (en) Satellite cloud image super-resolution reconstruction method based on deep learning
CN108090871A (en) A kind of more contrast MR image reconstruction methods based on convolutional neural networks
CN114494015A (en) Image reconstruction method based on blind super-resolution network
CN111127325B (en) Satellite video super-resolution reconstruction method and system based on cyclic neural network
CN110675347A (en) Image blind restoration method based on group sparse representation
CN106169174B (en) Image amplification method
Chen et al. Single image super-resolution using deep CNN with dense skip connections and inception-resnet
CN113538374A (en) Infrared image blur correction method for high-speed moving object
CN114170088A (en) Relational reinforcement learning system and method based on graph structure data
CN111242999B (en) Parallax estimation optimization method based on up-sampling and accurate re-matching
Huang et al. Deep gaussian scale mixture prior for image reconstruction
CN115526779A (en) Infrared image super-resolution reconstruction method based on dynamic attention mechanism
CN108122262B (en) Sparse representation single-frame image super-resolution reconstruction algorithm based on main structure separation
CN111310767A (en) Significance detection method based on boundary enhancement
CN114626984A (en) Super-resolution reconstruction method for Chinese text image
CN112837220B (en) Method for improving resolution of infrared image and application thereof
CN106981046A (en) Single image super resolution ratio reconstruction method based on multi-gradient constrained regression
CN115147426B (en) Model training and image segmentation method and system based on semi-supervised learning
You et al. INDigo: An INN-Guided Probabilistic Diffusion Algorithm for Inverse Problems
CN112686807A (en) Image super-resolution reconstruction method and system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination