CN112150354A - Single image super-resolution method combining contour enhancement and denoising statistical prior - Google Patents

Single image super-resolution method combining contour enhancement and denoising statistical prior Download PDF

Info

Publication number
CN112150354A
CN112150354A CN201910558515.6A CN201910558515A CN112150354A CN 112150354 A CN112150354 A CN 112150354A CN 201910558515 A CN201910558515 A CN 201910558515A CN 112150354 A CN112150354 A CN 112150354A
Authority
CN
China
Prior art keywords
image
denoising
resolution
network
prior
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201910558515.6A
Other languages
Chinese (zh)
Other versions
CN112150354B (en
Inventor
任超
何小海
翟森
王正勇
卿粼波
熊淑华
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sichuan University
Original Assignee
Sichuan University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sichuan University filed Critical Sichuan University
Priority to CN201910558515.6A priority Critical patent/CN112150354B/en
Publication of CN112150354A publication Critical patent/CN112150354A/en
Application granted granted Critical
Publication of CN112150354B publication Critical patent/CN112150354B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/73—Deblurring; Sharpening
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20172—Image enhancement details
    • G06T2207/20192—Edge enhancement; Edge preservation

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses a single image super-resolution method combining contour enhancement and denoising statistical prior. The method mainly comprises the following steps: introducing a priori based on deep learning and a continuity mechanism to obtain a single image super-resolution reconstruction framework based on an improved SBI algorithm; constructing and training a contour enhancement network PENet and a denoising statistical prior network DSPNet; constructing an image contour enhancement prior PEP, and applying the PEP to an image restoration inverse sub-problem; optimizing the image restoration inverse sub-problem of the first step by using a TFOCS technology; calculating the noise level σk(ii) a Constructing an image statistical prior DSP, and applying the image statistical prior DSP to a denoising subproblem; updating the parameters; and performing iterative reconstruction, and outputting a final super-resolution reconstruction result. The single image super-resolution reconstruction method can obtain good subjective and objective effects and has high running speed. Therefore, the invention is an effective single image super-resolution reconstruction method.

Description

Single image super-resolution method combining contour enhancement and denoising statistical prior
Technical Field
The invention relates to an image resolution improvement technology, in particular to a single image super-resolution method combining contour enhancement and denoising statistics prior, and belongs to the field of image processing.
Background
Image super-resolution reconstruction techniques utilize a single or a set of low resolution images (sequence) to produce high quality, high resolution images. The image super-resolution reconstruction application field is extremely wide, and the method has important application prospects in the aspects of military affairs, medicine, public safety, computer vision and the like. In the field of computer vision, the image super-resolution reconstruction technology can convert an image into a fine resolution level (identification level) so as to improve the identification capability and the identification precision of the image and provide abundant image information for a subsequent analysis process.
The single image super-resolution reconstruction method mainly comprises three types: interpolation-based methods, reconstruction-based methods, and learning-based methods. The reconstruction-based method mainly utilizes a specific prior term to constrain the super-resolution reconstruction process, and the learning-based convolutional neural network performs the super-resolution reconstruction by learning the mapping relation between a large number of high-low resolution image pairs. However, it is currently very challenging to significantly improve the super-resolution reconstruction effect simply by changing the traditional explicit prior term form or designing a deeper neural network structure.
Disclosure of Invention
The invention aims to obtain an explicit image outline Enhancement Prior (PEP) and an implicit image Denoising Prior (DSP) by using a Convolutional Neural Network (CNN) based on deep learning. Then, the two priors are respectively applied to an image restoration inverse sub-problem and a denoising sub-problem in an improved Split Bragman Iteration (SBI) algorithm based on a continuity mechanism, so that a single image super-resolution reconstruction method based on the improved SBI algorithm is constructed.
The invention provides a single image super-resolution method combining contour enhancement and denoising statistics prior, which mainly comprises the following operation steps:
(1) firstly, decomposing the original single-image super-resolution reconstruction problem by using an SBI algorithm to obtain an image restoration inverse sub-problem, a de-noising sub-problem and an auxiliary variable iteration equation; respectively introducing a priori based on deep learning to the obtained image restoration inverse sub-problem and the denoising sub-problem, and introducing a continuity mechanism to obtain a single image super-resolution reconstruction framework based on an improved SBI algorithm;
(2) constructing a Profile Enhancement Network (PENet) for predicting unknown high-resolution gradient Profile features aiming at an input low-resolution image;
(3) training the network constructed in the step (2) by using a training image data set, and training gradient profiles in four directions in total;
(4) aiming at the Denoising subproblem in the step (1), constructing a Denoising Statistics Prior Network (DSPNet) for predicting an unknown denoised image;
(5) training the denoising statistical prior network constructed in the step (4) by utilizing a training image data set, and training the denoising statistical prior network with twenty-five noise levels in total for dynamic denoising of the denoising subproblems in the iterative process;
(6) performing multi-directional gradient contour prediction on the input low-resolution image by using the contour enhancement network trained in the step (3) to obtain predicted high-resolution image gradient contour features in four directions, constructing an explicit image contour enhancement prior PEP by using the gradient contour features, and applying the prior to the image restoration inverse sub-problem in the step (1);
(7) optimizing the image restoration inverse sub-problem in the step (1) by using a TFOCS technology to obtain an iteratively updated high-resolution estimated image Xk+1；
(8) Calculating the noise level sigma corresponding to the de-noising subproblem in each iteration processk；
(9) Applying the denoise statistic prior network trained in step (5) to the denoise subproblem in step (1), wherein the noise level corresponds to the sigma in step (8)kAnd further improving the denoising performance by adopting a multi-estimation strategy, and finally constructing implicit image statistics in the denoising subproblemChecking DSP to obtain denoised image vk+1；
(10) Updating the penalty parameter mu in the image restoration inverse sub-problem in the step (1)kAnd iterating equation b by the auxiliary variables of step (1)k+1＝bk+Xk+1-vk+1For auxiliary variable bkUpdating is carried out;
(11) and (5) repeating the steps 7 to 10 until the specified iteration times are reached, and finally outputting the result which is the final super-resolution reconstruction result.
Drawings
FIG. 1 is a schematic block diagram of a single image super-resolution method based on joint contour enhancement and denoising statistical prior in the invention
FIG. 2 is a network architecture diagram of the profile enhancement network of the present invention
FIG. 3 is a network structure diagram of a denoising statistical prior network according to the present invention
FIG. 4 is a standard 12 test image used in the present invention
Fig. 5 is a comparison graph of the reconstruction results of the test image "Butterfly" according to the present invention and four methods (super-resolution reconstruction factor is 3, gaussian blur kernel size is 7 × 7, standard deviation is 1.5): wherein, (a) is a test image, (b) is a low-resolution image, and (c) (d) (e) (f) (g) (h) are bicubic interpolation, method 1, method 2, method 3, method 4, and the reconstruction result of the present invention
Fig. 6 is a comparison graph of the reconstruction results of the test image "Leaves" according to the present invention and four methods (super-resolution reconstruction factor is 3, gaussian blur kernel size is 7 × 7, standard deviation is 1.5, noise level is 5): wherein, (a) is a test image, (b) is a low-resolution image, and (c) (d) (e) (f) (g) (h) are bicubic interpolation, method 1, method 2, method 3, method 4, and the reconstruction result of the present invention
Detailed Description
The invention will be further described with reference to the accompanying drawings in which:
in fig. 1, the method for reconstructing the super-resolution single image based on the multi-directional feature prediction prior specifically includes the following eleven steps:
(1) firstly, decomposing the original single-image super-resolution reconstruction problem by using an SBI algorithm to obtain an image restoration inverse sub-problem, a de-noising sub-problem and an auxiliary variable iteration equation; respectively introducing a priori based on deep learning to the obtained image restoration inverse sub-problem and the denoising sub-problem, and introducing a continuity mechanism to obtain a single image super-resolution reconstruction framework based on an improved SBI algorithm;
(2) aiming at an input low-resolution image, constructing a contour enhancement network for predicting unknown high-resolution gradient contour characteristics;
(3) training the network constructed in the step (2) by using a training image data set, and training gradient profiles in four directions in total;
(4) aiming at the denoising subproblem in the step (1), constructing a denoising statistical prior network for predicting an unknown denoising image;
(5) training the denoising statistical prior network constructed in the step (4) by utilizing a training image data set, and training the denoising statistical prior network with twenty-five noise levels in total for dynamic denoising of the denoising subproblems in the iterative process;
(6) performing multi-directional gradient contour prediction on the input low-resolution image by using the contour enhancement network trained in the step (3) to obtain predicted high-resolution image gradient contour features in four directions, constructing an explicit image contour enhancement prior PEP by using the gradient contour features, and applying the prior to the image restoration inverse sub-problem in the step (1);
(7) optimizing the image restoration inverse sub-problem in the step (1) by using a TFOCS technology to obtain an iteratively updated high-resolution estimated image Xk+1；
(8) Calculating the noise level sigma corresponding to the de-noising subproblem in each iteration processk；
(9) Applying the denoise statistic prior network trained in step (5) to the denoise subproblem in step (1), wherein the noise level corresponds to the sigma in step (8)kAnd further improving the denoising performance by adopting a multiple estimation strategy, and finally constructing an implicit image statistical prior DSP in the denoising subproblem to obtain denoised imageImage vk+1；
(10) Updating the penalty parameter mu in the image restoration inverse sub-problem in the step (1)kAnd iterating equation b by the auxiliary variables of step (1)k+1＝bk+Xk+1-vk+1For auxiliary variable bkUpdating is carried out;
(11) and (5) repeating the steps 7 to 10 until the specified iteration times are reached, and finally outputting the result which is the final super-resolution reconstruction result.
Specifically, in the step (1), the expression of the original single image super-resolution reconstruction problem is as follows:
Figure BDA0002107557470000031
wherein X is an unknown high-resolution map, λ is a regularization parameter,
Figure BDA0002107557470000032
phi (-) is a prior term for the data term.
Decomposing the original super-resolution problem into an image restoration inverse sub-problem, a de-noising sub-problem and an auxiliary variable iteration equation by using an SBI algorithm, wherein the method comprises the following steps:
Figure BDA0002107557470000033
Figure BDA0002107557470000034
bk+1＝bk+Xk+1-vk+1
wherein mu is a penalty parameter, v is an estimated denoised image, k is an iteration number, and b is an auxiliary variable.
The image restoration inverse sub-problem and the denoising sub-problem obtained by using the SBI algorithm are respectively introduced with a priori based on deep learning, namely, an image enhancement prior PEP is added into the original image restoration inverse sub-problem to enhance the high-frequency details of the reconstructed image, and a denoising statistical prior network is used for replacing an explicit denoising prior in the original denoising sub-problem. In the single image super-resolution reconstruction framework provided by the invention, the specific formulas of the image restoration inverse problem and the denoising subproblem are as follows:
Figure BDA0002107557470000041
Figure BDA0002107557470000042
wherein X is an unknown high-resolution image, Y is a low-resolution image, H is a fuzzy matrix, D is a down-sampling matrix, and Xk+1For iteratively updated high resolution estimate maps, vkTo de-noise the image, mukB is a penalty parameter, b is an auxiliary variable, η is a regularization parameter, F is a feature extraction matrix, P is a predicted feature matrix, I is an identity matrix, σ is a normalized feature extraction matrixkFor the noise level during the current iteration,
Figure BDA0002107557470000043
representing the denoising algorithm, the expressions for Z and K are as follows:
Figure BDA0002107557470000044
Figure BDA0002107557470000045
with the increase of the number of iterations, the residual noise is less and less, which indicates that the noise level in the denoising subproblem is continuously attenuated, while the noise level adopted in the iterative solution process of the general denoising subproblem is fixed, which significantly affects the denoising effect. In order to improve the denoising effect, the invention introduces a continuity mechanism to adaptively adjust the noise level sigma corresponding to the denoising subproblem in the iteration processk. Firstly, the methodFor penalty parameter mukUpdating:
μk+1＝ρμk
where ρ is a constant greater than 1. Then according to the relation between the noise level and the penalty parameter:
Figure BDA0002107557470000046
the noise level is updated. λ is the regularization parameter.
In the step (2), unlike the traditional super-resolution method based on deep learning, which directly trains a single network to map a low-resolution image to a high-resolution image, the contour enhancement network constructed in the invention as shown in fig. 2 can predict a plurality of high-resolution direction gradient contour features of an input image, and is further applied to the image restoration inverse sub-problem in the step (1). The Network comprises a Feature Extraction Network module (FENet), a nonlinear Mapping Network (NMNet) module, an Upsampling Network (USNet) module and an Enhanced Reconstruction Network (ERNet) module.
For an input low-resolution image Y, firstly, up-sampling is carried out, then, gradient operators are used for extracting contour features, and then down-sampling and normalization are carried out to obtain the low-resolution contour features after normalization
Figure BDA0002107557470000051
Then the
Figure BDA0002107557470000052
By FENet, the feature B representing the extraction is obtained0：
Figure BDA0002107557470000053
Where ↓and ↓ denote upsampling and downsampling operations, respectively,
Figure BDA0002107557470000054
gradient operators corresponding to the extracted directional profiles, g (x) ═ x/510+0.5, respectively, and represent the gradient values ranging from-255,255]Normalized to [0,1 ]]，
Figure BDA0002107557470000055
A feature extraction function representing a contour enhancement network. Next, B0Obtaining output B after passing through c-th local residual unit through four stacked local residual unitsc：
Figure BDA0002107557470000056
Wherein
Figure BDA0002107557470000057
Representing the c-th local residual function, in the present invention, each local residual unit consists of five CRC (Conv + ReLU + Concat, which is named as a combination of the english initials of the three network layers constituting it) blocks and one convolutional layer (Conv, english abbreviation), where each CRC block consists of one convolutional layer, one excitation layer (which uses the excitation function ReLU) and one cascade layer (Concat, english abbreviation). After the stacked local residual units, feature fusion is performed, which is expressed as:
R＝ffusion(B0，…BL)
ffusionand representing a characteristic fusion function, wherein R is global residual error output, and L is 4 in the invention. Mixing R and B0Adding to obtain output R + B of NMNet0。
Upsampling the output of the NMNet, the expression being as follows:
U＝fdeconv(R+B0)
wherein f isdeconvRepresents the deconvolution layer. And U is subjected to ERNet to obtain final output:
Figure BDA0002107557470000058
wherein f isenhanceRepresenting an enhancement function.
In the step (3), namely the training phase of the contour enhancement network, firstly, the training image data set is subjected to double-triple down sampling to obtain a low-resolution image data set, and then the low-resolution image data set and the original image data set are subjected to directional gradient contour feature extraction by using the same gradient operator to construct a high-resolution and low-resolution training image pair. The invention extracts the gradient characteristics of four directions of 0 degree, 45 degrees, 90 degrees and 135 degrees, and the corresponding four gradient operators are as follows:
Figure BDA0002107557470000059
when a training image time is constructed, firstly, up-sampling is carried out on a low-fraction image, then, contour feature extraction is carried out by using a gradient operator, and then down-sampling and normalization are carried out to obtain a normalized low-resolution contour feature
Figure BDA00021075574700000510
Directly extracting and normalizing the profile features of the high-resolution image to obtain normalized high-resolution profile features
Figure BDA00021075574700000511
XlAnd YlRepresenting a high resolution map and a low resolution map, # represents an upsampling operation, # represents a downsampling operation, # represents a normalization function, and # x/510+0.5, and ranges gradient values from [ -255,255,]normalized to [0,1 ]]。
And (3) then, respectively updating the parameters of the contour enhancement network constructed in the step (2) by using the training set corresponding to each directional gradient contour feature and adopting a minimized two-norm loss function. The training cost function can be expressed by the following formula:
Figure BDA0002107557470000061
wherein N is the number of samples,
Figure BDA00021075574700000613
representing a cost function, ΘpeA trainable set of parameters is represented and,
Figure BDA0002107557470000062
is a global mapping function. The ADAM is adopted to optimize the cost function, and finally the contour enhancement network capable of predicting the four high-resolution direction gradient contour features of the input image is obtained.
In the step (4), unlike the conventional denoising network, the denoising statistical prior network proposed in the present invention as shown in fig. 3 can process a plurality of noise levels (the number of noise levels processed in the present invention is set to 25). Firstly, the noise input image y is subjected to feature extraction to obtain a feature C0：
Figure BDA0002107557470000063
Wherein the content of the first and second substances,
Figure BDA0002107557470000064
a feature extraction function is represented. C0As the initial input to the next 5-layer stacked CBR block (Conv + BatchNormal + ReLU, the name of which is the combination of the english initials of the three network layers that make up it). The block consists of a convolutional layer, a batch normal layer and an excitation layer. The output for the d-th CBR block can be expressed as:
Figure BDA0002107557470000065
wherein
Figure BDA0002107557470000066
Denotes the operation of the d-th CBR Block, CdAnd d represents the output of the d-th CBR block, and the value of d is 1 to 5.
And then de-noising reconstruction is performed. The final output is expressed as:
Figure BDA0002107557470000067
wherein f isreconGlobal residual reconstruction function, C, representing a denoised statistical prior networkCRepresents the output of the c-th CBR block (c 5 in the present invention),
Figure BDA0002107557470000068
representing the de-noised reconstructed image.
In the step (5), namely the training stage of the denoising statistic prior network, each noise level is measured
Figure BDA0002107557470000069
And adding noise to the images in the training image data set to obtain a noise image, so as to construct a training image pair of the original image and the noise image. And (3) respectively updating the parameters of the denoising statistical prior network constructed in the step (4) by using a training set corresponding to each noise level and adopting a minimized two-norm loss function, wherein the loss function expression is as follows:
Figure BDA00021075574700000610
wherein, N is the number of samples,
Figure BDA00021075574700000611
representing a cost function, ΘdsRepresenting a trainable set of parameters, σ is the standard deviation of additive white Gaussian noise, ylAnd vlRepresents a pair of images of the training sample images,
Figure BDA00021075574700000612
is a global mapping function. The ADAM is adopted to optimize the cost function, and finally a denoising statistical prior network capable of processing 25 noise levels is obtained.
In the step (6), the image contour enhancement prior is constructed by using the contour enhancement network trained in the step (3). Performing four-direction gradient profile feature prediction on the input low-resolution image, wherein the m-th direction normalized profile feature prediction is expressed as
Figure BDA0002107557470000071
After contour features in four directions are obtained, a prediction feature matrix P can be constructed, which is defined as:
Figure BDA0002107557470000072
wherein
Figure BDA0002107557470000073
(m is 1,2,3,4) is a predicted high-resolution map contour feature in a certain direction
Figure BDA0002107557470000074
In the form of a matrix.
In the invention, gradient operators in four directions
Figure BDA0002107557470000075
The matrix of (a) is defined as follows:
Figure BDA0002107557470000076
wherein
Figure BDA00021075574700000711
i and j are the two-dimensional coordinates of the matrix elements. The feature extraction matrix F can then be expressed as:
Figure BDA0002107557470000077
according to the contour characteristics of four directions predicted by a network, the invention constructs an explicit image contour enhancement prior and applies the explicit image contour enhancement prior to the image restoration inverse sub-problem in the step (1) as a constraint term, wherein a specific formula of the explicit prior term is as follows:
Figure BDA0002107557470000078
where X is the unknown high resolution image.
In the step (7), the TFOCS technology is used to optimize the inverse sub-problem of the image restoration in the step (1) to obtain the iteratively updated high-resolution estimated image Xk+1。
In the step (8), the noise level sigma corresponding to the de-noising subproblem in each iteration process is calculatedk。
In the step (9), the denoising statistical prior network trained in the step (5) is used to apply to the denoising subproblem in the step (1). Making the noise level of the network correspond to sigma in step (8)kI.e. in obtaining the noise level σ of the current iteration processkThen, selecting one network from the 25 trained denoising statistic prior networks in the step (5), wherein the selection basis is the noise level when the network is trained
Figure BDA0002107557470000079
Noise level sigma of current iteration processkAbsolute value of the difference of
Figure BDA00021075574700000710
And minimum.
And denoising by adopting a multiple estimation strategy. Firstly, a data augmentation strategy is adopted to noise image y1,k+1＝Xk+1+bkRotating by 90 degrees, rotating by 180 degrees, rotating by 270 degrees, performing left-right mirror image turning, rotating by 90 degrees, performing left-right mirror image turning, rotating by 180 degrees, performing left-right mirror image turning, rotating by 270 degrees, performing left-right mirror image turning to obtain 8 amplified images including the original noise image { y1 ,k+1，...，y8,k+1}. Inputting the network to obtain 8 corresponding denoising estimation images:
Figure 1
wherein the content of the first and second substances,
Figure 5
for the estimated nth denoised image, k +1 represents the number of iterations,
Figure BDA0002107557470000083
representing a global mapping equation constructed by a denoising statistical prior network,
Figure BDA0002107557470000084
the network feature extraction layer is represented by a network feature extraction layer,
Figure BDA0002107557470000085
c represents the c CBR block of the network, the value of c is 1 to 5, frecon(. cndot.) represents a network-built global residual reconstruction equation,
Figure BDA0002107557470000086
representing the learning parameters of the network.
Then, fusing 8 denoising images estimated by the network to obtain a final denoising result, and enabling
Figure BDA0002107557470000087
Represent 8 denoised images, { w1,…,w8Denotes a fusion weight. The fused denoising result can be obtained by adopting the estimation based on the uniform weight:
Figure BDA0002107557470000088
wherein v isk+1Namely the fused denoising image.
In the step (10), the penalty parameter mu in the image restoration inverse sub-problem in the step (1) is updatedkAnd iterating equation b by the auxiliary variables of step (1)k+1＝bk+Xk+1-vk+1For auxiliary variable bkAnd (6) updating.
In the step (11), the steps (7) to (10) are repeated until the specified iteration times are reached, and finally the output is the final super-resolution reconstruction result.
To verify the effectiveness of the method of the present invention, the present invention performed experiments with 12 images "Bird", "Butterfly", "Flower", "Leaves", "Plants", "Woman", "Bike", "Comic", "Fish", "Legs", "Yachts", "Zooey" in the standard test image Set 12. The degraded low-resolution image is generated in the following manner: the high resolution test image was blurred with a 7 × 7 gaussian kernel with a standard deviation of 1.5, then 3 times down-sampled, and finally the sampled image was denoised with a gaussian noise with a noise level of 5. And selecting bicubic interpolation and four single image super-resolution algorithms as comparison methods. Wherein the models of the four comparison methods are retrained in accordance with the degradation process of the present invention. The four contrast super-resolution reconstruction algorithms are as follows:
the method comprises the following steps: the methods proposed by Dong et al, references "W.Dong, L.Zhang, G.Shi, and X.Li," non-localized spark prediction for image retrieval, "IEEE trans. image processing, vol.22, No.4, pp.1620-1630, and Apr.2013"
The method 2 comprises the following steps: ren et al, references "C.ren, X.He, andT.Q.Nguyen," Single image super-resolution video adaptive high-dimensional non-local total variation and adaptive geographic feature, "IEEE trans.image Process, vol.26, No.1, pp.90-106, Jan.2017"
The method 3 comprises the following steps: the method proposed by Zhang et al, references "K.Zhang, W.Zuo, S.Gu, and L.Zhang," Learning deep cnn denoiser prior for image restoration, "in Proc.IEEE Conf.Comp.Vis.Pattern recognit., vol.2, Jun.2017, pp. 2808-"
The method 4 comprises the following steps: ren et al, references "C.ren, X.He, andT.Q.Nguyen," Adjusted non-local regression and direct society for image retrieval, "IEEE trans.multimedia, vol.21, No.3, pp.731-745,2019"
The contents of the comparative experiment are as follows:
experiment 1, low-resolution images generated by simulating 12 test images are reconstructed by 3 times by bicubic interpolation, method 1, method 2, method 3 and method 4 respectively. In the experiment, the fuzzy kernel is 7 × 7 of the Gaussian fuzzy kernel size, the standard deviation is 1.5, and the Gaussian noise level is 0. The PSNR (Peak Signal to Noise ratio) and SSIM (Structure Similarity index) parameters of the reconstruction results of the methods are shown in the table I. In addition, the results of the "Butterfly" image are given for visual comparison. The reconstruction results of the "Butterfly" original image, the low-resolution image, and the methods are shown in fig. 5(a), fig. 5(b), fig. 5(c), fig. 5(d), fig. 5(e), fig. 5(f), fig. 5(g), and fig. 5(h), respectively.
Watch 1
Figure BDA0002107557470000101
Experiment 2, low-resolution images generated by simulating 12 test images are reconstructed by 3 times by bicubic interpolation, method 1, method 2, method 3, method 4 and the method of the invention. In the experiment, the fuzzy kernel is 7 × 7 in Gaussian fuzzy kernel size, the standard deviation is 1.5, and the Gaussian noise level is 5. The average PSNR (Peak Signal to Noise ratio) and the average SSIM (Structure Similarity index) parameters of the reconstruction results of the methods are shown in Table II. In addition, for visual comparison, the results of the "Leaves" image are given. The reconstruction results of the "Leaves" original image, the low-resolution image, and the methods are shown in fig. 6(a), 6(b), 6(c), 6(d), 6(e), 6(f), 6(g), and 6(h), respectively.
Watch two
Figure BDA0002107557470000111
As can be seen from the experimental results shown in fig. 5 and fig. 6, the result of the bicubic interpolation method contains more obvious step effect and residual noise, and the visual effect of the image is poor; when no noise interference exists, the four comparison methods can obtain certain resolution improvement, and the detail parts of the methods 3 and 4 are fuzzy. When the image has noise interference, the methods 1,2 and 4 can remove part of noise, but the whole image is fuzzy; the method 3 has better detail effect, but the reconstructed image still has certain blur; in contrast, the result of the invention has no obvious noise, and the image is clearer, the edge is better kept and the visual effect is better. In addition, from the PSNR and SSIM parameters given in table one and table two, the present invention obtains the highest values in both indexes, and the improvement is obvious. Therefore, the subjective visual effect and the objective parameters of the reconstruction results of the methods are comprehensively compared, so that the reconstruction effect of the method is better, and the method is suitable for noise images. In conclusion, the invention is an effective single image super-resolution reconstruction method.

Claims (6)

1. The single image super-resolution method combining contour enhancement and denoising statistics prior is characterized by comprising the following steps of:
the method comprises the following steps: firstly, decomposing the original single-image super-resolution reconstruction problem by using an SBI algorithm to obtain an image restoration inverse sub-problem, a de-noising sub-problem and an auxiliary variable iteration equation; respectively introducing a priori based on deep learning to the obtained image restoration inverse sub-problem and the denoising sub-problem, and introducing a continuity mechanism to obtain a single image super-resolution reconstruction framework based on an improved SBI algorithm;
step two: aiming at an input low-resolution image, constructing a contour enhancement network PENet for predicting unknown high-resolution gradient contour characteristics;
step three: training the network constructed in the second step by using a training image data set, and training gradient profiles in four directions in total;
step four: aiming at the denoising subproblem in the step one, constructing a denoising statistical prior network DSPNet for predicting an unknown denoising image;
step five: training a denoising statistical prior network constructed in the fourth step by utilizing a training image data set, and training a denoising statistical prior network with twenty-five noise levels in total for dynamic denoising of a denoising subproblem in an iterative process;
step six: performing multi-directional gradient contour prediction on an input low-resolution image by using a contour enhancement network trained in the third step to obtain predicted high-resolution image gradient contour characteristics in four directions, constructing an explicit image contour enhancement prior PEP by using the gradient contour characteristics, and applying the prior to the image restoration inverse sub-problem in the first step;
step seven: optimizing the image restoration inverse sub-problem in the step one by using a TFOCS technology to obtain an iteratively updated high-resolution estimated image Xk+1；
Step eight: calculating the noise level sigma corresponding to the de-noising subproblem in each iteration processk；
Step nine: applying the denoising statistical prior network trained in the fifth step to the denoising subproblem in the first step, wherein the noise level corresponds to sigma in the eighth stepkAnd further improving the denoising performance by adopting a multiple estimation strategy, and finally constructing an implicit image statistical prior DSP in the denoising subproblem to obtain a denoised image vk+1；
Step ten: updating the penalty parameter mu in the image restoration inverse sub-problem described in the first stepkAnd iterating equation b by the auxiliary variables of step onek+1＝bk+Xk+1-vk+1For auxiliary variable bkUpdating is carried out;
step eleven: and repeating the seventh step to the tenth step until the specified iteration times are reached, and finally outputting the final super-resolution reconstruction result.
2. The single image super-resolution method combining contour enhancement and denoising statistical prior as claimed in claim 1, wherein the step one single image super-resolution reconstruction framework based on the improved SBI algorithm comprises: firstly, a priori based on deep learning is respectively introduced into an image restoration inverse sub-problem and a denoising sub-problem obtained by using an SBI algorithm, namely, an image enhancement prior PEP is added into an original image restoration inverse sub-problem to enhance high-frequency details of a reconstructed image, a denoising statistical prior network is used for replacing an explicit denoising prior in an original denoising sub-problem, and in the proposed single-image super-resolution reconstruction framework, specific formulas of the image restoration inverse sub-problem and the denoising sub-problem are as follows:
Figure FDA0002107557460000011
Figure FDA0002107557460000012
wherein X is an unknown high-resolution image, Y is a low-resolution image, H is a fuzzy matrix, D is a down-sampling matrix, and Xk+1For iteratively updated high resolution estimate maps, vkTo de-noise the image, mukIs a penalty parameter, k is the iteration number, b is an auxiliary variable, eta is a regularization parameter, F is a feature extraction matrix, P is a predicted feature matrix, I is an identity matrix, sigma iskFor the noise level during the current iteration,
Figure FDA0002107557460000021
representing the denoising algorithm, the expressions for Z and K are as follows:
Figure FDA0002107557460000022
Figure FDA0002107557460000023
then, in order to improve the denoising effect, a continuity mechanism is introduced into the proposed reconstruction framework to adaptively adjust the noise level σ corresponding to the denoising subproblem in the iterative processkFirstly, to punishment parametersSeveral mukUpdating:
μk+1＝ρμk
wherein rho is a constant greater than 1, and then according to the relationship between the noise level and the penalty parameter:
Figure FDA0002107557460000024
and updating the noise level, wherein lambda is a regularization parameter.
3. The single-image super-resolution method combining contour enhancement and denoising statistical prior as claimed in claim 1, wherein the contour enhancement network training four directional gradient contours in step two and step three: firstly, carrying out bicubic downsampling on a training image data set to obtain a low-resolution image data set, then carrying out directional gradient contour feature extraction on the low-resolution image data set and an original image data set by using the same gradient operator to construct a training image pair with high and low resolutions, extracting gradient features in four directions of 0 degrees, 45 degrees, 90 degrees and 135 degrees in a proposed reconstruction frame, wherein the corresponding four gradient operators are as follows:
Figure FDA0002107557460000025
when a training image time is constructed, firstly, up-sampling is carried out on a low-fraction image, then, contour feature extraction is carried out by using a gradient operator, and then down-sampling and normalization are carried out to obtain a normalized low-resolution contour feature
Figure FDA0002107557460000026
Directly extracting and normalizing the profile features of the high-resolution image to obtain normalized high-resolution profile features
Figure FDA0002107557460000027
XlAnd YlRepresenting a high resolution map and a low resolution map, # represents an up-sampling operation, # represents a down-sampling operation, # represents a normalization function, and # x/510+0.5, where the gradient values range from [ -255,255 +0.5]Normalized to [0,1 ]]；
And finally, updating the parameters of the contour enhancement network constructed in the second step by using a minimum two-norm loss function by using a training set corresponding to each directional gradient contour feature, and finally obtaining the contour enhancement network capable of predicting the four high-resolution directional gradient contour features of the input image, wherein the network uses a proposed local residual error unit, and the residual error unit consists of five CRC (Conv + ReLU + Concat) blocks and a convolutional layer (abbreviation Conv), wherein the block is named as a combination of English initials of three network layers forming the block, and each CRC block consists of one convolutional layer, one excitation layer (the excitation function used by the layer is ReLU) and one cascade layer (abbreviation Concat).
4. The single image super-resolution method combining contour enhancement and denoising statistical prior as claimed in claim 1, wherein the training of denoising statistical prior networks for multiple noise levels in step four and step five is performed by: for each noise level
Figure FDA0002107557460000031
And (3) adding noise to the images in the training image data set to obtain a noise image, so as to construct a training image pair of an original image and the noise image, respectively updating parameters of the denoising statistic prior network constructed in the fourth step by using a minimum two-norm loss function by using the training set corresponding to each noise level, and finally obtaining the denoising statistic prior network capable of processing 25 noise levels.
5. The single-image super-resolution method combining contour enhancement and denoising statistical prior according to claim 1, wherein the image contour enhancement prior in step six is constructed by: input low resolution image pair by using four contour enhancement network models trained in step threePerforming gradient profile feature prediction in four directions, wherein the normalized profile feature prediction in the m direction is expressed as
Figure FDA0002107557460000032
After contour features in four directions are obtained, a prediction feature matrix P can be constructed, which is defined as:
Figure FDA0002107557460000033
wherein
Figure FDA0002107557460000034
For high-resolution picture contour features of a certain predicted direction
Figure FDA0002107557460000035
In the form of a matrix;
in the present reconstruction framework, gradient operators in four directions
Figure FDA0002107557460000036
The matrix of (a) is defined as follows:
Figure FDA0002107557460000037
wherein
Figure FDA0002107557460000038
i and j are two-dimensional coordinates of the matrix elements, and the feature extraction matrix F can then be expressed as:
Figure FDA0002107557460000039
constructing an explicit image contour enhancement prior through contour features in four directions predicted by a network, and applying the explicit image contour enhancement prior to the image restoration inverse subproblem in the step one as a constraint term, wherein a specific formula of the explicit prior term is as follows:
Figure FDA00021075574600000310
where X is the unknown high resolution image.
6. The single-image super-resolution method combining contour enhancement and denoising statistical prior according to claim 1, wherein the method using denoising statistical prior network in step nine: obtaining the noise level sigma of the current iteration processkThen, selecting one network from the 25 trained denoising statistical prior networks in the step five according to the noise level when training the network
Figure FDA0002107557460000041
Noise level sigma of current iteration processkAbsolute value of the difference of
Figure FDA0002107557460000042
Minimum;
denoising by adopting a multiple estimation strategy, and firstly, denoising a noise image by adopting a data augmentation strategy
Figure FDA0002107557460000043
Rotating by 90 degrees, rotating by 180 degrees, rotating by 270 degrees, performing left-right mirror image turning, rotating by 90 degrees, performing left-right mirror image turning, rotating by 180 degrees, performing left-right mirror image turning, rotating by 270 degrees, performing left-right mirror image turning to obtain 8 amplified images including the original noise image
Figure FDA0002107557460000044
Inputting the network to obtain 8 corresponding denoising estimation images:
Figure 2
wherein the content of the first and second substances,
Figure 3
for the estimated n-th denoised image, k +1 is the iteration number,
Figure FDA0002107557460000047
representing a global mapping equation constructed by a denoising statistical prior network,
Figure FDA0002107557460000048
the network feature extraction layer is represented by a network feature extraction layer,
Figure FDA0002107557460000049
c represents the c CBR block of the network, the value of c is 1 to 5, frecon(. cndot.) represents a network-built global residual reconstruction equation,
Figure FDA00021075574600000410
learning parameters representing a network;
then, fusing 8 denoising images estimated by the network to obtain a final denoising result, and enabling
Figure FDA00021075574600000411
Represent 8 denoised images, { w1，…，w8The fusion weight is expressed, and the denoising result after fusion can be obtained by adopting the estimation based on the uniform weight:
Figure FDA00021075574600000412
wherein v isk+1Namely the fused denoising image.
CN201910558515.6A 2019-06-26 2019-06-26 Single image super-resolution method combining contour enhancement and denoising statistical prior Active CN112150354B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910558515.6A CN112150354B (en) 2019-06-26 2019-06-26 Single image super-resolution method combining contour enhancement and denoising statistical prior

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910558515.6A CN112150354B (en) 2019-06-26 2019-06-26 Single image super-resolution method combining contour enhancement and denoising statistical prior

Publications (2)

Publication Number Publication Date
CN112150354A true CN112150354A (en) 2020-12-29
CN112150354B CN112150354B (en) 2021-08-24

Family

ID=73868448

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910558515.6A Active CN112150354B (en) 2019-06-26 2019-06-26 Single image super-resolution method combining contour enhancement and denoising statistical prior

Country Status (1)

Country Link
CN (1) CN112150354B (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113298167A (en) * 2021-06-01 2021-08-24 北京思特奇信息技术股份有限公司 Character detection method and system based on lightweight neural network model
CN113392848A (en) * 2021-08-18 2021-09-14 海特锐（天津）科技有限公司 Deep learning-based reading method and device for OCR on cylinder
CN114372927A (en) * 2021-12-21 2022-04-19 南方医科大学 Deep network training method, endoscope image processing method, device and storage medium
CN114723614A (en) * 2021-01-05 2022-07-08 四川大学 Image restoration method based on wavelet domain ADMM depth network
CN114862687A (en) * 2021-02-04 2022-08-05 四川大学 Self-adaptive compressed image restoration method driven by depth deblocking operator

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107977932A (en) * 2017-12-28 2018-05-01 北京工业大学 It is a kind of based on can differentiate attribute constraint generation confrontation network face image super-resolution reconstruction method
US20180225807A1 (en) * 2016-12-28 2018-08-09 Shenzhen China Star Optoelectronics Technology Co., Ltd. Single-frame super-resolution reconstruction method and device based on sparse domain reconstruction
CN108492270A (en) * 2018-03-23 2018-09-04 沈阳理工大学 A kind of super-resolution method reconstructed based on fuzzy kernel estimates and variation
CN108537734A (en) * 2018-04-16 2018-09-14 西华大学 Single image super resolution ratio reconstruction method based on gradient profile example dictionary and Weighted adaptive p norms
CN109064396A (en) * 2018-06-22 2018-12-21 东南大学 A kind of single image super resolution ratio reconstruction method based on depth ingredient learning network
CN109214989A (en) * 2018-09-04 2019-01-15 四川大学 Single image super resolution ratio reconstruction method based on Orientation Features prediction priori
CN109919838A (en) * 2019-01-17 2019-06-21 华南理工大学 The ultrasound image super resolution ratio reconstruction method of contour sharpness is promoted based on attention mechanism

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180225807A1 (en) * 2016-12-28 2018-08-09 Shenzhen China Star Optoelectronics Technology Co., Ltd. Single-frame super-resolution reconstruction method and device based on sparse domain reconstruction
CN107977932A (en) * 2017-12-28 2018-05-01 北京工业大学 It is a kind of based on can differentiate attribute constraint generation confrontation network face image super-resolution reconstruction method
CN108492270A (en) * 2018-03-23 2018-09-04 沈阳理工大学 A kind of super-resolution method reconstructed based on fuzzy kernel estimates and variation
CN108537734A (en) * 2018-04-16 2018-09-14 西华大学 Single image super resolution ratio reconstruction method based on gradient profile example dictionary and Weighted adaptive p norms
CN109064396A (en) * 2018-06-22 2018-12-21 东南大学 A kind of single image super resolution ratio reconstruction method based on depth ingredient learning network
CN109214989A (en) * 2018-09-04 2019-01-15 四川大学 Single image super resolution ratio reconstruction method based on Orientation Features prediction priori
CN109919838A (en) * 2019-01-17 2019-06-21 华南理工大学 The ultrasound image super resolution ratio reconstruction method of contour sharpness is promoted based on attention mechanism

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
CHAO REN ET AL.: "Learning Image Profile Enhancement and Denoising Statistics Priors for Single-Image Super-Resolution", 《IEEE TRANSACTIONS ON CYBERNETICS》 *
龙法宁 等: "基于深层卷积网络的单幅图像超分辨率重建模型", 《广西科学》 *

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114723614A (en) * 2021-01-05 2022-07-08 四川大学 Image restoration method based on wavelet domain ADMM depth network
CN114723614B (en) * 2021-01-05 2023-05-09 四川大学 Image restoration method based on wavelet domain ADMM depth network
CN114862687A (en) * 2021-02-04 2022-08-05 四川大学 Self-adaptive compressed image restoration method driven by depth deblocking operator
CN114862687B (en) * 2021-02-04 2023-05-09 四川大学 Self-adaptive compressed image restoration method driven by depth deblocking operator
CN113298167A (en) * 2021-06-01 2021-08-24 北京思特奇信息技术股份有限公司 Character detection method and system based on lightweight neural network model
CN113392848A (en) * 2021-08-18 2021-09-14 海特锐（天津）科技有限公司 Deep learning-based reading method and device for OCR on cylinder
CN114372927A (en) * 2021-12-21 2022-04-19 南方医科大学 Deep network training method, endoscope image processing method, device and storage medium
CN114372927B (en) * 2021-12-21 2024-03-29 南方医科大学 Endoscopic image processing method, device and storage medium

Also Published As

Publication number Publication date
CN112150354B (en) 2021-08-24

Similar Documents

Publication Publication Date Title
CN112150354B (en) Single image super-resolution method combining contour enhancement and denoising statistical prior
CN112200750B (en) Ultrasonic image denoising model establishing method and ultrasonic image denoising method
Zhang et al. Adaptive residual networks for high-quality image restoration
CN113177882B (en) Single-frame image super-resolution processing method based on diffusion model
Li et al. Example-based image super-resolution with class-specific predictors
CN111127354A (en) Single-image rain removing method based on multi-scale dictionary learning
CN114723630A (en) Image deblurring method and system based on cavity double-residual multi-scale depth network
Liu et al. An efficient residual learning neural network for hyperspectral image superresolution
CN112801914A (en) Two-stage image restoration method based on texture structure perception
CN114820352A (en) Hyperspectral image denoising method and device and storage medium
CN115953784A (en) Laser coding character segmentation method based on residual error and feature blocking attention
Yang et al. An image super-resolution network based on multi-scale convolution fusion
CN113962905A (en) Single image rain removing method based on multi-stage feature complementary network
CN109741258B (en) Image super-resolution method based on reconstruction
CN113096032B (en) Non-uniform blurring removal method based on image region division
CN112200752B (en) Multi-frame image deblurring system and method based on ER network
CN112686817B (en) Image completion method based on uncertainty estimation
Hsu et al. Wavelet pyramid recurrent structure-preserving attention network for single image super-resolution
Tomosada et al. GAN-based image deblurring using DCT discriminator
Suryanarayana et al. Deep Learned Singular Residual Network for Super Resolution Reconstruction.
CN114331853B (en) Single image restoration iteration framework based on target vector updating module
CN115619681A (en) Image reconstruction method based on multi-granularity Vit automatic encoder
CN115082296A (en) Image generation method based on wavelet domain image generation framework
CN114331894A (en) Face image restoration method based on potential feature reconstruction and mask perception
Zhang et al. Deep residual network based medical image reconstruction

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant