CN114862687A - Self-adaptive compressed image restoration method driven by depth deblocking operator - Google Patents
Self-adaptive compressed image restoration method driven by depth deblocking operator Download PDFInfo
- Publication number
- CN114862687A CN114862687A CN202110155079.5A CN202110155079A CN114862687A CN 114862687 A CN114862687 A CN 114862687A CN 202110155079 A CN202110155079 A CN 202110155079A CN 114862687 A CN114862687 A CN 114862687A
- Authority
- CN
- China
- Prior art keywords
- deblocking
- image
- operator
- compressed image
- image restoration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 143
- 238000012549 training Methods 0.000 claims abstract description 18
- 238000000605 extraction Methods 0.000 claims abstract description 14
- 230000007246 mechanism Effects 0.000 claims abstract description 13
- 238000011478 gradient descent method Methods 0.000 claims abstract description 5
- 230000006835 compression Effects 0.000 claims description 19
- 238000007906 compression Methods 0.000 claims description 19
- 230000006870 function Effects 0.000 claims description 9
- 230000014509 gene expression Effects 0.000 claims description 9
- 230000003044 adaptive effect Effects 0.000 claims description 7
- 239000011159 matrix material Substances 0.000 claims description 7
- 230000008569 process Effects 0.000 claims description 7
- 238000013507 mapping Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 claims description 4
- 238000013135 deep learning Methods 0.000 claims description 3
- 238000006731 degradation reaction Methods 0.000 claims description 3
- 230000015556 catabolic process Effects 0.000 claims description 2
- 230000002238 attenuated effect Effects 0.000 claims 1
- 238000004364 calculation method Methods 0.000 claims 1
- 238000011084 recovery Methods 0.000 abstract description 7
- 230000000694 effects Effects 0.000 abstract 1
- 238000002474 experimental method Methods 0.000 description 13
- 238000012545 processing Methods 0.000 description 11
- 238000012360 testing method Methods 0.000 description 8
- 238000003909 pattern recognition Methods 0.000 description 7
- 238000013527 convolutional neural network Methods 0.000 description 3
- 239000000126 substance Substances 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 241000220317 Rosa Species 0.000 description 2
- 239000002253 acid Substances 0.000 description 2
- 238000012733 comparative method Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000002243 precursor Substances 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 241000258957 Asteroidea Species 0.000 description 1
- 101100365548 Caenorhabditis elegans set-14 gene Proteins 0.000 description 1
- 241000196324 Embryophyta Species 0.000 description 1
- 101100001677 Emericella variicolor andL gene Proteins 0.000 description 1
- 241000287531 Psittacidae Species 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 239000003638 chemical reducing agent Substances 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 238000002924 energy minimization method Methods 0.000 description 1
- 239000004744 fabric Substances 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 239000003999 initiator Substances 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 238000012797 qualification Methods 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 229940108461 rennet Drugs 0.000 description 1
- 108010058314 rennet Proteins 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000009897 systematic effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G06T5/70—
-
- G06T5/73—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20048—Transform domain processing
- G06T2207/20056—Discrete and fast Fourier transform, [DFT, FFT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Abstract
The invention discloses a self-adaptive compressed image restoration method driven by a depth deblocking operator. The method mainly comprises the following steps: aiming at the maximum posterior problem, a new data fitting item containing a deblocking operator is provided; introducing a continuity mechanism to obtain a compressed image restoration framework based on an improved split Brazilian iteration algorithm; constructing and training an image deblocking network constructed by using multi-level extraction and a per-element tensor attention mechanism; calculating the noise level σ k And the parameter q of the quality factor QF k (ii) a Obtaining a deblocking operator set, and applying the deblocking operator set to the image restoration inverse sub problem; applying a new data fitting item to the recovery inverse sub-problem, and solving by using a gradient descent method or a Fourier transform method to obtain an iteratively updated recovery estimation image; updating the parameters; performing iterative reconstruction, and outputting final imageAnd restoring the result. The compressed image restoration method can obtain good subjective and objective effects. Accordingly, the present invention is a flexible and efficient compressed image restoration method.
Description
Technical Field
The invention relates to a JPEG compressed image restoration technology, in particular to a self-adaptive compressed image restoration method driven by a depth deblocking operator, and belongs to the field of image processing.
Background
The JPEG compressed image restoration technique aims at restoring a high-quality image from a compressed image. In recent years, JPEG compression has been widely used to save bandwidth and memory space. In compression, the image is divided into non-overlapping 8 x 8 blocks and quantized in the DCT domain. Such batching and quantization will lead to undesirable compression artifacts. In order to make better use of image information, it is often necessary to recover a high-quality (HQ) image from a low-quality (LQ) compressed image.
The compressed image restoration method mainly comprises two types: model-based methods and learning-based methods. Model-based approaches can flexibly handle different Image Recovery (IR) tasks, but are computationally complex. The convolutional neural network based on learning performs compressed image restoration by learning the mapping relationship between a large number of compressed and uncompressed image pairs, and the network needs to be trained for each IR task respectively, thereby sacrificing the flexibility of the method. Therefore, it is still a difficult task to propose a universal and efficient framework for various Compressed Image Recovery (CIR) tasks.
Disclosure of Invention
The invention aims to train a set of deblocking operators by using a Convolutional Neural Network (CNN) based on deep learning. Then, the deblocking operators are applied to an image restoration inverse subproblem and a deblocking subproblem in an improved Split Bregman Iteration (SBI) algorithm based on a continuity mechanism, so that a compressed image restoration method based on the improved SBI algorithm is constructed.
The invention provides a self-adaptive compressed image restoration method driven by a depth deblocking operator, which mainly comprises the following operation steps:
(1) firstly, aiming at the problem of Maximum A Posteriori (MAP) of image restoration, a new data fitting item NF containing a deblocking operator is provided;
(2) decomposing the compressed image restoration problem introduced with the new data fitting item by using an SBI algorithm to obtain an image restoration inverse sub-problem, a deblocking sub-problem and an auxiliary variable iteration equation, and introducing a continuity mechanism to obtain a compressed image restoration framework based on the improved SBI algorithm;
(3) aiming at the data fitting items proposed in the step (1), an image deblocking network (MDETA-Net) is constructed by using Multi-level extraction (MD) and an Element-by-Element Tensor Attention mechanism (ETA) and is used for training a group of deblocking operators;
(4) training the network constructed in the step (3) by utilizing a training image data set to obtain a group of deblocking operators, and applying the deblocking operator set to the data fitting item NF in the step (1);
(5) calculating the noise level sigma corresponding to the deblocking subproblem in each iteration process k And the parameter q of the quality factor QF k ；
(6) Applying the set of deblocking operators trained in step (4) to the deblocking subproblems of step (2);
(7) applying the data fitting item NF in the step (4) to the restoration inverse sub-problem in the step (2), and solving by using a gradient descent method or a Fourier transform method to obtain an iteratively updated restoration estimation image X k+1 ；
(8) Updating the penalty parameter gamma in the image restoration inverse sub-problem in the step (2) k And iterating the equation d by the auxiliary variables of step (2) k+1 ＝d k +X k+1 -u k+1 For the auxiliary variable d k+1 Updating is carried out;
(9) and (5) repeating the steps (5) to (8) until the specified number of iterations is reached, and finally outputting the final compression and recovery result.
Drawings
FIG. 1 is a schematic block diagram of the adaptive compressed image restoration method driven by the depth deblocking operator according to the present invention
FIG. 2 is a network architecture diagram of the image deblocking network of the present invention
FIG. 3 is a graph comparing the recovery results of the test image "Butterfly" according to the present invention and nine methods (QF is 10): wherein (a) is a test image, (b) is a JPEG compressed image, (c) (d) (e) (f) (g) (h) (i) (j) (k) (l) are method 1, method 2, method 3, method 4, method 5, method 6, method 7, method 8, method 9, and the reconstruction results of the present invention
FIG. 4 is a graph comparing the results of the restoration of a test image "Img _ 099" according to the present invention with six methods (QF of 20): wherein, (a) is a test image, and (b) (c) (d) (e) (f) (g) (h) (i) are bicubic interpolation, method 1, method 2, method 3, method 4, method 5, method 6, and a reconstruction result of the present invention, respectively
Fig. 5 is a graph comparing the results of the restoration of a test image "Bike" according to the present invention with four methods: wherein (a) is a test image, (b) is a degraded image, and (c) (d) (e) (f) (g) (h) are method 1, method 2, method 3, method 4, method 5, method 6, respectively, and the reconstruction results of the present invention
Fig. 6 is a comparison graph of the reconstruction results of the real low-resolution images "Leaves", "Building", "Boy", "Girl" according to the present invention and four methods: wherein, (1a) is "Rose" low resolution image, and (1b) (1c) (1d) (1e) (1f) are bicubic interpolation, method 1, method 2, method 3, reconstruction result of the present invention, and so on, respectively, for the image.
Detailed Description
The invention will be further described with reference to the accompanying drawings in which:
in fig. 1, the adaptive compressed image restoration method driven by the depth deblocking operator may be specifically divided into the following nine steps:
(1) firstly, aiming at the problem of maximum posterior MAP for image restoration, a new data fitting item NF comprising a deblocking operator is provided;
(2) decomposing the compressed image restoration problem introduced with the new data fitting item by using an SBI algorithm to obtain an image restoration inverse sub-problem, a deblocking sub-problem and an auxiliary variable iteration equation, and introducing a continuity mechanism to obtain a compressed image restoration framework based on the improved SBI algorithm;
(3) aiming at the data fitting items proposed in the step (1), an image deblocking network (MDETA-Net) is constructed by using multi-stage extraction MD and an element-by-element tensor attention mechanism ETA and is used for training a group of deblocking operators;
(4) training the network constructed in the step (3) by utilizing a training image data set to obtain a group of deblocking operators, and applying the deblocking operator set to the data fitting item NF in the step (1);
(5) calculating the noise level sigma corresponding to the deblocking subproblem in each iteration process k And the parameter q of the quality factor QF k ；
(6) Applying the set of deblocking operators trained in step (4) to the deblocking subproblems of step (2);
(7) applying the data fitting item NF in the step (4) to the restoration inverse sub-problem in the step (2), and solving by using a gradient descent method or a Fourier transform method to obtain an iteratively updated restoration estimation image X k+1 ；
(8) Updating the penalty parameter gamma in the image restoration inverse sub-problem in the step (2) k And iterating the equation d by the auxiliary variables of step (2) k+1 ＝d k +X k+1 -u k+1 For the auxiliary variable d k+1 Updating is carried out;
(9) and (5) repeating the steps (5) to (8) until the specified number of iterations is reached, and finally outputting the final compression and recovery result.
Specifically, in step (1), the expression of the image restoration maximum a posteriori MAP problem is as follows:
where X is the unknown high quality HQ image, Y is the low quality LQ image, p (Y | X) is the conditional probability associated with the degradation model, and p (X) is the prior probability of the underlying HQ image.
The maximum a posteriori MAP problem described above is equivalent to the following unconstrained problem:
where X is the unknown uncompressed image, λ is the regularization parameter, H (-) is the data term, and φ (-) is the prior term.
According to the compression operation procedure, the above equation can be written as the following expression:
wherein, C q (. cndot.) is the compression operation, and H is the fuzzy matrix.
For the above equation, a new data fitting term NF is introduced as
Wherein D q (. cndot.) represents a deblocking operator,
an expression more suitable for the CIR problem is obtained:
in the step (2), an SBI algorithm is used to decompose the original compressed image restoration problem into an image restoration inverse sub-problem, a deblocking sub-problem and an auxiliary variable iteration equation, which are respectively as follows:
d k+1 ＝d k +X k+1 -u k+1
wherein gamma is a penalty parameter, u and d are auxiliary variables, and k is the iteration number.
And introducing a data fitting item based on deep learning to the image restoration inverse subproblem and the deblocking subproblem obtained by using the SBI algorithm, namely, replacing the original fitting item with the proposed new data fitting item in the original image restoration inverse subproblem. In the compressed image restoration framework proposed by the present invention, the specific formulas of the image restoration inverse subproblem and the deblocking subproblem are as follows:
where X is an unknown uncompressed image, Y is a compressed image, H is a blur matrix, X is an unknown uncompressed image k+1 Restoring the estimation diagram for the compression after iterative update, wherein gamma is a penalty parameter, u and d are auxiliary variables, I is a unit matrix, and sigma is k For the noise level during the current iteration,
representing the deblocking algorithm, the expressions for Z and K are as follows:
as the number of iterations increases, the residual compression noise will be less and less, indicating that the noise level is continuously decaying in the deblocking subproblem. The invention introduces a continuity mechanism to adaptively adjust the noise level sigma corresponding to the deblocking subproblem in the iteration process k . First, the penalty parameter mu k Updating:
γ k+1 ＝ργ k
where ρ is a constant greater than 1. Then according to the relation between the noise level and the penalty parameter:
the noise level is updated. λ is the regularization parameter.
When sigma is k The greater q k The lower should be and vice versa, thus innovatively constructing the value q of the adaptive noise level and quality factor QF k The relationship between:
q k ＝a/(b+σ k )
to q is k Updating is carried out, wherein a and b are positive constants.
In the step (3), unlike a general compression restoration method using a gaussian operator, the depth deblocker network (MDETA-Net) constructed by the present invention as shown in fig. 2 using a multi-level extraction MD and an element-by-element tensor attention mechanism ETA is used for training a set of deblockers, and is further applied to the new data fitting term NF in the step (1) and the deblocking subproblem in the step (2). The Network comprises an Initial Feature Extraction Network (IFENet) module, a nonlinear Mapping Network (NMNet) module and a Reconstruction Network (rennet) module.
For the input compressed image Y, a decimation layer is designed in the IFENet
First, Y is downsampled to obtain 4 sub-images, i.e.
Then, initial feature extraction is performed by using the convolution layer to obtain a feature F representing the extraction 0 ：
Where ↓ denotes a downsampling operation,
a convolution function representing the initial feature extraction network.
Next, F 0 Is fed into the NMNet. Wherein, NMNet is composed of initial sub NMNet, extraction sub NMNet and final sub NMNet. Assume that L (L ═ 4) ETA residual (ETAR) blocks are stacked to serve as feature maps in each sub-NMNet. The output of the initiator NMNet is:
wherein the content of the first and second substances,
indicating the operation of the initial sub-NMNet.
Then the
Is sent to a convolution layer with step size of 2
To further down sample the features, i.e.
. The subsequent L ETAR blocks are used to implement the non-linear mapping. Next, a convolution extraction layer, i.e. a deconvolution layer with step size of 2, is designed
To upsample the feature, i.e.
The subsequent ETA module is used for element-by-element recalibration. Finally, Global Residual-feature Learning (GRL) is employed to obtain the output of the decimator NMNet:
wherein the content of the first and second substances,
and
the ETA function in the decimator NMNet and the operation in the L ETAR blocks are shown separately.
Next, it will be sent to a stack of L ETAR blocks and an ETA module to obtain the re-calibration features subdivided by elements. Then, GRL is used to obtain the output of the final sub-NMNet:
wherein the content of the first and second substances,
and
the ETA function in the decimator NMNet and the operation in the L ETAR blocks are shown separately.
Finally, the output is sent to obtain HQ image X:
here, the convolution function and the combination operation (inverse operation of the convolution extraction layer) in ReNet are shown, respectively.
In the step (4), namely the training phase of the deep deblocker network, the network is trained by using an LQ-HQ training image pair. The training cost function can be expressed by the following formula:
wherein N is the number of samples, X l And Y l Respectively representing an uncompressed image and a compressed image, L(. to) represents a cost function, Θ represents a trainable set of parameters, f net (. cndot.) is a global mapping function.
In the step (5), the noise level sigma corresponding to the deblocking subproblem in each iteration process is calculated k And the parameter q of the quality factor QF k 。
In the step (6), the set of deblocking devices trained in the step (4) is applied to the deblocking subproblem in the step (2).
In the step (7), the data fitting term NF in the step (4) is applied to the recovered inverse sub-problem in the step (2), and a gradient descent method or a fourier transform method is used for solving, wherein an expression of a gradient descent algorithm is as follows:
X k+1 ＝X k -δ(K T (KX k -Z))
where δ is the step size.
In the compressed image deblurring task, the blur matrix H is cyclic and can be solved using fourier transform method:
where f (-) is a Fourier transform operator,
is the complex conjugate of f (·), and h is the blur kernel.
In the step (8), the penalty parameter γ in the inverse image restoration problem of the step (2) is updated k And iterating the equation d by the auxiliary variables of step (1) k+1 ＝d k +X k+1 -u k+1 For the auxiliary variable d k+1 And (6) updating.
In the step (9), the steps (5) to (8) are repeated until the specified iteration times are reached, and finally the output is the final super-resolution reconstruction result.
To verify the effectiveness of the method of the present invention, three typical CIR tasks were tested, including compressed image deblocking, Super-Resolution (SR), and deblurring. For the deblocking task, four quality factors QF (10, 20, 30, 40) were considered and evaluated using the Classic5, Live1 and Manga109 image datasets. For the compressed image SR, the original image was 2-fold down-sampled by bicubic interpolation before encoding, and four quality factors QF (10, 20, 30, 40) were considered and evaluated using Set5, Set14 and Urban100 image data sets. For compressed image deblurring, the QF is set to 40 using a 25 × 25 gaussian kernel with a standard deviation of 1.6 and the first two motion kernels used in Levin et al, and evaluated using 8 popular test images ("Bike", "Butterfly", "Flower", "Parrot", "Plants", "Starfish", "c.man", "Leaves"), references "a.levin, y.weiss, f.durand, and w.t.freeman," Understanding and evaluating blank resolution algorithms, "in IEEE Conference on video and Pattern Recognition,2009, pp.1964-1971," all results are evaluated by psnr (Signal Noise ratio) and ssim (structure index) index, while JPEG Y and JPEG are only concerned.
The contents of the experiment are as follows:
the method comprises the following steps: liu et al, references "X.Liu, X.Wu, J.Zhou, and D.Zhoo," Data-driven soft decoding of compressed images in dual transform-pixel domain, "IEEE Transactions on Image Processing, vol.25, No.4, pp.1649-1659,2016"
The method 2 comprises the following steps: the methods proposed by ZHao et al, references "C.ZHao, J.ZHang, S.Ma, X.Fan, Y.ZHang, and G.Wen," Re-duration image compression options by structural precision space representation and qualification constraint precursor, "IEEE Transactions on Circuits and Systems for Video Technology, vol.27, No.10, pp.2057-2071,2016"
The method 3 comprises the following steps: the methods proposed by Dong et al, references "C.Dong, Y.Deng, C.C.Loy, and X.Tang," Compression aspects reduction by a default volume network, "in IEEE International Conference on Computer Vision,2015, pp.576-584"
The method 4 comprises the following steps: methods proposed by Chen et al, reference is made to "Y.Chen and T.Pock," transmissible nonlinear interaction differentiation: A flexible frame for fast and effective image retrieval, "IEEE Transactions on Pattern Analysis and Machine Analysis, vol.39, No.6, pp.1256-1272,2017"
The method 5 comprises the following steps: the method proposed by Zhang et al, references "K.Zhang, W.Zuo, Y.Chen, D.Meng, andL.Zhang," Beyondagaus sian denoiser: residual learning of deep cn for Image denosing, "IEEE Transactions on Image processing, vol.26, No.7, pp.3142-3155,2016"
The method 6 comprises the following steps: methods proposed by Mao et al, references "X.J.Mao, C.Shen, and Y.B.Yang," Image differentiating using deep full conditional amino-decoder networks with systematic skip connections, "in Advances in Neural Information Processing Systems,2016, pp.2802-2810"
The method 7 comprises the following steps: methods proposed by Yang et al, references "Y.Tai, J.Yang, X.Liu, and C.xu", "Memnet: A permanent memory network for image retrieval", "in IEEE International Conference on computer Vision,2017, pp.4549-4557"
The method 8 comprises the following steps: the methods proposed by Fu et al, references "X.Fu, Z. -J.ZHa, F.Wu, X.Ding, and J.Paisley," Jpegartficiation reduction video default connected space coding, "in ICCV,2019, pp.2501-2510"
The method 9: the method proposed by Zhang et al, references "Y.Zhang, K.Li, K.Li, B.ZHong, and Y.Fu", "identification a non-local authentication networks for image retrieval", in ICLR,2019 "
Table one shows PSNR and SSIM parameters of the deblocking results of the respective methods. In addition, for visual comparison, the results are given for the "barbarba" image in the Classic5 dataset when QF is 10. The "Barbara" original image, the JPEG compressed image, and the restoration results of the methods are shown in fig. 3(a), 3(b), 3(c), 3(d), 3(e), 3(f), 3(g), 3(h), 3(i), 3(j), 3(k), and 3(l), respectively.
As shown in Table one, the performance of the present invention is superior to other methods in terms of average objective index. As shown in fig. 3, the present invention is more perceptually effective in removing artifacts and reconstructing image details.
Experiment 2, evaluating the processing of the SR task of the JPEG compressed image, six JPEG compressed image super-resolution algorithms are selected as comparison methods. The six comparison deblocking algorithms are:
the method comprises the following steps: he et al, references "T.Li, X.He, L.Qing, Q.Teng, and H.Chen," iterative framework of cascaded deblocking and super-resolution for compressed images, "IEEE Transactions on Multimedia, vol.20, No.6, pp.1305-1320,2018"
The method 2 comprises the following steps: the methods proposed by Zhang et al, references "K.Zhang, W.Zuo, S.Gu, and L.Zhang," Learning deep cnn denoiser prior for image retrieval, "in IEEE Conference on Computer Vision and Pattern Recognition,2017, pp.2808-2817"
The method 3 comprises the following steps: the method proposed by Peleg et al, reference "T.Peleg and M.Elad," A static prediction model based on specific prediction for single Image super-resolution, "IEEE Transactions on Image Processing, vol.23, No.6, pp.2569-2582,2014"
The method 4 comprises the following steps: the method proposed by Kim et al, references "J.Kim, J.K.Lee, and K.M.Lee," Accurate image super-resolution using top sensitivity networks, "in IEEE Conference on Computer Vision and Pattern registration, 2016, pp.1646-1654"
The method 5 comprises the following steps: methods proposed by Yang et al, references "Y.Tai, J.Yang, and X.Liu," Image super-resolution video deep reactive network, "in IEEE Conference on Computer Vision and Pattern Recognition,2017, pp.2790-2798"
The method 6 comprises the following steps: methods proposed by et al, references "H.ZHEN, X.Wang, and X.Gao," Fast and acid single image super-resolution video information distribution network, "in IEEE Conference on Computer Vision and Pattern Recognition,2018, pp.723-731"
Here, since method 3 cannot be directly used to compress the image SR, method 5 described in experiment 1 was used before this method to pre-deblock the input image. Table two shows the SNR and SSIM parameters of the deblocking results of the respective methods. In addition, for visual comparison, the result of an "Img _ 099" image in the Urban100 dataset when QF is 20 is given. The original image, "Img _ 099", the bicubic interpolated image, and the restoration results of the methods are shown in fig. 4(a), 4(b), 4(c), 4(d), 4(e), 4(f), 4(g), 4(h), and 4(i), respectively.
Watch two
As shown in table two and fig. 4, the present invention obtains the highest value in both indices in view of PSNR and SSIM parameters. The results of the present invention are also visually best on subjective comparison.
Experiment 3, evaluating the processing of the present invention to the task of deblurring JPEG compressed images, three common blur kernels as described in experiment 2 were considered. Six JPEG compressed image deblurring algorithms are selected as comparison methods. Where method 1 is a learning-based method and methods 2-6 are model-based methods, since these five model-based methods cannot be directly used for compressed image deblurring, method 5 described in experiment 1 was employed to pre-deblock the input image. The six comparison deblocking algorithms are:
the method comprises the following steps: IRCNN
The method 2 comprises the following steps: the method proposed by Portila et al, reference "J.Portila", "Image reproduction through l0 analysis-based spark option in light frames", "in IEEE International Conference on Image Processing,2010, pp.3909-3912"
The method 3 comprises the following steps: the methods proposed by Dong et al, references "W.Dong, L.Zhang, G.Shi, and X.Wu," Image subtraction and super-resolution by adaptive space domain selection and adaptive resolution, "IEEE Transactions on Image Processing, vol.20, No.7, pp.1838-1857,2011"
The method 4 comprises the following steps: the method proposed by Dong et al, references "W.Dong, L.Zhang, G.Shi, and X.Li," non localized space representation for Image retrieval, "IEEE Transactions on Image Processing, vol.22, No.4, pp.1618-1628,2013"
The method 5 comprises the following steps: the Methods proposed by Sulam et al, references "J.Sulam and M.Elad," extruded Pattern log likelihood with a spark precursor, "in International works work on Energy Minimization method in Computer Vision and Pattern Recognition,2015, pp.99-111"
The method 6 comprises the following steps: the methods proposed by Zhang et al, references "J.Zhang, D.Zhao, and W.Gao," Group-based space representation for Image retrieval, "IEEE Transactions on Image Processing, vol.23, No.8, pp.3336-3351,2014"
Table three shows PSNR and SSIM parameters of the deblocking results of the respective methods. In addition, for the purpose of visual comparison, the deblurring results of the method of the present invention are given after the "Bike" image is blurred by the motion kernel 1. The original image, the blurred image, and the restoration results of the methods are shown in fig. 5(a), 5(b), 5(c), 5(d), 5(e), 5(f), 5(g), 5(h), and 5(i), respectively.
Watch III
As shown by the results, the present invention performed well on objective indices and produced images with sharp edges and fine details.
As shown by the results, it can be found that the network with ETA performs better than the network without ETA. The PSNR/SSIM value is relatively low when MD is deleted from MDETA-Net. After the MD is added, the performance is obviously improved. Indicating that ETA and MD are essential for MDETA-Net performance.
Experiment 4, evaluating the superiority of the higher gaussian operator of the deblocking operator used in the invention and the effectiveness of using a new data fitting item NF in the CIR, three D's were selected 2 A variant of CIR (i.e. the inventive algorithm) was used as a comparison method, and experiments were performed using a compressed image SR with QF 40 as an example. Three comparative methods are as follows:
the method comprises the following steps: CIR driven by noise reducer of IRCNN (DN-CIR)
The method 2 comprises the following steps: general fitting term GF driven D 2 CIR(GF-D 2 CIR)
The method 3 comprises the following steps: d 2 CIR
The results of the experiments are shown in Table four for D for the other variants on the three SR data sets 2 PSNR/SSIM gain distribution of CIR (statistical cloth length of 0.1dB and 0.002, respectively).
Watch four
As shown by the results, the Gaussian operator has the lowest PSNR/SSIM result in dealing with the problem of the compressed image SR. By replacing the Gaussian operator with the deblocker provided by the invention, the performance is obviously improved, and the deblocker has better CIR generalization capability than the Gaussian operator. Without NF, the fitting term would have a large compression noise, and a deblocker with a large compression noise level would have to be used as an implicit prior, resulting in a too smooth result. The performance is obviously improved after the NF is used.
Experiment 6, D proposed for further demonstration 2 Validity of CIR method, experiments were performed using real LQ images. In the experiment, four real LQ images downloaded from the internet, namely "Rose", "Building", "Boy", and "Girl", were used. For color images, because the human eye is more sensitive to the Y channel, the proposed method is only applied to the Y channel and uses a bicubic interpolation method to amplify the Cb and Cr channels. Three representative learning-based comparison methods were selected. All methods use a 2-fold compressed image SR model with QF 40. Three comparative methods are as follows:
the method comprises the following steps: the method proposed by Kim et al, references "J.Kim, J.K.Lee, and K.M.Lee," Accurate image super-resolution using top priority conditional networks, "in IEEE Conference on computer Vision and Pattern registration, 2016, pp.1646-1654"
The method 2 comprises the following steps: methods proposed by Yang et al, references "Y.Tai, J.Yang, and X.Liu," Image super-resolution video deep reactive network, "in IEEE Conference on Computer Vision and Pattern Recognition,2017, pp.2790-2798"
The method 3 comprises the following steps: methods proposed by et al, references "H.ZHEN, X.Wang, and X.Gao," Fast and acid single image super-resolution video information distribution network, "in IEEE Conference on Computer Vision and Pattern Recognition,2018, pp.723-731"
The test original image and the experimental results of each method are shown in fig. 6.
From the results, it can be seen that the present invention can produce HQ images that look more reasonable, restoring sharper edges and finer details, demonstrating the effectiveness of the present invention on actual LQ compressed images.
Claims (5)
1. The self-adaptive compressed image restoration method driven by the depth deblocking operator is characterized by comprising the following steps of:
the method comprises the following steps: firstly, aiming at the problem of maximum posterior MAP for image restoration, a new data fitting item NF comprising a deblocking operator is provided;
step two: decomposing the compressed image restoration problem introduced with the new data fitting item by using a split Brahman iteration SBI algorithm to obtain an image restoration inverse sub-problem, a deblocking sub-problem and an auxiliary variable iteration equation, and introducing a continuity mechanism to obtain a compressed image restoration framework based on an improved SBI algorithm;
step three: aiming at the data fitting item provided in the step one, an effective and simplified image deblocking network MDETA-Net is constructed by using a multi-stage extraction MD and an element-by-element tensor attention mechanism ETA and is used for training a group of deblocking operators;
step four: training the network constructed in the third step by using a training image data set to obtain a group of deblocking operators, and applying the deblocking operator set to the data fitting item NF in the first step;
step five: calculating the noise level sigma corresponding to the deblocking subproblem in each iteration process k And the parameter q of the quality factor QF k ；
Step six: applying the deblocking operator trained in the fourth step to the deblocking subproblem in the second step;
step seven: applying the data fitting item NF in the fourth step to the restoration inverse sub-problem in the second step, and solving by using a gradient descent method or a Fourier transform method to obtain an iteratively updated restoration estimation image X k+1 ；
Step eight: updatingPenalty parameter gamma in the image restoration inverse sub-problem described in the second step k And iterating the equation d by the auxiliary variables of step two k+1 ＝d k +X k+1 -u k+1 For the auxiliary variable d k+1 Updating is carried out;
step nine: and repeating the fifth step to the eighth step until reaching the specified iteration times, and finally outputting the result which is the final compression and restoration result.
2. The method of claim 1, wherein the new data fitting term NF: the innovative introduction of the deblocking operator into the data fitting term NF of the compressed image restoration CIR problem, specifically, the cost function expression in the image restoration IR problem is as follows:
where X is the unknown high quality HQ image, Y is the low quality LQ image, p (Y | X) is the conditional probability associated with the degradation model, and p (X) is the prior probability of the underlying high quality HQ image, the above equation is equivalent to the following unconstrained problem:
where X is an unknown uncompressed image, λ is a regularization parameter, H (-) is a data term, φ (-) is a prior term, the above equation can be written as the following expression, according to the compression operation process:
wherein, C q (. H) is a compression operation and H is a fuzzy matrix, and for the above formula, the deblocking operator is innovatively introduced into the data fitting term NF to obtain a table more suitable for compressing the image and restoring the CIR problemThe expression is as follows:
wherein
Fitting terms NF, D for new data q (. cndot.) represents a deblocking operator,
3. the method for restoring a compressed image driven by a depth deblocking operator according to claim 1, wherein the compressed image restoration framework based on the modified SBI algorithm in step two: the deblocking operator is used as a driving operator for solving the problem of compressed image restoration for the first time, a data fitting item based on deep learning is introduced for the image restoration inverse sub-problem and the deblocking sub-problem obtained by using an SBI algorithm, namely, the data fitting item adopting the deblocking operator is used for replacing the original fitting item in the original image restoration inverse sub-problem, and in the proposed compressed image restoration framework, the specific formulas of the image restoration inverse sub-problem and the deblocking sub-problem are as follows:
where X is an unknown uncompressed image, Y is a compressed image, H is a blur matrix, X is an unknown uncompressed image k+1 Restoring the estimation diagram for the compression after iterative update, wherein gamma is a penalty parameter, u and d are auxiliary variables, I is a unit matrix, and sigma is k For the noise level during the current iteration,
expressions representing deblocking operators, Z and K, are shown below.
4. The method for restoring an adaptively compressed image driven by a depth deblocking operator according to claim 1, wherein said efficient and compact image deblocking network of step three and step four: the network can train a group of deblocking operators by using a multi-stage extraction MD and an element-by-element tensor attention mechanism ETA, and further apply the new data fitting item NF in the step one and the deblocking subproblem in the step two; the network comprises an initial feature extraction network IFENet module, a nonlinear mapping network NMNet module and a reconstruction network ReNet module;
in the training phase, training a network by using a low-quality-high-quality LQ-HQ training image pair; the training cost function can be expressed by the following formula:
wherein N is the number of samples, X l And Y l Representing an uncompressed image and a compressed image, respectively, L (-) represents a cost function, Θ represents a trainable parameter set, f net (. cndot.) is a global mapping function.
5. The method of claim 1, wherein the noise level σ corresponding to the deblocking subproblem is determined by the method of adaptive compressed image reconstruction driven by the depth deblocking operator k And the parameter q of the quality factor QF k The calculation of (2):as the iteration number increases, the residual compression noise is less and less, which indicates that the noise level in the deblocking subproblem is continuously attenuated, and a continuity mechanism is introduced to adaptively adjust the noise level sigma corresponding to the deblocking subproblem in the iteration process k First, for the penalty parameter μ k Updating:
γ k+1 ＝ργ k
wherein rho is a constant greater than 1, and then according to the relationship between the noise level and the penalty parameter:
updating the noise level, wherein lambda is a regularization parameter, and when sigma is k The greater q k The lower should be and vice versa, thus innovatively constructing an adaptive noise level and QF q k Relationship between parameters:
q k ＝a/(b+σ k )
to q is k Updating is carried out, wherein a and b are positive constants.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110155079.5A CN114862687B (en) | 2021-02-04 | 2021-02-04 | Self-adaptive compressed image restoration method driven by depth deblocking operator |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110155079.5A CN114862687B (en) | 2021-02-04 | 2021-02-04 | Self-adaptive compressed image restoration method driven by depth deblocking operator |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114862687A true CN114862687A (en) | 2022-08-05 |
| CN114862687B CN114862687B (en) | 2023-05-09 |
Family
ID=82623468
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110155079.5A Active CN114862687B (en) | 2021-02-04 | 2021-02-04 | Self-adaptive compressed image restoration method driven by depth deblocking operator |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114862687B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107817493A (en) * | 2017-10-25 | 2018-03-20 | 电子科技大学 | A kind of 2D near fields diameter radar image method for reconstructing based on compressed sensing |
| CN108304809A (en) * | 2018-02-06 | 2018-07-20 | 清华大学 | The damaged appraisal procedure of near real-time based on aerial images after shake |
| CN112150354A (en) * | 2019-06-26 | 2020-12-29 | 四川大学 | Single image super-resolution method combining contour enhancement and denoising statistical prior |
| CN112218094A (en) * | 2019-07-11 | 2021-01-12 | 四川大学 | JPEG image decompression effect removing method based on DCT coefficient prediction |
-
2021
- 2021-02-04 CN CN202110155079.5A patent/CN114862687B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107817493A (en) * | 2017-10-25 | 2018-03-20 | 电子科技大学 | A kind of 2D near fields diameter radar image method for reconstructing based on compressed sensing |
| CN108304809A (en) * | 2018-02-06 | 2018-07-20 | 清华大学 | The damaged appraisal procedure of near real-time based on aerial images after shake |
| CN112150354A (en) * | 2019-06-26 | 2020-12-29 | 四川大学 | Single image super-resolution method combining contour enhancement and denoising statistical prior |
| CN112218094A (en) * | 2019-07-11 | 2021-01-12 | 四川大学 | JPEG image decompression effect removing method based on DCT coefficient prediction |
Non-Patent Citations (2)
| Title |
|---|
| 李春;陈静思;王鹏彦;李健;罗泽;: "基于高阶正则与非光滑数据拟合项的图像边缘检测模型" * |
| 肖宿;: "非盲图像复原综述" * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114862687B (en) | 2023-05-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Jin et al. | A flexible deep CNN framework for image restoration | |
| CN110969577B (en) | Video super-resolution reconstruction method based on deep double attention network | |
| CN111028177B (en) | Edge-based deep learning image motion blur removing method | |
| Zhang et al. | Adaptive residual networks for high-quality image restoration | |
| Yu et al. | Deep convolution networks for compression artifacts reduction | |
| Wang et al. | D3: Deep dual-domain based fast restoration of JPEG-compressed images | |
| CN109389552B (en) | Image super-resolution algorithm based on context-dependent multitask deep learning | |
| WO2020015167A1 (en) | Image super-resolution and non-uniform blur removal method based on fusion network | |
| Yu et al. | A unified learning framework for single image super-resolution | |
| CN112801877B (en) | Super-resolution reconstruction method of video frame | |
| CN108900848B (en) | Video quality enhancement method based on self-adaptive separable convolution | |
| CN112801901A (en) | Image deblurring algorithm based on block multi-scale convolution neural network | |
| CN113808032B (en) | Multi-stage progressive image denoising algorithm | |
| CN106709875A (en) | Compressed low-resolution image restoration method based on combined deep network | |
| Zuo et al. | Convolutional neural networks for image denoising and restoration | |
| CN112150354B (en) | Single image super-resolution method combining contour enhancement and denoising statistical prior | |
| Luo et al. | Lattice network for lightweight image restoration | |
| CN116681584A (en) | Multistage diffusion image super-resolution algorithm | |
| CN114723630A (en) | Image deblurring method and system based on cavity double-residual multi-scale depth network | |
| CN112218094A (en) | JPEG image decompression effect removing method based on DCT coefficient prediction | |
| CN108122262B (en) | Sparse representation single-frame image super-resolution reconstruction algorithm based on main structure separation | |
| Xing et al. | Residual swin transformer channel attention network for image demosaicing | |
| CN113962882B (en) | JPEG image compression artifact eliminating method based on controllable pyramid wavelet network | |
| Amaranageswarao et al. | Joint restoration convolutional neural network for low-quality image super resolution | |
| CN113240581A (en) | Real world image super-resolution method for unknown fuzzy kernel |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |