CN114418883B - Blind image deblurring method based on depth priori - Google Patents
Blind image deblurring method based on depth priori Download PDFInfo
- Publication number
- CN114418883B CN114418883B CN202210052867.6A CN202210052867A CN114418883B CN 114418883 B CN114418883 B CN 114418883B CN 202210052867 A CN202210052867 A CN 202210052867A CN 114418883 B CN114418883 B CN 114418883B
- Authority
- CN
- China
- Prior art keywords
- image
- network
- fuzzy
- kernel
- depth
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 84
- 230000006870 function Effects 0.000 claims abstract description 42
- 238000006731 degradation reaction Methods 0.000 claims description 11
- 238000011478 gradient descent method Methods 0.000 claims description 8
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 7
- 230000015556 catabolic process Effects 0.000 claims description 7
- 238000013507 mapping Methods 0.000 claims description 5
- 238000005457 optimization Methods 0.000 abstract description 11
- 238000012549 training Methods 0.000 abstract description 8
- 238000013178 mathematical model Methods 0.000 abstract description 5
- 238000013528 artificial neural network Methods 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 4
- 238000003384 imaging method Methods 0.000 description 4
- 238000013135 deep learning Methods 0.000 description 3
- 229920006395 saturated elastomer Polymers 0.000 description 3
- 230000003321 amplification Effects 0.000 description 2
- 238000012937 correction Methods 0.000 description 2
- 238000003199 nucleic acid amplification method Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000001575 pathological effect Effects 0.000 description 1
- 238000012887 quadratic function Methods 0.000 description 1
- 238000011158 quantitative evaluation Methods 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/73—Deblurring; Sharpening
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a blind image deblurring method based on depth priori, which uses a depth convolution neural network DIP-Net to implicitly model an image smoothness priori constraint to generate a clear image; estimating a blur kernel by solving an exact solution for the blur kernel optimization problem; and alternately and iteratively updating the fuzzy core and the clear image, calculating a loss function by using the restored clear image and the fuzzy core, and updating network parameters. Jointly modeling the fuzzy image and the fuzzy core, and simultaneously estimating the clear image and the fuzzy core by adopting an alternate iterative network model and a mathematical model; blind deblurring of end-to-end self-supervised learning is achieved with only blurred images without any additional implicit or explicit image priors. The regularization method is realized by combining a depth network structure, and a fuzzy image and a fuzzy kernel true value training network are not required to be used; compared with the traditional model method, the method has the advantages that the fuzzy kernel is estimated from thick to thin in an image pyramid mode, and noise existing in the restored image is effectively restrained.
Description
Technical Field
The invention relates to the field of image deblurring, in particular to a blind image deblurring method based on depth priori.
Background
In the image acquisition process, the acquired image has a certain degree of blurring due to the influence of atmospheric turbulence, relative motion between the imaging device and the target, focusing inaccuracy of the imaging device and other factors. In many fields such as traffic monitoring, biomedical, astronomical observation, remote sensing and telemetry, clear images can provide more useful information. In order to meet the requirements of the respective application fields for clear images, two aspects of hardware and software are generally started. The method for improving the hardware has the problems of high cost, high technical difficulty, easy environmental influence and the like, and the image deblurring technology refers to recovering a clear image from a blurred image by an image processing means from an algorithm level.
Image deblurring estimates a potentially sharp image by modeling the image degradation process, solving the inverse of the image degradation process. Image deblurring methods can be classified into non-blind image deblurring and blind image deblurring, depending on whether the blur kernel is known or not. The blind image deblurring method is to estimate the original clear image and the fuzzy core simultaneously under the condition of unknown degradation model or degradation parameters, has serious pathological condition, and needs to introduce prior information of the image to restrict the space of feasible solution. Existing blind image deblurring methods can be broadly divided into two categories: an optimization model-based method and a deep learning-based method. The blind image deblurring method based on the optimization model carries out mathematical modeling on the statistical characteristics of the natural image. Earlier work has mainly utilized image gradient priors. Since the gradient of a sharp image generally follows the tailing distribution (Heavy-tailed distribution), levin et al model approximately the tailing distribution with a mixed gaussian model. Perrone et al use a projection alternating minimization strategy to solve the deblurred objective function effectively avoids the generation of trivial solutions. Image gradients are relationships between adjacent pixels of an image, while image block priors can represent larger image structures. Michaeli et al estimate the blur kernel using the similarity between cross-scale similar image blocks as a regular constraint term. Pan et al introduce the dark channel a priori into the deblurring objective function in the form of a regularized term. The blind image deblurring method based on the optimized model utilizes a priori designed manual construction image prior model, and is generally difficult to model complex image features. Deep learning (Deep learning) based methods adaptively learn features of a sharp image through a Deep neural network (Deep neural network, DNN) and can be classified into supervised learning and unsupervised learning methods. The blind image deblurring method for supervised learning requires training a network by using paired fuzzy/clear image data sets, and learns the mapping from a fuzzy image to a clear image, and when the characteristic difference between an image to be restored and the training data set is large, the reconstruction quality of the network cannot be ensured. The blind image deblurring method for unsupervised learning does not need a data set training network, and the self-supervised learning is a common unsupervised learning method, uses a blurred image as a supervision signal, and does not need a blur kernel or a clear image true value. The SelfDeblu model proposed by Ren et al utilizes DIP-Net to estimate a clear image, utilizes a fully connected network to estimate a fuzzy core, and simultaneously updates the parameter estimation fuzzy core and the clear image of two networks.
The natural image prior provides effective additional information for image restoration, and restricts the space in which the image restoration problem can be solved. The traditional prior carries out mathematical modeling on the statistical characteristics of the natural image, but the mathematical expression is difficult to express the complex natural image prior. In recent years researchers have proposed modeling image prior information using a depth network, such prior information represented by the depth network being referred to as depth prior. The depth priori adaptively learns the characteristic of the clear image or the mapping relation of the blurred image to the clear image through a depth network, and the priori is not required to be explicitly expressed by a mathematical model. Depth priors can be divided into two categories, explicit and implicit modeling. One type uses a data set training network to learn some potential priori information of an image; another class uses a priori information of the network structure modeling image.
The invention discloses a blind image deblurring algorithm based on depth priori, which combines a network model and a mathematical model, models a clear image and a fuzzy core in a combined way, and restores the clear image end to end by adopting a self-supervision learning method. The invention uses DIP-Net to model image smoothness prior constraint implicitly to estimate clear image, estimates fuzzy core by solving the accurate solution of the minimum problem of fuzzy core, and accelerates model convergence; and calculating a loss function by using the restored clear image and the blur kernel, updating network parameters, and alternately and iteratively estimating the clear image and the blur kernel. Compared with the traditional model method, the method does not need to utilize an image pyramid model to estimate the fuzzy kernel from thick to thin; compared with a blind image deblurring method for supervised learning, the method takes the blurred image as a self-supervision signal, does not need a blur kernel or a clear image truth value, and does not have a training process. Compared with SelfDeblu, the invention solves the accurate solution of the fuzzy core optimization problem, can directly solve the optimal fuzzy core of the current estimated image, accelerates model convergence, and effectively reduces model complexity; meanwhile, the invention directly obtains the fuzzy core of the two-dimensional representation by solving the fuzzy core optimization problem, thereby accelerating network iteration. The method disclosed by the invention can accurately estimate the clear image and the fuzzy core, and effectively inhibit noise existing in the restored image.
Disclosure of Invention
In view of this, the embodiment of the invention provides a blind image deblurring method based on depth priori to restore the original clear image.
In order to achieve the above object, the embodiment of the present invention provides the following solutions:
a blind image deblurring method based on depth priors, comprising the following 4 steps:
step 1, constructing an image generation network model and initializing network parameters
The invention uses the image generation network DIP-Net to realize the mapping x=f (z; θ) of the random vector z to the clear image x, utilizes the network itself to suppress noise, and implicitly models the smoothness constraint prior term. The DIP-Net has a U-shaped encoding and decoding structure and comprises five groups of downsampling and upsampling convolution structures, each group of convolution operation fuses the characteristics of a downsampling layer with the characteristics of upsampling layers with the same corresponding dimension through cross-layer connection, and the number of channels of the cross-layer connection is fixed to be 16. The network input z is a random vector uniformly distributed over the interval (0, 1), i.e. z-U (0, 1), the size of which is consistent with the blurred image, the number of channels is typically set to 8 or 16, and the invention is set to 8.
The parameter setting of the invention comprises a learning rate eta, a network input random vector z, a fuzzy kernel size s and a fuzzy kernel regularization parameter lambda h Maximum number of iterations K. Randomly initializing image generation network parameters θ 0 An initial estimate x of the sharp image can be obtained 0 ＝f(z；θ 0 ) The degradation image is used as a self-supervision signal, and the gradient descent method is utilized to update the parameter theta * The loss function is converged.
Step 2, estimating fuzzy kernel
Fixed network parameter θ k-1 ，x k-1 ＝f(z；θ k-1 ) The fuzzy kernel h is estimated using k ：
In the method, in the process of the invention,representing the fourier transform +.>Representing the complex conjugate of the fourier transform,>representing inverse fourier transform ++>Lambda is the partial derivative of the image in the horizontal and vertical directions h Regularizing parameters for the fuzzy kernel.
Step 3. Estimating a clear image
Step 3.1, calculating a loss function:
fixing the estimate h of the current blur kernel k Given θ k-1 Update θ k . The invention uses DIP-Net to generate clear image, so the loss function of the network is:
the above equation is a mean square error loss function, and other continuous derivative functions can be used as the loss function of the network.
Step 3.2 updating the image generation network parameters:
calculating gradient of loss function with respect to network parameters, and updating theta by gradient descent method k ：
Where η represents a learning rate. The invention uses Adam gradient descent method to update parameters.
Step 3.3, generating a clear image: generating a sharp image x using an updated parameter image generation network k ＝f(z；θ k )。
Step 4, judging convergence, and outputting estimation of fuzzy core and clear image
Through the step 2 and the step 3, one iteration solution to the objective function is completed, and the estimation h of the fuzzy core is obtained k And estimate x of the sharp image k-1 Updated to x k . If the algorithm converges or reaches the maximum iteration number at the moment, stopping iteration, and outputting final fuzzy kernel and clear image estimation; otherwise, let k=k+1, and then repeat steps 2 and 3.
Preferably, the blur kernel regularization parameter λ h The initial value is 2×10 -5 。
Preferably, the number of input data channels in the image generation network is fixed to 8.
Preferably, the learning rate decay factor is 0.5.
The invention discloses a blind image deblurring algorithm based on depth priori, which recovers a clear image end to end. The invention combines the network model and the mathematical model, and jointly models the clear image and the fuzzy core. Estimating a clear image by implicitly modeling an image smoothness prior constraint by using a depth convolutional neural network DIP-Net, and estimating a fuzzy core by solving a precise solution of a fuzzy core minimization problem; and alternately and iteratively updating the fuzzy core and the clear image, calculating a loss function by using the restored clear image and the fuzzy core, and updating network parameters. The invention does not need to utilize an image pyramid model to estimate the fuzzy core from thick to thin, and simultaneously takes the fuzzy image as a self-supervision signal, does not need the fuzzy core or a clear image true value, and does not have a training process. The method disclosed by the invention can accurately estimate the clear image and the fuzzy core, and effectively inhibit noise existing in the restored image.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of blind image deblurring according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a structure of a blind image deblurring method based on depth priors according to an embodiment of the present invention;
FIG. 3 is a flow chart of a blind image deblurring method based on depth priors according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of an image generation network according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of an image generation and network parameter update process according to an embodiment of the present invention;
FIG. 6 is a graph showing a comparison of average PSNR and SSIM for various methods on a Lai dataset provided by an embodiment of the present invention;
FIG. 7 is an average PSNR, SSIM, ER and run-time comparison of various methods over a Levin dataset provided by an embodiment of the present invention;
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In the image deblurring problem, the degradation process of a uniformly blurred image can be expressed as a convolution form as follows:
in the formula of y=h×x+n (1), y is a blurred image, h is a blur kernel, x is a clear image, and n is noise. Under the convolution model, the blind image deblurring method is used for researching how to simultaneously estimate a blur kernel h and a clear image x from a blurred image y, as shown in fig. 1.
The invention uses the image generation network to estimate the clear image, introduces regularization constraint terms of the fuzzy kernel h as the fuzzy kernel priori, and the objective function can be expressed as:
wherein y is a blurred image, f (z; theta) is an image generation network with network input of z and network parameters of theta, h is a blur kernel, lambda h Regularizing parameters for the fuzzy kernel. The former item in the objective function is a data fidelity item, so that the reconstructed image is ensured to accord with a degradation model; the latter term is the regularization constraint term of the fuzzy kernel h, which is usedThe norm constrains the motion blur kernel, so that the motion blur kernel meets convex optimization conditions, can quickly converge to a global optimal solution, and performs sparse processing on the motion blur kernel so as to ensure the sparsity of the motion blur kernel.
Fig. 2 is an overall structure of the blind image deblurring method based on depth priori disclosed by the invention, wherein the invention combines a network model and a mathematical model, and simultaneously estimates a clear image and a blur kernel by alternately solving the optimization problem shown in formula (2). The clear image is estimated by using DIP-Net, the process of calculating the loss function to update the network parameter theta is the process of updating the image, the fuzzy core is estimated according to the optimality condition of the problem of minimizing the fuzzy core, and the clear image and the fuzzy core are alternately and iteratively updated.
The embodiment of the invention discloses a blind image deblurring method based on depth priori so as to restore an original clear image. Referring to fig. 3, the above method includes the following 4 steps.
Step 1, constructing an image generation network model and initializing network parameters
The image generation network DIP-Net implements a mapping of the random vector z to the sharp image x x=f (z; θ). DIP-Net aims at solving the problem of image denoising, the network takes a noisy image as a supervision signal, calculates a loss function to update network parameters, and preferentially generates a clear image without noise in the process of fitting the noisy image by a random vector.
The DIP-Net of the image generation network used in the invention has a U-shaped encoding and decoding structure, the specific structure of which is shown in figure 4, and comprises five groups of downsampling and upsampling convolution structures, each group of convolution operation fuses the characteristics of a downsampling layer with the characteristics of an upsampling layer corresponding to the same dimension through cross-layer connection, and the number of channels of the cross-layer connection is fixed to be 16. The network input z is a random vector uniformly distributed over the interval (0, 1), i.e. z-U (0, 1), the size of which is consistent with the blurred image, the number of channels is typically set to 8 or 16, and the invention is set to 8.SelfDeblu only deblurs the brightness channel of the blurred image, the network outputs a single-channel gray scale image, and image color distortion can occur during color image synthesis. The invention recovers RGB channels of the fuzzy image at the same time, the number of output channels of the model is 3, and the network directly generates the color image without a synthesizing process. The input vector size in the figure is 8×271×271, the feature pattern size is reduced from 271×271 to 9×9 from the first to sixth set of convolution layers, and the number of feature channels per layer is fixed to 128.
The parameter setting of the method comprises the learning rate eta, the network input random vector z, the fuzzy core size s and the fuzzy core regularization parameter lambda h Maximum number of iterations K. Randomly initializing image generation network parameters θ 0 An initial estimate x of the sharp image can be obtained 0 ＝f(z；θ 0 ) The degradation image is used as a self-supervision signal, and the gradient descent method is utilized to update the parameter theta * The loss function is converged.
The invention adopts an alternate solving mode to solve the network parameter theta and the fuzzy kernel h in the (2), namely, firstly, the network parameter theta of the clear image is fixedly estimated k-1 Solving a fuzzy kernel h k The estimate h of the blur kernel is fixed again k Solving and estimating network parameter theta of clear image k Until convergence or a maximum number of iterations is reached.
Step 2, estimating fuzzy kernel
Updating fuzzy cores, i.e. fixed network parameters θ k-1 Estimating a blur kernel h k The optimization problem at this time can be expressed as:
the invention directly solves the accurate solution of the fuzzy kernel h in the frequency domain to accelerate the solving speed of the objective function. Since the periodicity inherent to fourier operations can cause ringing, partial derivatives of the image are used to model data fidelity terms. Let x k-1 ＝f(z；θ k-1 ) Formula (3) can be written as:
in the method, in the process of the invention,for gradient operator->Is the partial derivative of the image in the horizontal and vertical directions. Equation (4) is a quadratic function with respect to h, and there is a closed-form solution. Calculating the derivative of the objective function with respect to h in equation (4) and letting it be 0:
the finishing method can obtain:
where, A is the inversion operation. From the convolution theorem, the convolution of the image in the space domain is equivalent to the frequency domain product of the fourier transform, and the equation (6) is converted into the frequency domain solution:
in the method, in the process of the invention,representing the fourier transform +.>Representing the complex conjugate of the fourier transform. The closed-loop solution of the blur kernel is found according to equation (7):
in the method, in the process of the invention,representing the inverse fourier transform.
Step 3. Estimating a clear image
The non-blind image deblurring problem can be modeled generally as an optimization problem as follows:
wherein y is a blurred image, h is a blur kernel, x is a clear image, R (x) is a smoothness function, lambda x Is a regular term coefficient. The former item in the objective function is a data fidelity item, so that the reconstructed image is ensured to accord with a degradation model; the latter term is a smoothness prior constraint term, and noise amplification is restrained.
DIP-Net generates a network f (z; theta) through an image, noise is suppressed by the network itself, the method is equivalent to the step of implicitly establishing a smoothness constraint prior term R (x) in a mode (9), and a network loss function is as follows:
wherein θ is a network parameter, z is a network input, x * ＝f(z；θ * ) θ for network generated sharp images * Is the optimal network parameter for solving.
DIP-Net is a self-supervision learning method, and image truth and training processes are not needed. It is essentially a regularization method, requiring estimation of network parameters for each image, the parameter update process being in fact the solution of the optimization problem. Since the image restoration process is the inverse of the image degradation process, noise is usually amplified in the inverse process, and the smoothness constraint of DIP-Net can inhibit noise amplification, so that the method can be applied to solving various image inverse problems.
Step 3.1, calculating a loss function:
estimating a sharp image, i.e. fixing the estimate h of the current blur kernel k Given θ k-1 Update θ k At this time, the objective function is reduced to:
the invention uses DIP-Net to generate clear image, and the objective function in the formula (11) is the loss function of the network:
equation (12) is a mean square error loss function, and other continuously-derivable functions may be used as the loss function of the network.
Step 3.2 updating the image generation network parameters:
gradient of the loss function of equation (12) with respect to network parameters, updating θ using gradient descent k ，
Where η represents a learning rate. Fig. 5 shows the process of image generation and network parameter updating. In the image estimation process, given an input vector z and an initial network parameter θ 0 . At the kth iteration, the clear image x generated by the previous time of the network is utilized k-1 ＝f(z；θ k-1 ) Computing a gradient of the loss function with respect to the network parameter with the blurred image y, back-propagating the updated network parameter θ k Based on the currently estimated network parameter θ k Generating a sharp image x k ＝f(z；θ k ) Repeating the above process until the loss function converges or the maximum iteration number is reached.
The invention uses Adam gradient descent method to update network parameters, and Adam algorithm uses momentum v k And second order momentum s in RMSProp algorithm k . To simplify the mathematical expression, let the gradientInitializing v 0 ＝s 0 =0, given the hyper-parameter 0. Ltoreq.β 1 Momentum v of < 1, kth iteration k Expressed as gradient g k-1 Is a exponentially weighted moving average of (2):
v k ＝β 1 v k-1 +(1-β 1 )g k-1 (14)
given super parameter 0.ltoreq.beta 2 ＜1，s k Expressed as the square term g of the gradient k-1 ⊙g k-1 Is a exponentially weighted moving average of (2):
s k ＝β 2 s k-1 +(1-β 2 )g k-1 ⊙g k-1 (15)
in the formula, the term ". Su means multiplication by element. Due to v 0 Sum s 0 All elements in (a) are initialized to zero, and on the kth iteration, the momentum v k Expressed as:
the gradient weights of each previous iteration are added, and, as a result,
when k is small, the sum of the gradient weights for each iteration will be small. To eliminate such effects, for the kth iteration, v will be k Divided byLet the sum of the gradient weights of each iteration in the past be 1, called bias correction. In Adam algorithm, for variable v k Sum s k Deviation correction is carried out:
adam algorithm uses the offset corrected variable v' k And s' k Gradient g 'of learning rate eta update' k-1 ，
Wherein eta is the learning rate, and each element of the independent variable in Adam has different learning rates; e is a constant for avoiding the case where the denominator is 0 in equation (20). Using g 'in the kth iteration' k-1 The network parameters are updated and the network parameters are updated,
θ k ＝θ k-1 -g′ k-1 (21)
step 3.3, generating a clear image: generating a sharp image x using an updated parameter image generation network k ＝f(z；θ k )。
Step 4, judging convergence, and outputting estimation of fuzzy core and clear image
Through the step 2 and the step 3, one iteration solution to the objective function is completed, and the estimation h of the fuzzy core is obtained k And estimate x of the sharp image k-1 Updated to x k . If the algorithm converges or reaches the maximum iteration number at the moment, stopping iteration, and outputting final fuzzy kernel and clear image estimation; otherwise, let k=k+1, and then repeat steps 2 and 3.
Preferably, the fuzzy core regularization parameter lambda is set h The initial value is 2×10 -5 The number of input data channels in the image generation network is fixed to 8, and the learning rate attenuation coefficient is 0.5.
The present invention validates the disclosed methods on Lai data sets and Levin data sets. The simulated blurred image set of the Lai data set comprises 25 clear images and 4 blur kernels with different sizes, and 100 blurred images are generated through convolution operation of the clear images and the blur kernels. Blurred images in the Lai dataset can be divided into five classes, artificial (Manmade), natural (Natural), human (People/Face), saturated (Saturated), and Text (Text), each class containing 20 blurred images. Levin et al image the image printed on the plane, fix the imaging device in the imaging process, manually control the motion state of the camera during exposure, make the acquired image produce motion blur, record the motion trail of the camera at the same time, regard it as blur kernel true value. The Levin dataset includes 4 sharp images and 8 blur kernels for a total of 32 blurred images.
The invention uses peak signal-to-noise ratio PSNR, structural similarity SSIM and error ratio ER as quantitative evaluation indexes. PSNR is equivalent to calculating the mean square error between the restored sharp image and the truth image, SSIM measures the similarity of the restored image and the truth image from three factors of brightness, contrast and structure, and the result is [0,1 ]]In between, the higher the two indices, the better the image reconstruction quality. ER is a fuzzy core evaluation index, and a true fuzzy core h and an estimated fuzzy core h are used for calculation * Ratio of difference between restored image and truth image:
an ER value of 1 when using a true blur kernel for image restoration, a smaller ER value indicates that the estimated blur kernel is closer to the true blur kernel.
The methods such as Michaeli, perrone, levin, pan and Ren are currently widely accepted image deblurring methods, wherein Michaeli, perrone, levin and Pan are all blind deblurring methods based on an optimization model, and image restoration needs to be carried out by combining a non-blind image deblurring method. And Ren and the like do not need to construct an image pyramid, and can directly estimate a fuzzy core and a clear image without resorting to a non-blind deblurring method. Fig. 6 shows the average PSNR and SSIM of the respective algorithms on different classes of images in the Lai simulated image dataset. It can be seen that the average PSNR/SSIM of the method of the invention on Manmade, natural, saturated and Text class images reaches a maximum, and on Peole/Face class images, the average PSNR/SSIM of the method of the invention is higher than Michaeli et al, perrone et al, and Pan et al. Fig. 7 shows the average PSNR, SSIM, ER and running time of various image deblurring algorithms on the Levin dataset, and it can be seen that the average PSNR and SSIM of the method of the present invention are both highest, the average ER is closer to 1, and the algorithm running time is effectively reduced compared to Pan et al and Ren et al.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (4)
1. A blind image deblurring method based on depth priors, comprising the following 4 steps:
step 1, constructing an image generation network model and initializing network parameters
The invention uses the image generation network DIP-Net to realize the mapping x=f (z; theta) from the random vector z to the clear image x, utilizes the network itself to inhibit noise, and implicitly models the smoothness constraint prior term; the DIP-Net has a U-shaped encoding and decoding structure and comprises five groups of downsampling and upsampling convolution structures, each group of convolution operation fuses the characteristics of a downsampling layer with the characteristics of upsampling layers with the same corresponding dimension through cross-layer connection, and the number of channels of the cross-layer connection is fixed to be 16; the network input z is a random vector uniformly distributed on the interval (0, 1), namely z-U (0, 1), the size of the network input z is consistent with that of a blurred image, the number of channels is generally set to be 8 or 16, and the number of channels is set to be 8;
the parameter setting of the invention comprises a learning rate eta, a network input random vector z, a fuzzy kernel size s and a fuzzy kernel regularization parameter lambda h Maximum iteration number K; randomly initializing image generation network parameters θ 0 An initial estimate x of the sharp image can be obtained 0 ＝f(z；θ 0 ) The degradation image is used as a self-supervision signal, and the gradient descent method is utilized to update the parameter theta * Converging the loss function;
step 2, estimating fuzzy kernel
Fixed network parameter θ k-1 ，x k-1 ＝f(z；θ k-1 ) The fuzzy kernel h is estimated using k ：
In the method, in the process of the invention,representing the fourier transform +.>Representing the complex conjugate of the fourier transform,>representing the inverse fourier transform, lambda is the partial derivative of the image in the horizontal and vertical directions h Regularization parameters for the fuzzy kernel;
step 3. Estimating a clear image
Step 3.1, calculating a loss function:
fixing the estimate h of the current blur kernel k Given θ k-1 Update θ k The method comprises the steps of carrying out a first treatment on the surface of the The invention uses DIP-Net to generate clear image, so the loss function of the network is:
the mean square error loss function is adopted, and other continuous derivative functions can be used as the loss function of the network;
step 3.2 updating the image generation network parameters:
calculating gradient of loss function with respect to network parameters, and updating theta by gradient descent method k ：
Wherein η represents a learning rate; the invention uses Adam gradient descent method to update parameters;
step 3.3, generating a clear image: generating a sharp image x using an updated parameter image generation network k ＝f(z；θ k )；
Step 4, judging convergence, and outputting estimation of fuzzy core and clear image
Through the step 2 and the step 3, one iteration solution to the objective function is completed, and the estimation h of the fuzzy core is obtained k And estimate x of the sharp image k-1 Updated to x k The method comprises the steps of carrying out a first treatment on the surface of the Such asIf the algorithm converges or reaches the maximum iteration times at the moment, stopping iteration, and outputting final fuzzy kernel and clear image estimation; otherwise, let k=k+1, and then repeat steps 2 and 3.
2. The depth-prior-based blind image deblurring method of claim 1, wherein the blur kernel regularization parameter λ h The initial value is 2×10 -5 。
3. A depth-a-priori based blind image deblurring method according to claim 1, in which the number of input data channels in the image generation network is fixed at 8.
4. A depth-a-based blind image deblurring method according to claim 1, wherein the learning rate attenuation coefficient is 0.5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210052867.6A CN114418883B (en) | 2022-01-18 | 2022-01-18 | Blind image deblurring method based on depth priori |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210052867.6A CN114418883B (en) | 2022-01-18 | 2022-01-18 | Blind image deblurring method based on depth priori |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114418883A CN114418883A (en) | 2022-04-29 |
| CN114418883B true CN114418883B (en) | 2024-03-29 |
Family
ID=81273046
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210052867.6A Active CN114418883B (en) | 2022-01-18 | 2022-01-18 | Blind image deblurring method based on depth priori |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114418883B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110969582B (en) * | 2019-08-13 | 2023-12-15 | 中国科学院遥感与数字地球研究所 | Remote sensing image blind restoration method based on multi-source reference information |
| CN114967121B (en) * | 2022-05-13 | 2023-02-03 | 哈尔滨工业大学 | Design method of end-to-end single lens imaging system |
| CN114897734B (en) * | 2022-05-18 | 2024-05-28 | 北京化工大学 | Measured target image restoration method based on gradient direction priori |
| CN114662519B (en) * | 2022-05-24 | 2022-09-27 | 武汉朗修科技有限公司 | QR code blind deblurring method based on position detection graph gradient and intensity prior |
| CN115861099B (en) * | 2022-11-24 | 2024-02-13 | 南京信息工程大学 | Satellite cloud image restoration method introducing physical imaging priori knowledge constraint |
| CN116091367B (en) * | 2023-04-10 | 2023-07-18 | 中国科学院空天信息创新研究院 | Blind deblurring method, device, equipment and medium for optical remote sensing image |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108564544A (en) * | 2018-04-11 | 2018-09-21 | 南京邮电大学 | Image Blind deblurring based on edge perception combines sparse optimization method |
| WO2018227882A1 (en) * | 2017-06-15 | 2018-12-20 | 北京大学深圳研究生院 | A priori constraint and outlier suppression based image deblurring method |
| CN109345474A (en) * | 2018-05-22 | 2019-02-15 | 南京信息工程大学 | Image motion based on gradient field and deep learning obscures blind minimizing technology |
| CN110675347A (en) * | 2019-09-30 | 2020-01-10 | 北京工业大学 | Image blind restoration method based on group sparse representation |
| CN111598809A (en) * | 2020-05-18 | 2020-08-28 | 哈尔滨理工大学 | Image motion blur blind removal algorithm based on deep learning |
| CN112085674A (en) * | 2020-08-27 | 2020-12-15 | 河海大学 | Aerial image deblurring algorithm based on neural network |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11748853B2 (en) * | 2020-04-28 | 2023-09-05 | Carnegie Mellon University | Method and architecture for blind image deconvolution |
-
2022
- 2022-01-18 CN CN202210052867.6A patent/CN114418883B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018227882A1 (en) * | 2017-06-15 | 2018-12-20 | 北京大学深圳研究生院 | A priori constraint and outlier suppression based image deblurring method |
| CN108564544A (en) * | 2018-04-11 | 2018-09-21 | 南京邮电大学 | Image Blind deblurring based on edge perception combines sparse optimization method |
| CN109345474A (en) * | 2018-05-22 | 2019-02-15 | 南京信息工程大学 | Image motion based on gradient field and deep learning obscures blind minimizing technology |
| CN110675347A (en) * | 2019-09-30 | 2020-01-10 | 北京工业大学 | Image blind restoration method based on group sparse representation |
| CN111598809A (en) * | 2020-05-18 | 2020-08-28 | 哈尔滨理工大学 | Image motion blur blind removal algorithm based on deep learning |
| CN112085674A (en) * | 2020-08-27 | 2020-12-15 | 河海大学 | Aerial image deblurring algorithm based on neural network |
Non-Patent Citations (3)
| Title |
|---|
| 基于显著性强度和梯度先验的多尺度图像盲去模糊;陈晨;许金鑫;危才华;李庆武;;激光与光电子学进展;20201231(04);271-277 * |
| 基于超拉普拉斯先验与核谱特性噪声图像盲去模糊;余义斌;吴承鑫;彭念;袁仕芳;;工程数学学报;20181215(06);42-48 * |
| 联合深度去噪先验图像盲去模糊;杨爱萍;王金斌;杨炳旺;何宇清;;光学学报;20180524(10);143-151 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114418883A (en) | 2022-04-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114418883B (en) | Blind image deblurring method based on depth priori | |
| Tirer et al. | Image restoration by iterative denoising and backward projections | |
| Schmidt et al. | Discriminative non-blind deblurring | |
| Hawe et al. | Analysis operator learning and its application to image reconstruction | |
| CN106920220B (en) | The turbulent flow method for blindly restoring image optimized based on dark primary and alternating direction multipliers method | |
| CN107025632B (en) | Image super-resolution reconstruction method and system | |
| CN107133923B (en) | Fuzzy image non-blind deblurring method based on adaptive gradient sparse model | |
| CN109003234B (en) | For the fuzzy core calculation method of motion blur image restoration | |
| CN110675347A (en) | Image blind restoration method based on group sparse representation | |
| CN113538374B (en) | Infrared image blurring correction method for high-speed moving object | |
| CN114202459B (en) | Blind image super-resolution method based on depth priori | |
| CN112116541B (en) | Based on gradient l0Fuzzy image restoration method based on norms and total variation regularization constraint | |
| CN112184549B (en) | Super-resolution image reconstruction method based on space-time transformation technology | |
| Xiao et al. | Discriminative transfer learning for general image restoration | |
| CN108629741B (en) | Fuzzy kernel estimation method based on L0 and L1 regular terms | |
| Wen et al. | The power of complementary regularizers: Image recovery via transform learning and low-rank modeling | |
| CN112184567A (en) | Multi-channel blind identification adaptive optical image restoration method based on alternate minimization | |
| CN116862795A (en) | Multistage motion blur removing method based on pixel-by-pixel degradation prediction network | |
| CN116563138A (en) | Image sequence denoising method and device based on time-space domain total variation regularization | |
| CN111008930B (en) | Fabric image super-resolution reconstruction method | |
| CN110223238B (en) | Encoding illumination imaging reconstruction method and device | |
| CN108681988B (en) | Robust image resolution enhancement method based on multiple images | |
| Tanaka et al. | Locally adaptive learning for translation-variant MRF image priors | |
| Rezayi et al. | Huber Markov random field for joint super resolution | |
| CN112163186A (en) | Plug-and-play large-scale efficient robust phase recovery method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |