CN109544475A - Bi-Level optimization method for image deblurring - Google Patents

Bi-Level optimization method for image deblurring Download PDF

Info

Publication number
CN109544475A
CN109544475A CN201811389239.7A CN201811389239A CN109544475A CN 109544475 A CN109544475 A CN 109544475A CN 201811389239 A CN201811389239 A CN 201811389239A CN 109544475 A CN109544475 A CN 109544475A
Authority
CN
China
Prior art keywords
image
loss
conv
level
loss function
Prior art date
Application number
CN201811389239.7A
Other languages
Chinese (zh)
Inventor
李革
张毅伟
王荣刚
王文敏
高文
Original Assignee
北京大学深圳研究生院
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京大学深圳研究生院 filed Critical 北京大学深圳研究生院
Priority to CN201811389239.7A priority Critical patent/CN109544475A/en
Publication of CN109544475A publication Critical patent/CN109544475A/en

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/001Image restoration
    • G06T5/003Deblurring; Sharpening
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/001Image restoration
    • G06T5/002Denoising; Smoothing
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/001Image restoration
    • G06T5/005Retouching; Inpainting; Scratch removal

Abstract

The present invention relates to the Bi-Level optimization method for image deblurring, design allows two kinds of level loss functions to alternate carry out model optimization, Bi-Level Optimization Mechanism is divided into two steps, the first step loses condition with MSE and trains a basic model, and second step carries out model fine tuning operation using the double-deck grade loss interactive iteration.This is because at training initial stage, divergence between recovery effect and clear image is bigger, the effect of noise can be ignored, and phase noise by continuous amplification keeps its negative effect further obvious after training, therefore introduces perception loss and carry out noise suppressed and MSE loss is changed to L simultaneously1Loss to guarantee structural continuity enough.The present invention has the advantages that reconstruction is accurate, depth characteristic is matched with pixel value.

Description

Bi-Level optimization method for image deblurring

Technical field

The present invention relates to digital image processing fields should particularly for the Bi-Level optimization method of image deblurring Method proposes a kind of Bi-Level optimization method during blur image restoration.

Technical background

Deblurring technology is image and the theme that field of video processing is widely studied.Based on fuzzy caused by camera shake The image quality of image, vision perception are seriously affected in a sense.As one, image preprocessing field and its important Branch, the promotion of deblurring technology directly affect the performance of other computer vision algorithms makes, such as foreground segmentation, object detection, row For analysis etc.;It also affects the coding efficiency of image simultaneously.Therefore, a kind of high performance deblurring algorithm gesture is studied must Row.

Document 1-3 is the background information for the deep learning deblurring algorithm that the present invention compares: document 1:Kupyn O, Budzan V,Mykhailych M,et al.DeblurGAN:Blind Motion Deblurring Using Conditional Adversarial Networks[J].arXiv preprint arXiv:1711.07064,2017.Document 2:Nah S, Kim T H, Lee K M.Deep multi-scale convolutional neural network for dynamic scene deblurring[C]//CVPR.2017,1(2):3.Document 3:Sun J, Cao W, Xu Z, et al.Learning a convolutional neural network for non-uniform motion blur removal[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2015:769-777。

In general, image deblurring algorithm can be divided into traditional algorithm based on probabilistic model and based on deep learning Deblurring algorithm.Traditional algorithm explains that the fuzzy origin cause of formation, the process of camera shake can be mapped as fuzzy core using convolution model Track PSF (Point Spread Function).Clear image is restored in the case where fuzzy core is unknown, this problem belongs to Fixed (ill-posed) problem of discomfort recycles the fuzzy core of assessment to be returned so needing first ambiguous estimation core on ordinary meaning Convolution operation obtains restored image.Deblurring algorithm based on deep learning obtains the potential of image using deep layer network structure Information, and then realize blur image restoration.Fuzzy kernel estimates and non-blind deconvolution may be implemented in the deblurring algorithm of deep learning Two operate to carry out image restoration, while can also be using generation countercheck come restored image.This patent aims to solve the problem that depth Degree study deblurring algorithm there are the shortcomings that:

1) reconstruction inaccuracy,

2) depth characteristic and pixel value mismatch problem.

Summary of the invention

It is an object of the invention to: the Bi-Level optimization method for image deblurring is proposed, the Bi-Level is excellent Change method optimizes GAN (Generative Adversarial Network), it is intended to solve existing deep learning and go Fuzzy algorithmic approach there are the shortcomings that.By comparing existing optimal algorithm, the present invention can go up in image complex pattern originality and averagely improve 1.3dB。

Technical solution provided by the invention is as follows:

MSE loss can guarantee identity of the optimization process in pixel level and feature level, but the problem of bringing is Introduce a large amount of noise;And perceiving loss is a kind of good alternative solution to a certain extent, but not can guarantee optimization Identity.It is not same (non-identical) in order to solve the problems, such as this conditional loss, while the complexity of optimization is reduced, The invention proposes a kind of Bi-Level optimization methods.

Specifically, design allows two kinds of level loss functions to alternate carry out model optimization.In view of L1Loss and MSE Loss meets the same relation, and L1 can introduce more noises but can retain more textures compared to MSE, and perceiving loss has Excessive smoothing effect, the present invention by three kinds of losses while introducing in optimization process.Exist simultaneously in order to balance each conditional loss These losses are normalized to same magnitude according to (formula 4) by the effect in optimization process, the present invention.As shown in Fig. 2, Bi-Level is excellent Change method is divided into two steps, and the first step loses condition with MSE and trains a basic model, and second step is using the double-deck grade loss Interactive iteration carries out model fine tuning operation.This is because at training initial stage, compared with the divergence between recovery effect and clear image Greatly, the effect of noise can be ignored, and phase noise by continuous amplification keeps its negative effect further obvious after training, therefore Perception loss is introduced to carry out noise suppressed while MSE loss is changed to L1Loss to guarantee structural continuity enough.

The present invention has the following technical effect that

1, reconstruction is accurate, and during image restoration, the similitude of image pixel value can guarantee depth characteristic It is similar, on the contrary it is but and invalid;Image detail information repairing failure may cause using the perception loss function of feature level.

2, depth characteristic is matched with pixel value, it is contemplated that MSE, L1 of Pixel-level often will cause noise amplification, and perceive Loss can effectively inhibit noise again, and by the way that the above loss function to be combined, the invention patent both may be used to a certain extent To guarantee that pixel texture is replied accurately, and it can guarantee that depth characteristic is matched with pixel value.

Detailed description of the invention

Fig. 1 generates confrontation flow through a network;

Fig. 2 Bi-Level Optimizing Flow;

Fig. 3 Bi-Skip-Net structure chart;

Fig. 4 Generator Design: Bi-Skip-Net+ residual error;

The subjective comparative of Fig. 5 present invention and other algorithms, in which:

Fig. 5 a is blurred picture comparison diagram;

Fig. 5 b is the recovery effect comparison diagram of Nah et al.;

Fig. 5 c is the recovery effect comparison diagram of Kupyn et al.;

Fig. 5 d is the recovery effect comparison diagram of Bi-Level optimization method.

Specific embodiment

With reference to the accompanying drawings and examples, the present invention is described in detail, but do not limit the invention in any way Range.

Fig. 1 generates confrontation flow through a network, Fig. 4 Generator Design: Bi-Skip-Net+ residual error, and table 1 is arbiter parameter list, As shown,

1. arbiter parameter list of table

# Layer Parameter dimensions Step-length 1 conv 32x3x5x5 2 2 conv 64x32x5x5 1 3 conv 64x64x5x5 2 4 conv 128x64x5x5 1 5 conv 128x128x5x5 4 6 conv 256x128x5x5 1 7 conv 256x256x5x5 4 8 conv 512x256x5x5 1 9 conv 512x512x4x4 4 10 fc 512x1x1x1 -

Specific step is as follows for the Bi-Level optimization method of image deblurring by the present invention:

(1) carry out blur image restoration using generating confrontation flow through a network, designed using Fig. 4 and table 1 generator and Arbiter;

(2) as shown in Figure 1, blurred picture, which is input to generator, obtains restored image;Later, by restored image and clearly Image is input to arbiter and distinguishes to obtain clear image.As shown in figure 4, the present invention is with Bi-Skip-Net come training image Residual error, using blurred picture+Image Residual mode come restored image.

(3) network is trained using following loss function;

WhereinTo fight loss function,For conditional loss function, λ is the weight of conditional loss function.

Pass through maximizationTo optimize arbiter D;

Optimize generator G by minimum formula 3;

WhereinIt designs as follows:

Wherein, L, S respectively indicate model in the output and true value of different levels, and α value is 1 or 2, entire conditional loss letter Number is standardized by port number c, width w and height h.

(4) network is optimized using Li-Level optimization method shown in Fig. 2;

Li-Level optimization method of the invention includes that two steps (set in training process epoch number as N).

Step1: when the number of iterations is less than 1/3N, the present invention is instructed using Pixel-level mean square error (MSE) as loss function Practice model;

Step2: when the number of iterations is more than or equal to 1/3N, the present invention uses the sense of the L1 loss function and feature level of Pixel-level Loss function is known to replace training pattern, and the present invention carries out a loss function replacement every 2 this iteration during the experiment.

(5) using trained network as final restoration model.

Fig. 3 be Bi-Skip-Net structure chart, as shown in figure 3, as shown, Bi-Skip-Net by compressed path (D*), Twin spans connection path (S*) and 3 part of path expander (U*) composition.The depth characteristic and shallow-layer feature of compressed path extraction image; Twin spans connection path connects the up-sampling feature in characteristics of image and path expander;Path expander realizes feature up-sampling.

Fig. 5 is the subjective comparative of the present invention with other algorithms, wherein Fig. 5 a is blurred picture comparison diagram;Fig. 5 b is Nah etc. The recovery effect comparison diagram of people;Fig. 5 c is the recovery effect comparison diagram of Kupyn et al.;Fig. 5 d is answering for Bi-Level optimization method Former effect contrast figure, i.e., recovery effect comparison diagram of the invention.Wherein, Fig. 5 a, Fig. 5 b, Fig. 5 c and Fig. 5 d are three identical Photo especially has chosen two points with two boxes in every picture, and the enlarged drawing of the two boxes is accordingly placed on the picture Lower section, recovery effect can be seen clearly.Its comparing result is shown in Table 2 present invention and test of other algorithms on GoPro data set Comparison.

2. present invention of table and test comparison of other algorithms on GoPro data set

It should be noted that the purpose for publicizing and implementing example is to help to further understand the present invention, but the skill of this field Art personnel, which are understood that, not to be departed from the present invention and spirit and scope of the appended claims, and various substitutions and modifications are all It is possible.Therefore, the present invention should not be limited to embodiment disclosure of that, and the scope of protection of present invention is with claim Subject to the range that book defines.

Claims (4)

1. being used for the Bi-Level optimization method of image deblurring, comprise the concrete steps that:
The first step trains a basic model with MSE loss condition;
Second step carries out model fine tuning operation using double-deck grade loss interactive iteration.
2. the Bi-Level optimization method according to claim 1 for image deblurring, it is characterised in that:
(1) generator described in designs in the following manner: Bi-Skip-Net+ residual error, and arbiter parameter is according to table 1:
1. arbiter parameter list of table
# Layer Parameter dimensions Step-length 1 conv 32x3x5x5 2 2 conv 64x32x5x5 1 3 conv 64x64x5x5 2 4 conv 128x64x5x5 1 5 conv 128x128x5x5 4 6 conv 256x128x5x5 1 7 conv 256x256x5x5 4 8 conv 512x256x5x5 1 9 conv 512x512x4x4 4 10 fc 512x1x1x1 -
(2) with Bi-Skip-Net come training image residual error, using blurred picture+Image Residual mode come restored image;
(3) basic model is trained with MSE loss condition, network is trained using following loss function;
WhereinTo fight loss function,For conditional loss function, λ is the weight of conditional loss function.
Pass through maximizationTo optimize arbiter D;
Optimize generator G by minimum formula 3;
WhereinIt designs as follows:
Wherein, L, S respectively indicate model in the output and true value of different levels, and α value is 1 or 2, entire conditional loss function quilt Port number c, width w and height h are standardized;
(4) model fine tuning operation is carried out using the double-deck grade loss interactive iteration, specifically using Li-Level Optimization Mechanism come pair Network optimizes;
(5) using trained network as final restoration model.
3. the Bi-Level optimization method according to claim 2 for image deblurring, it is characterised in that:
The Bi-Skip-Net is by 3 part group of compressed path (D*), twin spans connection path (S*) and path expander (U*) At;The depth characteristic and shallow-layer feature of compressed path extraction image;Twin spans connection path connects in characteristics of image and path expander Up-sampling feature;Path expander realizes feature up-sampling.
4. the Bi-Level optimization method according to claim 2 for image deblurring, it is characterised in that: step (4) The Li-Level Optimization Mechanism include two steps (set in training process epoch number as N),
Step1: when the number of iterations is less than 1/3N, the present invention is using Pixel-level mean square error (MSE) as loss function training mould Type;
Step2: when the number of iterations is more than or equal to 1/3N, the present invention is using the L1 loss function of Pixel-level and the perception damage of feature level Function is lost to replace training pattern, the present invention carries out a loss function replacement every 2 this iteration during the experiment.
CN201811389239.7A 2018-11-21 2018-11-21 Bi-Level optimization method for image deblurring CN109544475A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201811389239.7A CN109544475A (en) 2018-11-21 2018-11-21 Bi-Level optimization method for image deblurring

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201811389239.7A CN109544475A (en) 2018-11-21 2018-11-21 Bi-Level optimization method for image deblurring
PCT/CN2018/117635 WO2020103171A1 (en) 2018-11-21 2018-11-27 Bi-level optimization method for image deblurring

Publications (1)

Publication Number Publication Date
CN109544475A true CN109544475A (en) 2019-03-29

Family

ID=65848630

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201811389239.7A CN109544475A (en) 2018-11-21 2018-11-21 Bi-Level optimization method for image deblurring

Country Status (2)

Country Link
CN (1) CN109544475A (en)
WO (1) WO2020103171A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110175961A (en) * 2019-05-22 2019-08-27 艾特城信息科技有限公司 A kind of descreening method for dividing confrontation thought based on facial image

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7856150B2 (en) * 2007-04-10 2010-12-21 Arcsoft, Inc. Denoise method on image pyramid
US8866936B2 (en) * 2008-07-24 2014-10-21 Florida State University of Research Foundation Systems and methods for training an active random field for real-time image denoising
CN106157268B (en) * 2016-07-28 2018-11-20 浙江工业大学 One kind being based on the convex approximate degraded image restored method of L0
CN106709875B (en) * 2016-12-30 2020-02-18 北京工业大学 Compressed low-resolution image restoration method based on joint depth network
CN107507141A (en) * 2017-08-07 2017-12-22 清华大学深圳研究生院 A kind of image recovery method based on adaptive residual error neutral net

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110175961A (en) * 2019-05-22 2019-08-27 艾特城信息科技有限公司 A kind of descreening method for dividing confrontation thought based on facial image

Also Published As

Publication number Publication date
WO2020103171A1 (en) 2020-05-28

Similar Documents

Publication Publication Date Title
Xu et al. A trilateral weighted sparse coding scheme for real-world image denoising
Cui et al. Deep network cascade for image super-resolution
US9524450B2 (en) Digital image processing using convolutional neural networks
Chen et al. Weighted couple sparse representation with classified regularization for impulse noise removal
Sun et al. A quantitative analysis of current practices in optical flow estimation and the principles behind them
Wang et al. Esrgan: Enhanced super-resolution generative adversarial networks
Li et al. Single image dehazing via conditional generative adversarial network
Raju et al. A fast and efficient color image enhancement method based on fuzzy-logic and histogram
Liu et al. Progressive image denoising through hybrid graph Laplacian regularization: A unified framework
Xie et al. Joint super resolution and denoising from a single depth image
Li et al. An all-in-one network for dehazing and beyond
Teodoro et al. Image restoration and reconstruction using variable splitting and class-adapted image priors
Guo et al. Adaptive Perona–Malik model based on the variable exponent for image denoising
Tian et al. Image denoising using deep CNN with batch renormalization
Aittala et al. Burst image deblurring using permutation invariant convolutional neural networks
Morillas et al. Fuzzy peer groups for reducing mixed Gaussian-impulse noise from color images
Zhang et al. Wavelet inpainting by nonlocal total variation
Lan et al. Efficient belief propagation with learned higher-order markov random fields
Cai et al. Linearized Bregman iterations for frame-based image deblurring
Buades et al. Nonlocal image and movie denoising
Chan et al. An augmented Lagrangian method for total variation video restoration
Liu et al. Image denoising via adaptive soft-thresholding based on non-local samples
CN106709875A (en) Compressed low-resolution image restoration method based on combined deep network
CN108520503B (en) Face defect image restoration method based on self-encoder and generation countermeasure network
Reinbacher et al. Real-time intensity-image reconstruction for event cameras using manifold regularisation

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination