CN104021524A - Image inpainting method based on degradation conversion - Google Patents

Abstract

The invention provides an image inpainting method based on degration conversion, which comprises the steps of: firstly establishing an image inpainting model; then converting the image inpainting model to an element-separatable full-variation model; and finally solving the element-separatable full-variation model for obtaining an inpainted image. According to the image inpainting method based on degeneration conversion, a two-order approximation operator is utilized for converting a double-degeneration model which contains noise and blur in a traditional image inpainting to a single-degeneration model with only dynamic noise, thereby converting a complicated image inpainting problem to a relatively simple denoising problem. Based on a traditional full-variation model, through low-dimension differential projection and a linear operator theory, the element-separatable full-variation model is established, thereby greatly simplifying the traditional full-variation denoising model.

Description

Image repair method based on the conversion of degenerating

Technical field

The invention belongs to technical field of image processing, be specifically related to a kind of image repair method based on the conversion of degenerating.

Background technology

Along with popularizing of digital product, image repair becomes an important research direction.The principal element that causes image deterioration is noise and fuzzy.Due to spatially overlapping of fuzzy and noise, deblurring and denoising normally separately carry out, and implementation method is varied: Richardson-Lucy class methods, the method requires few to the priori of degenerative process, but recovery effects is general, owing to adopting iterative approach thought, consuming time longer; Regularization method, this class methods recovery effects is better, but algorithm is complicated, and regularization parameter is difficult to estimate, often needs to coordinate parameter estimation algorithm, comparatively loaded down with trivial details; Small echo class methods, this type of algorithm be take multiresolution analysis as basis, often needs in conjunction with Wiener or Gabor wave filter, and has " ring " phenomenon, needs to introduce the ancillary method of eliminating " ring "; Total variation norm class methods, these class methods are implemented in balancing energy principle, be combined and can receive extraordinary effect with optimum theory, effective preserving edge detailed information, but fuzzy core is difficult to separated with total variation norm, need to introduce special splitting method, as Split-Bregman and Linearized-Bregman method; Also have some not too conventional methods as neural network.

Summary of the invention

The object of this invention is to provide a kind of image repair method based on the conversion of degenerating, on the basis of traditional total variation model, by low-dimensional difference projection and linear operator theory, set up the separable total variation model of element, thereby greatly simplified traditional total variation denoising model.

The technical solution adopted in the present invention is that the image repair method based on the conversion of degenerating, specifically comprises the following steps:

The first step, sets up inpainting model;

Second step, inpainting model is converted to the separable total variation model of element;

The 3rd step, solves the separable total variation model of element, the image after being repaired.

Feature of the present invention is also,

The first step is specially:

Inpainting model can be expressed as

A*x+w＝b (1)

In formula, matrix x, w and b represent to be respectively of a size of n ₁* n ₂noise-free picture, noise and degraded image, A is fuzzy operator, * represents convolution;

For convenience's sake, formula (1) is written as

Hx+w＝b (2)

In formula, H is the Block-Toeplitz matrix that fuzzy operator A is corresponding;

(2) formula is carried out to Regularization to be obtained:

subject to x＝{x _i,j,0≤x _i,j≤1} (3)

In formula, λ is Lagrange multiplier, represents || x|| _tVweight in min F (x); (3) first of objective function in formula for differentiable convex function, second || x|| _tVfor non-differentiability convex function.

Second step is specially:

If G (a)=f (a)+g (a), f (a) is differentiable convex function, and g (a) is any convex function, and L (f) is the Lipschitz constant of f (a), for have following formula to set up:

$f\left(a_{k-1}\right)+<f^{\&prime;}\left(a_{k-1}\right),a-a_{k-1}>\&le;f\left(a\right)\&le;f\left(a_{k-1}\right)+<f^{\&prime;}\left(a_{k-1}\right),a-a_{k-1}>+\frac{L\left(f\right)}{2}{\left|\right|a-a_{k-1}\left|\right|}_2^2---\left(4\right)$

If the right of inequality is T (a), according to Approximation Operator (proximal operator) theory, work as a _k-1during → a, T (a) is the tight upper bound of f (a), so

$\min G\left(a\right)=f\left(a\right)+g\left(a\right)\stackrel{a_{k-1}\&RightArrow;a}{\&DoubleLeftRightArrow;}\min [T\left(a\right)+g\left(a\right)]---\left(5\right)$

So becoming, solution (5) approaches iterative problem

$a_k=\underset{a}{\arg \min }\{f\left(a_{k-1}\right)+<f^{\&prime;}\left(a_{k-1}\right),a-a_{k-1}>+\frac{L\left(f\right)}{2}{\left|\right|a-a_{k-1}\left|\right|}_2^2+g\left(a\right)\}---\left(6\right)$

(6) formula is merged to abbreviation to be obtained

$a_k=\underset{a}{\arg \min }\{\frac{L\left(f\right)}{2}{\left|\right|a-e_l\left|\right|}_2^2+g\left(a\right)\}---\left(7\right)$

E in formula _k=a _k-1-L ^-1(f) f ' (a _k-1);

For (3) formula, differentiable convex function replace f (a), non-differentiability convex function xTV replaces g (a), obtains the equivalent form of value

$x_k=\underset{x}{\arg \min }\{\frac{L\left(f\right)}{2}{\left|\right|x-d_{k-1}\left|\right|}_F^2+2\mathrm{\&lambda;}{\left|\right|x\left|\right|}_{\mathrm{TV}}\}$

subject to x＝{x _i,j,0≤x _i,j≤1}

In formula

Then, (8) formula being converted to the separable total variation model of element obtains:

The 3rd step specific algorithm flow process is:

Input: fuzzy operator A or corresponding Toeplitz matrix H, Lipschitz constant L (f), image b to be repaired, Lagrange multiplier λ, internal layer number of iterations k, external iteration is counted K+1;

Output: the image x after reparation;

Step1 (initialization): make x=x ₀=0;

Step m (2,3 ..., K): calculate

for j＝1；

$(p_0,q_0)=(0_{(n_1-1)\&times;n_2},0_{n_1\&times;(n_2-1)});$

for j＝2,3,..,k；

for j＝k+1；

$x_{m+1}=P_{B_{\mathrm{0,1}}}(d_m-2\mathrm{\&lambda;}{L}^{-1}\left(f\right)\mathrm{\&psi;}(p_{k+1},q_{k+1}));$

Step K+1: calculate $x=P_{B_{\mathrm{0,1}}}(d_K-2\mathrm{\&lambda;}{L}^{-1}\left(f\right)\mathrm{\&psi;}(p_K,q_K)).$

The invention has the beneficial effects as follows,

1. the present invention is based on the image repair method of the conversion of degenerating, utilize second order Approximation Operator by not only Noise but also be converted into the single degradation model that only has " dynamic noise " containing fuzzy two degradation models in traditional images reparation, thereby complicated image repair problem is converted to a relatively simply Denoising Problems.

2. the present invention is based on the image repair method of the conversion of degenerating, on the basis of traditional total variation model, by low-dimensional difference projection and linear operator theory, set up the separable total variation model of element, thereby greatly simplified traditional total variation denoising model.

3. the present invention is based on the image repair method of the conversion of degenerating, utilize improved total variation model to remove " dynamic noise ", thereby guarantee to complete the image under very noisy+strong fuzzy enviroment.

Accompanying drawing explanation

Fig. 1 is 1024 * 1024 standard lena images;

Fig. 2 is 25 * 25 for adding weak Gaussian noise that variance is 0.001 and mask, the lena image of the strong Gaussian Blur of standard deviation Sigma=8;

The lena image of Fig. 3 for adopting the present invention to recover Fig. 2;

Fig. 4 is face's enlarged drawing of Fig. 3;

Fig. 5 is 25 * 25 for adding medium tenacity Gaussian noise that variance is 0.05 and mask, the lena image of the strong Gaussian Blur of standard deviation Sigma=8;

The lena image of Fig. 6 for adopting the present invention to recover Fig. 5;

Fig. 7 is face's enlarged drawing of Fig. 6;

Fig. 8 is 49 * 49 for add adding Gaussian noise, moving displacement that variance is 0.001, direction of motion is the fuzzy lena image of sharp movement of 10 degree;

The lena image of Fig. 9 for adopting the inventive method to recover Fig. 8;

Figure 10 is face's enlarged drawing of Fig. 9;

Figure 11 is for adding the Gaussian noise that variance is 0.001, the fuzzy lena image of strong average that mask is 50 * 50;

The lena image of Figure 12 for adopting the inventive method to recover Figure 11;

Figure 13 is face's enlarged drawing of Figure 12;

Figure 14 is the PSNR that the image under different masks recovers.

Embodiment

Below in conjunction with the drawings and specific embodiments, the present invention is described in detail.

The image repair method that the present invention is based on the conversion of degenerating, specifically comprises the following steps:

The first step, set up inpainting model:

Be specially:

Inpainting model can be expressed as

A*x+w＝b (1)

In formula, matrix x, w and b represent to be respectively of a size of n ₁* n ₂noise-free picture, noise and degraded image, A is fuzzy operator, * represents convolution;

For convenience's sake, formula (1) is written as

Hx+w＝b (2)

In formula, H is the Block-Toeplitz matrix that fuzzy operator A is corresponding;

(2) formula is carried out to Regularization to be obtained:

subject to x＝{x _i,j,0≤x _i,j≤1} (3)

In formula, λ is Lagrange multiplier, represents || x|| _tVweight in min F (x); (3) first of objective function in formula for differentiable convex function, second || x|| _tVfor non-differentiability convex function;

Second step, inpainting model is converted to the separable total variation model of element:

Be specially:

If G (a)=f (a)+g (a), f (a) is differentiable convex function, and g (a) is any convex function, and L (f) is the Lipschitz constant of f (a), for have following formula to set up:

$f\left(a_{k-1}\right)+<f^{\&prime;}\left(a_{k-1}\right),a-a_{k-1}>\&le;f\left(a\right)\&le;f\left(a_{k-1}\right)+<f^{\&prime;}\left(a_{k-1}\right),a-a_{k-1}>+\frac{L\left(f\right)}{2}{\left|\right|a-a_{k-1}\left|\right|}_2^2---\left(4\right)$

If the right of inequality is T (a), according to Approximation Operator (proximal operator) theory, work as a _k-1during → a, T (a) is the tight upper bound of f (a), so

$\min G\left(a\right)=f\left(a\right)+g\left(a\right)\stackrel{a_{k-1}\&RightArrow;a}{\&DoubleLeftRightArrow;}\min [T\left(a\right)+g\left(a\right)]---\left(5\right)$

So becoming, solution (5) approaches iterative problem

$a_k=\underset{a}{\arg \min }\{f\left(a_{k-1}\right)+<f^{\&prime;}\left(a_{k-1}\right),a-a_{k-1}>+\frac{L\left(f\right)}{2}{\left|\right|a-a_{k-1}\left|\right|}_2^2+g\left(a\right)\}---\left(6\right)$

(6) formula is merged to abbreviation to be obtained

$a_k=\underset{a}{\arg \min }\{\frac{L\left(f\right)}{2}{\left|\right|a-e_l\left|\right|}_2^2+g\left(a\right)\}---\left(7\right)$

E in formula _k=a _k-1-L ^-1(f) f ' (a _k-1);

For (3) formula, differentiable convex function replace f (a), non-differentiability convex function || x|| _tVreplace g (a), obtain the equivalent form of value

$x_k=\underset{x}{\arg \min }\{\frac{L\left(f\right)}{2}{\left|\right|x-d_{k-1}\left|\right|}_F^2+2\mathrm{\&lambda;}{\left|\right|x\left|\right|}_{\mathrm{TV}}\}$

subject to x＝{x _i,j,0≤x _i,j≤1} (8)

In formula

Then, (8) formula being converted to the separable total variation model of element obtains:

The 3rd step, solves the separable total variation model of element:

Specific algorithm flow process is:

Input: fuzzy operator A or corresponding Toeplitz matrix H, Lipschitz constant L (f), image b to be repaired, Lagrange multiplier λ, internal layer number of iterations k, external iteration is counted K+1;

Output: the image x after reparation;

Step1 (initialization): make x=x ₀=0;

Step m (2,3 ..., K): calculate

for j＝1；

$(p_0,q_0)=(0_{(n_1-1)\&times;n_2},0_{n_1\&times;(n_2-1)});$

for j＝2,3,..,k；

for j＝k+1；

$x_{m+1}=P_{B_{\mathrm{0,1}}}(d_m-2\mathrm{\&lambda;}{L}^{-1}\left(f\right)\mathrm{\&psi;}(p_{k+1},q_{k+1}));$

Step K+1: calculate $x=P_{B_{\mathrm{0,1}}}(d_K-2\mathrm{\&lambda;}{L}^{-1}\left(f\right)\mathrm{\&psi;}(p_K,q_K)).$

The present invention is based on the image repair method of the conversion of degenerating, utilize second order Approximation Operator by not only Noise but also be converted into the single degradation model that only has " dynamic noise " containing fuzzy two degradation models in traditional images reparation, thereby complicated image repair problem is converted to a relatively simply Denoising Problems.

The present invention is based on the image repair method of the conversion of degenerating, on the basis of traditional total variation model, by low-dimensional difference projection and linear operator theory, set up the separable total variation model of element, thereby greatly simplified traditional total variation denoising model.

The present invention is based on the image repair method of the conversion of degenerating, utilize improved total variation model to remove " dynamic noise ", thereby guarantee to complete the image under very noisy+strong fuzzy enviroment.

By following emulated data and image, further illustrate advantage of the present invention.

1, simulated conditions

(1) in emulation, adopt 1024 * 1024 standard lena image (as Fig. 1) as simulation object.

(2) to 1024 * 1024 standard lena image, add fuzzy, the mask of every kind of fuzzy employing different size.

2, emulation content

Emulation 1, with the inventive method to adding weak Gaussian noise that variance is 0.001 and mask, be 25 * 25,1024 * 1024 standard lena images (as Fig. 2) of the strong Gaussian Blur of standard deviation Sigma=8 carry out deblurring processing.Result is referring to Fig. 3 and Fig. 4.

Emulation 2, with the inventive method to adding medium tenacity Gaussian noise that variance is 0.05 and mask, be 25 * 25,1024 * 1024 standard lena images (as Fig. 5) of the strong Gaussian Blur of standard deviation Sigma=8 carry out deblurring processing.Result is referring to Fig. 6 and Fig. 7.

Emulation 3, carries out deblurring processing with 1024 * 1024 fuzzy standard lena images (as Fig. 8) of sharp movement that the inventive method is 49 * 49 to add adding Gaussian noise, moving displacement that variance is 0.001, direction of motion is 10 degree.Result is referring to Fig. 9 and Figure 10.

Emulation 4, the Gaussian noise that is 0.001 to interpolation variance by the inventive method, 1024 * 1024 fuzzy standard lena images (as Figure 11) of strong average that mask is 50 * 50 carry out deblurring processing.Result is referring to Figure 12 and Figure 13.

Emulation 5, recovers adding the image of different masks by the inventive method, gets its size and is respectively 4 * 4,9 * 9,16 * 16,25 * 25,36 * 36,49 * 49,64 * 64.Its result is referring to Figure 14.

3, analysis of simulation result

The lena image of Fig. 3 for adopting the present invention to recover Fig. 2; Fig. 4 is face's enlarged drawing of Fig. 3, calculates the PSNR (Y-PSNR) of image after repairing and reaches 31.2dB, and the details such as the face feature of reparation image and hair keep finely, entire image clean cut, and ambiguity removal must be cleaner.

In order to adopt, the inventive method is 25 * 25 to adding medium tenacity Gaussian noise that variance is 0.05 and mask to Fig. 6, the repairing effect of the lena image of the strong Gaussian Blur of standard deviation Sigma=8, face's enlarged drawing that Fig. 7 is Fig. 6.Because noise is stronger, interior loop needs higher iterations.Can find out from 6, the recovery effects of low frequency part is fine; The details such as Tu7Zhong face and hair have obvious fuzzy remnants, and recovery effects is slightly poorer than Fig. 3, but is more or less the same, and illustrates that the inventive method is stronger to " opposing " ability of noise, has stronger anti-noise robustness.The PSNR that repairs image is 30.2dB, does not decline too many than Fig. 3, still belongs to comparatively ideal result.

In order to adopt, the inventive method is 49 to adding Gaussian noise, moving displacement that variance is 0.001 to Fig. 9, direction of motion is the fuzzy restoration result of sharp movement of 10 degree, face's enlarged drawing that Fig. 9 is Fig. 8, and PSNR is 32.5dB.Compare Fig. 3, Fig. 9 and Figure 10 all have the fuzzy striped of obvious residual motions, this is because the Block-Toeplitz matrix of motion blur and the correlativity of Gaussian noise matrix are more weak, be noise more " independence ", need more internal layer iterations could eliminate residual motion blur striped completely.

The Gaussian noise that Figure 12 is 0.001 for employing the inventive method to interpolation variance, the fuzzy restoration result of strong average that mask is 50 * 50, face's enlarged drawing that Figure 13 is Figure 12, PSNR is 31.1dB.The recovery effects at Figure 12 low frequency place is several fuzzy inner best, but detail textures recovery effects is general, just can see the residual fuzzy striped of obvious face from Figure 12 and Figure 13.This is because average is fuzzy own little on the impact at low frequency place, and very large on high frequency treatment impact.

Figure 14 is the PSNR that the image under different masks recovers.In Figure 14, transverse axis is the root of mask size, and the longitudinal axis is PSNR value, is respectively 2,3,4,5,6,7,8, and the longitudinal axis is PSNR value.Can find out, in three kinds of vague category identifiers, the recovery effects of motion blur is best, declines the mildest; When the root of mask size is increased to 6 when above, Gaussian Blur PSNR tends towards stability; The root linear approximate relationship of the PSNR that average is fuzzy and mask size.

Claims (4)

1. the image repair method based on the conversion of degenerating, is characterized in that, specifically comprises the following steps:

The first step, sets up inpainting model;

Second step, inpainting model is converted to the separable total variation model of element;

The 3rd step, solves the separable total variation model of element, the image after being repaired.

2. the image repair method based on the conversion of degenerating according to claim 1, is characterized in that, the first step is specially:

Inpainting model can be expressed as

A*x+w＝b (1)

In formula, matrix x, w and b represent to be respectively of a size of n ₁* n ₂noise-free picture, noise and degraded image, A is fuzzy operator, * represents convolution;

For convenience's sake, formula (1) is written as

Hx+w＝b (2)

In formula, H is the Block-Toeplitz matrix that fuzzy operator A is corresponding;

(2) formula is carried out to Regularization to be obtained:

subject to x＝{x _i,j,0≤x _i,j≤1} (3)

In formula, λ is Lagrange multiplier, represents || _x|| _tVweight in min F (x); (3) first of objective function in formula for differentiable convex function, second || x|| _tVfor non-differentiability convex function.

3. the image repair method based on the conversion of degenerating according to claim 1, is characterized in that, second step is specially:

If G (a)=f (a)+g (a), f (a) is differentiable convex function, and g (a) is any convex function, and L (f) is the Lipschitz constant of f (a), for have following formula to set up:

$f\left(a_{k-1}\right)+<f^{\&prime;}\left(a_{k-1}\right),a-a_{k-1}>\&le;f\left(a\right)\&le;f\left(a_{k-1}\right)+<f^{\&prime;}\left(a_{k-1}\right),a-a_{k-1}>+\frac{L\left(f\right)}{2}{\left|\right|a-a_{k-1}\left|\right|}_2^2---\left(4\right)$

If the right of inequality is T (a), according to Approximation Operator (proximal operator) theory, work as a _k-1during → a, T (a) is the tight upper bound of f (a), so

$\min G\left(a\right)=f\left(a\right)+g\left(a\right)\stackrel{a_{k-1}\&RightArrow;a}{\&DoubleLeftRightArrow;}\min [T\left(a\right)+g\left(a\right)]---\left(5\right)$

So becoming, solution (5) approaches iterative problem

$a_k=\underset{a}{\arg \min }\{f\left(a_{k-1}\right)+<f^{\&prime;}\left(a_{k-1}\right),a-a_{k-1}>+\frac{L\left(f\right)}{2}{\left|\right|a-a_{k-1}\left|\right|}_2^2+g\left(a\right)\}---\left(6\right)$

$a_k=\underset{a}{\arg \min }\{\frac{L\left(f\right)}{2}{\left|\right|a-e_l\left|\right|}_2^2+g\left(a\right)\}---\left(7\right)$

E in formula _k=a _k-1-L ^-1(f) f ' (a _k-1);

For (3) formula, differentiable convex function replace f (a), non-differentiability convex function || x|| _tVreplace g (a), obtain the equivalent form of value

$x_k=\underset{x}{\arg \min }\{\frac{L\left(f\right)}{2}{\left|\right|x-d_{k-1}\left|\right|}_F^2+2\mathrm{\&lambda;}{\left|\right|x\left|\right|}_{\mathrm{TV}}\}$

subject to x＝{x _i，j，0≤x _i，j≤1}

In formula

Then, (8) formula being converted to the separable total variation model of element obtains:

4. the image repair method based on the conversion of degenerating according to claim 1, is characterized in that, the 3rd step specific algorithm flow process is:

Input: fuzzy operator A or corresponding Toeplitz matrix H, Lipschitz constant L (f), image b to be repaired, Lagrange multiplier λ, internal layer number of iterations k, external iteration is counted K+1;

Output: the image x after reparation;

Step1 (initialization): make x=x ₀=0;

Step m (2,3 ..., K): calculate

for j＝1；

$(p_0,q_0)=(0_{(n_1-1)\&times;n_2},0_{n_1\&times;(n_2-1)});$

for j＝2,3,..,k；

for j＝k+1；

$x_{m+1}=P_{B_{\mathrm{0,1}}}(d_m-2\mathrm{\&lambda;}{L}^{-1}\left(f\right)\mathrm{\&psi;}(p_{k+1},q_{k+1}));$

Step K+1: calculate $x=P_{B_{\mathrm{0,1}}}(d_K-2\mathrm{\&lambda;}{L}^{-1}\left(f\right)\mathrm{\&psi;}(p_K,q_K)).$