CN105260995B - An image inpainting and denoising method and system - Google Patents

Abstract

The invention discloses an image repairing and denoising method and system. For a given original training image, the method includes: by introducing the idea of joint low-rank and sparse matrix decomposition, and using convex optimization technology, the given training sample The image data matrix is decomposed into a joint low-rank and sparse principal component feature encoding matrix and a sparse error matrix, and the joint low-rank and sparse principal component features of the training image sample data are determined according to the low-rank and sparse characteristics of the training image sample data. and an error matrix, so as to perform repairing and denoising processing on the original image that may contain errors, and obtain an image after repairing and denoising. The image repairing and denoising method and system provided by the present invention fully consider the robust low rank and sparse characteristics of the data while characterizing the image data, so as to overcome the deficiencies of the prior art and improve the image repairing and denoising. noise performance and model robustness.

Description

An image inpainting and denoising method and system

technical field

The present invention relates to the technical field of computer vision and image processing, in particular to an image restoration and denoising method and system.

Background technique

In a large number of practical applications, real data can be described by high-dimensional attributes or features, such as visual images, but high-dimensional data often contain a lot of redundant information or noise. Therefore, how to perform effective image restoration and how to effectively describe images through feature learning or low-rank, sparse coding techniques has attracted extensive attention in recent years.

Feature extraction aims to obtain descriptive and compact features through mapping or transformation methods, and realize transformation from high-dimensional to low-dimensional. PCA (Principal Component Analysis) is one of the most representative unsupervised feature learning models. The specific operation is: for a given data matrix (where n is the sample dimension and N is the number of samples), PCA optimizes a potential projection matrix by maximizing the covariance structure of the data

Among them, I is an identity matrix, d represents the dimension to be reduced to, || · || ₂ is the l ² norm, is the average of all samples. PCA can effectively reveal the linear relationship between the data, but the PCA model based on the L2 norm has been proved to be very sensitive to noisy data, outliers or pixel corruption, so it may not actually reveal the subspace structure of the data accurately.

In order to make up for the shortcomings of PCA and enhance the robustness of the model to noisy data and erroneous data, some robust principal component feature extraction models have been proposed in recent years, among which RPCA (Robust Principal Component Analysis) is a relatively novel and effective method. . RPCA technology performs data repair and feature extraction through the following kernel norm minimization problem:

<L,E>=arg min _L,E ||L|| _* +γ||E|| ₁ ,stX=L+E

Among them, γ>0 is a trade-off parameter, E=XL represents sparse error data, and the optimal solution L ^* of the above problem is the "lowest rank description" of the original data, which also corresponds to the best principal component feature of the original data.

When the degree of data error is low, RPCA can effectively restore the original data. However, the RPCA technology only considers the low-rank robustness of the data, and does not consider the sparse robustness in the data description process. Therefore, the extraction of principal component features may have a certain negative impact on the results of image inpainting.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a method and system for image inpainting and denoising, and the purpose is to solve the problem that the robust low-rank and sparse characteristics of data are not considered in the prior art at the same time.

In order to solve the above technical problems, the present invention provides an image restoration and denoising method, including:

Preprocess the training image sample data and initialize the model parameters;

By introducing the idea of joint low-rank and sparse matrix decomposition, and using convex optimization technology, the given training image sample data matrix is decomposed into joint low-rank and sparse principal component feature encoding matrix and sparse error matrix;

Repair and denoise the original image to get the repaired and denoised image.

Optionally, by introducing the idea of joint low-rank and sparse matrix decomposition, and using convex optimization technology, the given training image sample data matrix is decomposed into joint low-rank and sparse principal component feature encoding matrix and sparse error matrix including:

for vector sets Decompose the data matrix X into a joint low-rank and sparse principal component feature encoding matrix L ^S and a sparse error matrix E:

where λ>0 is the trade-off parameter, α∈[0,1] is the trade-off parameter between low-rank and sparse coding terms, ||·|| _* is the kernel norm of the matrix, and ||·|| ₁ is l ¹ norm, The optimal solution L ^S* obtained by optimization is the joint low-rank and sparse principal component feature coding matrix of the training image sample data, x _i is a sample in the training image sample data, n is the dimension of the image sample, N is the total number of image samples.

Optionally, by introducing the idea of joint low-rank and sparse matrix decomposition, and using convex optimization technology, the given training image sample data matrix is decomposed into joint low-rank and sparse principal component feature encoding matrix and sparse error matrix including:

Decompose the data matrix X into a joint low-rank and sparse principal component feature encoding matrix L ^S and a sparse error matrix E:

Define the augmented Lagrangian function:

Among them, Y ₁ , Y ₂ , and Y ₃ are Lagrangian multipliers, and μ is a weight parameter. Through the augmented Lagrangian function to alternately update the variable:

By iterative optimization of the convex subproblem, the solution is completed by sequentially updating the variable values:

J is solved by a singular value contraction operation, and F and E are solved by a scalar contraction operation.

Optionally, after obtaining the restored and denoised image, the method further includes:

The image restoration effect is evaluated by visual observation and index quantification of the restored and denoised images.

The present invention also provides an image restoration and denoising system, comprising:

The preprocessing module is used to preprocess the training image sample data and initialize the model parameters;

The decomposition module is used to decompose a given training image sample data matrix into a joint low-rank and sparse principal component feature encoding matrix and a sparse error matrix by introducing the idea of joint low-rank and sparse matrix decomposition and using convex optimization technology;

The repair module is used to repair and denoise the original image, and obtain the repaired and denoised image.

Optionally, the decomposition module is specifically used for:

for vector sets Decompose the data matrix X into a joint low-rank and sparse principal component feature encoding matrix L ^S and a sparse error matrix E:

where λ>0 is the trade-off parameter, α∈[0,1] is the trade-off parameter between low-rank and sparse coding terms, ||·|| _* is the kernel norm of the matrix, and ||·|| ₁ is l ¹ norm, The optimal solution L ^S* obtained by optimization is the joint low-rank and sparse principal component feature coding matrix of the training image sample data, x _i is a sample in the training image sample data, n is the dimension of the image sample, N is the total number of image samples.

Optionally, the decomposition module is specifically used for:

Decompose the data matrix X into a joint low-rank and sparse principal component feature encoding matrix L ^S and a sparse error matrix E:

Define the augmented Lagrangian function:

Among them, Y ₁ , Y ₂ , and Y ₃ are Lagrangian multipliers, and μ is a weight parameter. Through the augmented Lagrangian function to alternately update the variable:

By iterative optimization of the convex subproblem, the solution is completed by sequentially updating the variable values:

J is solved by a singular value contraction operation, and F and E are solved by a scalar contraction operation.

Optionally, also include:

The evaluation module is used to evaluate the restoration effect of the image by visual observation and index quantification of the restored and denoised image after obtaining the restored and denoised image.

The image repairing and denoising method and system provided by the present invention perform preprocessing operations on training image sample data and initialize model parameters; The given training image sample data matrix is decomposed into a joint low-rank and sparse principal component feature encoding matrix and a sparse error matrix; the original image is repaired and denoised, and the repaired and denoised image is obtained. . It can be seen that the image repairing and denoising method and system provided by the present invention fully consider the robust low-rank and sparse characteristics of the data while characterizing the image data, so as to overcome the deficiencies of the prior art and improve the image repairing performance. with denoising performance and model robustness.

Description of drawings

In order to illustrate the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that are used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only It is an embodiment of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to the provided drawings without creative efforts.

FIG. 1 is a flowchart of a specific implementation manner of an image restoration and denoising method provided by the present invention;

FIG. 2 is a flowchart of another specific implementation manner of the image restoration and denoising method provided by the present invention;

FIG. 3 is a schematic diagram showing the comparison of the results of describing the handwritten image data of the MNIST data set;

Figure 4 is a schematic diagram showing the comparison of the results of describing the face image data in the JAFFE data set, the AR data set, the Yale data set and the Yale-B data set;

5 is a schematic diagram of a quantitative evaluation result of image restoration and denoising effects provided by the present invention;

Figures 6(a)-6(f) are schematic diagrams of the results under the original image, 10% pixel destruction, 30% pixel destruction, 50% pixel destruction, 70% pixel destruction, and 90% pixel destruction degree, respectively;

FIG. 7 is a structural block diagram of an image repairing and denoising system provided by an embodiment of the present invention.

Detailed ways

In order to make those skilled in the art better understand the solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, but not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

A flowchart of a specific implementation manner of the image restoration and denoising method provided by the present invention is shown in FIG. 1 , and the method includes:

Step S101: preprocessing the training image sample data, and initializing the model parameters;

Due to various conditions and random interference, the original image cannot be used directly in the visual system. Therefore, in this step, the original image can be preprocessed as the training image sample data, and the model can be used as the sample data. Initialize each parameter in .

Step S102: Decompose a given training image sample data matrix into a joint low-rank and sparse principal component feature encoding matrix and a sparse error matrix by introducing the idea of joint low-rank and sparse matrix decomposition and using convex optimization technology;

Step S103: Repair and denoise the original image to obtain a repaired and denoised image.

The image repairing and denoising method provided by the present invention performs preprocessing operations on training image sample data and initializes model parameters; by introducing the idea of joint low-rank and sparse matrix decomposition, and using convex optimization technology, the given The given training image sample data matrix is decomposed into a joint low-rank and sparse principal component feature encoding matrix and a sparse error matrix; the original image is repaired and de-noised to obtain a repaired and de-noised image. . It can be seen that the image inpainting and denoising method provided by the present invention fully considers the robust low-rank and sparse characteristics of the data while characterizing the image data, so as to overcome the shortcomings of the prior art and improve the image inpainting and denoising. noise performance and model robustness.

In another specific embodiment of the image inpainting and denoising method provided by the present invention, the specific process of determining the joint low-rank and sparse principal component features of the training image sample data and the error matrix is described in detail.

For any given set of vectors that may contain erroneous data: Among them, x _i represents a sample, which is the vectorized description of the original image, n is the dimension of the image sample, and N is the total number of image samples.

This embodiment decomposes a given data matrix X into a joint low-rank and sparse principal component feature L ^S and a sparse error term E by optimizing the following minimization problem:

Among them, the trade-off parameter λ>0 depends on the noise level of the dataset, and α∈[0,1] is the trade-off parameter between low-rank and sparse coding terms. ||·|| _* represents the nuclear norm of a matrix, that is, the sum of the singular values in the matrix, and ||·|| ₁ is the l ¹ norm, which are defined as follows:

where A is a matrix.

By optimizing the above problem, the obtained optimal solution L ^S* is called the joint lowest rank and the most sparse principal component feature of the original data, and also corresponds to the optimal low rank and sparse restoration of the original data.

The above-mentioned problems can be specifically implemented through a variety of solutions. In order to improve efficiency, an inexact augmented Lagrange multiplier method is used in this embodiment. Specifically, the method includes:

First, the original problem is transformed into the following equivalent problem:

If the l ¹ norm regularization is performed on the sparse error term E, the corresponding augmented Lagrangian function can be defined as follows:

Among them, Y ₁ , Y ₂ , Y ₃ are Lagrange multipliers, and μ is a positive weight parameter.

By solving the augmented Langrangian function as follows to update variables alternately and alternately:

Since the variables are all interdependent, the above problems cannot be solved directly. In this embodiment, when calculating a certain variable, all other variables are fixed, and the solution is completed by iteratively optimizing the following convex sub-problems, and updating the variable values in turn:

Among them, J can be solved by singular value contraction operation, and F and E can be solved by scalar contraction operation. The problem to be optimized in each step of the method is a convex sub-problem, so it can be solved efficiently.

The specific algorithm of the transductive joint low-rank and sparse principal component feature coding algorithm adopted in this embodiment is as follows:

Input: raw data matrix Control parameters α, λ.

Output: Joint low-rank and sparse principal component features Sparse noise or error matrix (E ^* ←E _k+1 ).

initialization:

while has not converged do

Fix other variables to update the low-rank matrix J:

Fix other variables and update the sparse matrix F:

Fix other variables and update the principal component feature L ^S :

Fix other variables and update the sparse error matrix E:

Fix other variables and update the Lagrange multipliers Y ₁ , Y ₂ , Y ₃ :

Update parameter μ:

μ _k+1 =min(ημ _k ,max _μ );

Check for convergence:

like then stop;

otherwise k=k+1

end while.

A flowchart of another specific implementation of the image repairing and denoising method provided by the present invention is shown in FIG. 2 . On the basis of any of the above embodiments, in this embodiment, after obtaining the image after repairing and denoising It can further include:

Step S104: Evaluate the restoration effect of the image by performing visual observation and index quantification on the restored and denoised image.

Specifically, after the image is inpainted and denoised, the inpainting effect can be visually displayed through image visualization technology, and can be visually compared with the results of other related methods.

In addition, quantitative evaluation indicators based on reconstruction accuracy can also be used, and the reconstruction accuracy is defined as follows:

in, represents the recovery result of the original given data, Indicates the effect of image inpainting under different image pixel damage levels. From the above definition, the result when repaired in the presence of noise with the repair results on the original data When closer, the reconstruction accuracy will be a larger number; otherwise, to a smaller value. That is, The larger the value, the better the effect of image pixel error correction or the effect of image denoising.

The image inpainting and denoising method provided by the present invention takes into account the robust low-rank and sparse characteristics of the data when performing robust principal component feature extraction by introducing the current popular ideas of low-rank matrix restoration and sparse matrix description. Therefore, more descriptive and robust principal component features can be extracted, and the performance and model robustness of image inpainting and denoising can be further improved, and a better image inpainting effect can be obtained.

Based on the method of the present invention, the handwriting restoration effect can be tested, and specifically, the test can be carried out in the database of MNIST handwritten digits. The MNIST database has a total of 60,000 training samples and 10,000 test samples, from which 24 digital "0" samples were extracted for testing. The face description, inpainting, and denoising effect tests are performed on the following four face databases: Japanese Female Facial Expression Database (JAFFE), AR Face Database, Extended Yale-B Face Database, and Yale Face Database. The JAFFE face database has a total of 213 facial expressions made by 10 Japanese women in 7 types of expressions. The images in the Yale face database differ in terms of lighting, facial expressions, whether or not they wear glasses. The extended Yale-B face database has about 64 images of each person under different lighting and different expressions. The AR face database has more changes, such as different lighting, different expressions, and whether it is obscured by scarves or sunglasses. In this embodiment, 24 facial images are selected from each database to form a data matrix with a size of 128*192.

FIG. 3 and FIG. 4 are schematic diagrams showing the effect of the image repairing and denoising method provided by the present invention. Among them, Figure 3 shows the results of describing the handwriting image data of the MNIST data set, showing the comparison results between the method of the present invention and RPCA and SR respectively. The first row of pictures is the handwriting restoration effect of the RPCA method; the second row of pictures is The handwriting restoration effect of SR, the third row is the handwriting restoration effect of the present invention, and the pictures are the original picture, the restored picture and the error in order from left to right. Figure 4 shows the results of describing the face image data in the JAFFE dataset, AR dataset, Yale dataset and Yale-B dataset. From the first row to the fourth row are JAFFE, AR, Yale-B and The facial image inpainting results of the Yale dataset, the pictures are the original picture, the restored picture and the error from left to right. It can be seen from the results that the method of the present invention can be effectively used for image description and automatic error detection by considering the low rank and sparse characteristics of data at the same time, and achieves better image restoration and denoising effects.

Fig. 5 and Fig. 6(a)-6(f) are schematic diagrams of quantitative evaluation results of image restoration and denoising effects provided by the present invention, wherein Fig. 5 is the quantitative evaluation result, and Fig. 6(a)-6(f) is The original image and several examples of different pixel destruction levels, where 6(a)-6(f) correspond to the original image, 10% pixel destruction, 30% pixel destruction, 50% pixel destruction, 70% pixel destruction, 90% pixel destruction. In this embodiment, several images in the AR face data set are used for experiments, and the pixels of the face image are destroyed to varying degrees, and then the method of the present invention and other related methods are used for repairing and denoising, and the reconstruction accuracy is achieved. Quantitatively measure the effect of the method of the present invention and other related methods on image denoising, wherein the damaged pixel gray value is replaced by its inverse pixel gray value, that is, any damaged pixel gray value g is replaced by 255 -g. It can be seen that the reconstruction accuracy of each method decreases as the degree of pixel corruption increases. However, the method of the present invention shows the property of being more robust to the damaged pixels, and achieves better repairing and denoising effects.

The method is described in detail in the above-mentioned embodiments disclosed in the present invention, and the method of the present invention can be implemented by various forms of systems, so the present invention also discloses a system. The image inpainting and denoising system provided by the embodiments of the present invention will be introduced below. The image inpainting and denoising system described below and the image inpainting and denoising method described above may refer to each other correspondingly. FIG. 7 is a structural block diagram of an image repairing and denoising system provided by an embodiment of the present invention. Referring to FIG. 7, the image repairing and denoising system may include:

The preprocessing module 100 is used to perform preprocessing operations on the training image sample data and initialize the model parameters;

The decomposition module 200 is used to decompose a given training image sample data matrix into a joint low-rank and sparse principal component feature encoding matrix and a sparse error matrix by using the convex optimization technique by introducing the idea of joint low-rank and sparse matrix decomposition;

The repairing module 300 is used for repairing and denoising the original image to obtain an image after repairing and denoising.

As a specific implementation manner, in the image restoration and denoising system provided by the present invention, the above-mentioned decomposition module 200 can be specifically used for:

for vector sets Decompose data matrix X into joint low-rank and sparse principal component features L ^S and sparse error matrix E:

Among them, λ>0 is the trade-off parameter, α∈[0,1] is the trade-off parameter between low-rank and sparse coding terms, ||·|| _* represents the kernel norm of the matrix, and ||·|| ₁ is l ¹ norm, The optimal solution L ^S* obtained by optimization is the joint low-rank and sparse principal component features of the training image sample data, x _i is a sample in the training image sample data, n is the dimension of the image sample, and N is the The total number of image samples.

As a specific embodiment, in the image restoration and denoising system provided by the present invention, the above-mentioned decomposition module 200 is specifically used for:

Decompose data matrix X into joint low-rank and sparse principal component features L ^S and sparse error matrix E:

Define the augmented Lagrangian function:

Among them, Y ₁ , Y ₂ , and Y ₃ are Lagrangian multipliers, and μ is a weight parameter. Through the augmented Lagrangian function to alternately update the variable:

By iterative optimization of the convex subproblem, the solution is completed by sequentially updating the variable values:

J is solved by a singular value contraction operation, and F and E are solved by a scalar contraction operation.

As a specific embodiment, the image restoration and denoising system provided by the present invention may further include:

The evaluation module is used to evaluate the restoration effect of the image by performing visual observation and index quantification on the restored and denoised image after obtaining the restored and denoised image.

The image restoration and denoising system provided by the present invention performs preprocessing operations on the training image sample data, and initializes the model parameters; by introducing the idea of joint low-rank and sparse matrix decomposition, and using the convex optimization technology, the given The given training image sample data matrix is decomposed into a joint low-rank and sparse principal component feature encoding matrix and a sparse error matrix; the original image is repaired and de-noised to obtain a repaired and de-noised image. . It can be seen that the image inpainting and denoising system provided by the present invention fully considers the robust low-rank and sparse characteristics of the data while characterizing the image data, so as to overcome the shortcomings of the prior art and improve the image inpainting and denoising. noise performance and model robustness.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments may be referred to each other.

The above description of the disclosed embodiments enables any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (4)

1. An image restoration and denoising method is characterized by comprising the following steps:

preprocessing the training image sample data and initializing and setting model parameters;

decomposing a given training image sample data matrix into a combined low-rank and sparse principal component feature coding matrix and a sparse error matrix by introducing the idea of combining low-rank and sparse matrix decomposition and utilizing a convex optimization technology;

repairing and denoising the original image to obtain a repaired and denoised image;

the method for decomposing the given training image sample data matrix into the joint low-rank and sparse principal component feature coding matrix and the sparse error matrix by introducing the idea of joint low-rank and sparse matrix decomposition and utilizing the convex optimization technology comprises the following steps of:

for vector setsDecomposing a data matrix X into a joint low-rank and sparse principal component feature coding matrix L^SAnd sparse error matrix E:

wherein, λ > 0 is a trade-off parameter, α ∈ [0,1 ]]For trade-off parameters between low-rank and sparse coding terms, | · | | luminance_*Is the kernel norm of the matrix, | ·| luminance₁Is 1¹The norm of the number of the first-order-of-arrival,σ_i(A) singular values in the matrix A; optimal solution L obtained by optimization^S*Coding a matrix, x, for joint low rank and sparse principal component features of the training image sample data_iFor one sample in the training image sample data, N is the dimension of the image sample, and N is the total number of the image samples;

the method for decomposing the given training image sample data matrix into the joint low-rank and sparse principal component feature coding matrix and the sparse error matrix by introducing the idea of joint low-rank and sparse matrix decomposition and utilizing the convex optimization technology comprises the following steps of:

decomposing a data matrix X into a joint low-rank and sparse principal component feature coding matrix L^SAnd sparse error matrix E:

defining an augmented Lagrangian function:

wherein, Y₁,Y₂,Y₃Is a Lagrange multiplier, mu is a weight parameter, and the Lagrange function is augmented by the weight parameterTo alternately update the variables in turn:

and (3) sequentially updating variable values to complete solution by iteratively optimizing the convex sub problem:

j is solved by a singular value contraction operation, and F and E are solved by a scalar contraction operation.

2. The method for restoring and denoising an image according to claim 1, wherein after obtaining the restored and denoised image, the method further comprises:

and evaluating the repairing effect of the image by performing visual observation and index quantification on the repaired and denoised image.

3. An image inpainting and denoising system, comprising:

the preprocessing module is used for preprocessing the training image sample data and carrying out initialization setting on model parameters;

the decomposition module is used for decomposing a given training image sample data matrix into a combined low-rank and sparse principal component characteristic coding matrix and a sparse error matrix by introducing the idea of combined low-rank and sparse matrix decomposition and utilizing a convex optimization technology;

the restoration module is used for restoring and denoising the original image to obtain a restored and denoised image;

the decomposition module is specifically configured to:

for vector setsDecomposing a data matrix X into a joint low-rank and sparse principal component feature coding matrix L^SAnd sparse error matrix E:

wherein, λ > 0 is a trade-off parameter, α ∈ [0,1 ]]For trade-off parameters between low-rank and sparse coding terms, | · | | luminance_*Is the kernel norm of the matrix, | ·| luminance₁Is 1¹The norm of the number of the first-order-of-arrival,σ_i(A) singular values in the matrix A; optimal solution L obtained by optimization^S*Coding a matrix, x, for joint low rank and sparse principal component features of the training image sample data_iFor one sample in the training image sample data, N is the dimension of the image sample, and N is the total number of the image samples;

the decomposition module is specifically configured to:

decomposing a data matrix X into a joint low-rank and sparse principal component feature coding matrix L^SAnd sparse error matrix E:

defining an augmented Lagrangian function:

wherein, Y₁,Y₂,Y₃Is a Lagrange multiplier, mu is a weight parameter, and the Lagrange function is augmented by the weight parameterTo alternately update the variables in turn:

and (3) sequentially updating variable values to complete solution by iteratively optimizing the convex sub problem:

j is solved by a singular value contraction operation, and F and E are solved by a scalar contraction operation.

4. The image inpainting and denoising system of claim 3, further comprising:

and the evaluation module is used for evaluating the restoration effect of the image by performing visual observation and index quantification on the restored and denoised image after the restored and denoised image is obtained.