CN108986060B - Multi-image reflected light suppression method based on sparse and low-rank matrix decomposition - Google Patents

Abstract

The invention discloses a method for suppressing the reflected light of a plurality of pictures based on sparse and low-rank matrix decomposition, which comprises the following steps of; (1) acquiring a plurality of images shot for the same scene at different angles; (2) aligning the plurality of images by utilizing a siftflow algorithm; (3) reference picture P₀Dividing the data into a plurality of patches with the same size; (4) for each patch, the following steps are performed: A. acquiring column vectors formed by RGB values of a patch similar to the current patch and other images, and forming an RGB matrix by the column vectors; B. acquiring a column vector formed by the minimum gradient value of each pixel in a patch similar to the current patch and other images, and taking the column vector as the column vector of each column to form a minimum gradient matrix; C. performing sparse matrix and low-rank matrix decomposition according to the minimum gradient matrix and the RGB matrix; (5) and recombining the low-rank matrixes corresponding to all the patch to obtain an image with the reflected light suppressed. The invention has better image processing effect.

Description

Multi-image reflected light suppression method based on sparse and low-rank matrix decomposition

Technical Field

The invention relates to image processing, in particular to a multi-picture reflected light suppression method based on sparse and low-rank matrix decomposition.

Background

In life, when an image is taken through glass, the taken picture includes not only a scene to be taken inside the glass but also an unnecessary scene reflected outside the glass. The reflected light removal is to remove an unnecessary reflected scene from the captured image. Most current methods consider the glass image as a linear combination of transmission and reflection images, and reconstruct the transmission image by suppressing the reflection image, such as the following:

the first category of methods relies on a single image. Such methods primarily utilize a priori knowledge and assumptions about images generated from glass, such as sparseness of gradients, relative smoothness, ghosting cues, and the like. Levin and Weiss reconstruct an optimized reflectance image by minimizing the cost function using a model of the gradient distribution in the image. Li and Brown achieve the effect of removing reflection components by utilizing the reflection energy inhibition of a gradient domain by virtue of the characteristic that the gradient of reflected light is smoother and fuzzy than that of refracted light. However, this assumption does not apply to very light scenes. Shin et al consider the multiple reflection characteristics of glass and suppress reflection by the ghost effect of light reflected from both the front and back surfaces of the glass. Such methods are described in references A.Levin and Y.Weiss.user associated separation of reflections from a single image using a spatial priority.IEEE Trans.image Process, 29(9), 1647-1654, Sept.2007; Y.Li and M.S.Brown.Single image layer separation using relative smoothening.in Proc.IEEE CVPR, pages 2752-; shih, D.Krishan, F.Durand, and W.T.Freeman.Refraction removal using stimulating cups.In Proc.IEEE CVPR, pages 3193-.

The second category of methods relies on multiple related images. Such methods may be aided by special camera devices such as polarizers, flash lamps, etc. The principle of utilizing the polarizer to remove the reflected light lies in the difference of the polarization of the refracted light wave and the reflected light wave, and a plurality of pictures are shot by adjusting the angle of the polarizer, so that an ideal transmission image is reconstructed. Besides, a plurality of related pictures can also be acquired by changing the viewpoint of the camera. Li and Brown analyze the gradient features of multiple pictures by aligning the images taken at these different angles, separating the gradients of the refractive and reflective images. Xue et al evaluate the different densities of the transmission and reflection scenes, respectively, to optimize the transmission and reflection images. Such methods are described in references y.li and m.s.brown.application conversion for automatic reflection removal. in proc.ieee ICCV, pages 2432-; xue, m.rubinstein, c.liu, and w.t.freeman.a computational approach for operation-free photopraphy.acm trans.graph, 34(4) 79: 1-79: 11, July 2015.

In general, the effect obtained by removing reflected light from a plurality of pictures is more desirable than that based on a single picture, and the used scenes are wider.

Disclosure of Invention

The purpose of the invention is as follows: aiming at the problems in the prior art, the invention provides a method for inhibiting the reflected light of a plurality of pictures based on sparse and low-rank matrix decomposition.

The technical scheme is as follows: the method for suppressing the reflected light of the multiple pictures based on sparse and low-rank matrix decomposition comprises the following steps of;

(1) acquiring a plurality of images shot for the same scene at different angles;

(2) aligning the plurality of images by utilizing a siftflow algorithm;

(3) selecting one of the aligned images as a reference image P₀And reference picture P₀Dividing the data into a plurality of patches with the same size;

(4) for each patch, the following steps are performed:

A. acquiring column vectors formed by RGB values of a patch similar to the current patch and other images, and forming an RGB matrix by the column vectors;

B. acquiring a column vector formed by the minimum gradient value of each pixel in a patch similar to the current patch and other images, and taking the column vector as the column vector of each column to form a minimum gradient matrix;

C. performing sparse matrix and low-rank matrix decomposition according to the minimum gradient matrix and the RGB matrix;

(5) and recombining the low-rank matrixes corresponding to all the patch to obtain an image with the reflected light suppressed.

Further, step (4) a specifically includes:

a-1, marking the current patch as ph₀And matching one from each of the other aligned images with ph₀The most similar patch, get a similar patch set PH₁＝{ph_j1, …, K-1}, where ph_jIs shown in picture P_jNeutral ph₀The most similar patch, K is the number of all images;

a-2, adding ph₀Adding to a set PH₁In (b), a new set PH is formed₂＝{ph_j|j＝0,…,K-1}；

A-3, will gather PH₂The pixel RGB value of each patch is used as a column vector, and all the column vectors form an RGB matrix I ═ pr₀,...,pr_K-1]Wherein pr is_kWatch PH₂Formed by the RGB values of all pixels of the kth patchA column vector.

Further, step (4) B specifically includes:

b-1, marking the current patch as ph₀And matching one from each of the other aligned images with ph₀The most similar patch, get a similar patch set PH₁＝{ph_j1, …, K-1}, where ph_jIs shown in picture P_jNeutral ph₀The most similar patch, K is the number of all images;

b-2, adding ph₀Adding to a set PH₁In (b), a new set PH is formed₂＝{ph_j|j＝0,…,K-1}；

B-3, calculating the set PH₂The gradient value of each pixel of each patch in the set is obtained

Denotes patch ph_jA gradient value at a pixel position p, area representing a pixel position area of patch;

b-4, for any position q, selecting a minimum value from the gradient values of the pixel positions q of all the latches

Namely, it is

Taking the minimum gradient value as the minimum gradient value of the pixel position q, repeating the step to obtain the minimum gradient values of all the pixel positions, and forming the minimum gradient values of all the pixel positions into a column vector t;

b-5, forming the column vectors into a minimum gradient matrix G_min＝[t,...,t]；G_minIs equal to the number of columns of the RGB matrix.

Further, step (4) C specifically includes:

c-1, establishing an objective optimization function as follows:

s.t.I＝T+R,R＝H

in the formula, I represents an RGB matrix, and T represents a low-rank matrix, namely refraction components in an image; r is a sparse matrix, namely a reflection component in an image; g_minRepresenting a minimum gradient matrix; λ and τ are user-defined parameters that are used to adjust the weight of each term in the optimization objective function; h is an auxiliary variable, and the auxiliary variable is,

a gradient of H;

and C-2, solving the target optimization function by adopting an iterative augmented Lagrange function method to obtain a low-rank matrix T.

Further, the step C-2 specifically comprises:

c-2-1: mixing I, G_minAs an input;

c-2-2: initialization of T ═ I, H ═ T

C-2-3: update as follows:

T_k+1＝D_1/(2u)[(I-R_k+H_k+(Y_1k-Y_2k)/u)/2]

R_k+1＝S_λ/u[I-T_k+Y_1k/u]

Y_1(k+1)＝Y_1k+u(I-T_k-R_k)

Y_2(k+1)＝Y_2k+u(T_k-H_k)

in which the index k denotes the kth iteration, D_1/(2u)(X)＝US_1/(2u)(∑)V^*，S_1/(2u)(∑) is sng (Σ) max (Σ | -1/(2u),0), Σ is a diagonal matrix composed of eigenvalues of the matrix X, U, V^*The corresponding matrix is SVD decomposed for matrix X, i.e. X ═ U ∑ V^*U is a penaltyWeight factor of the term, Y₁And Y₂Is a lagrange multiplier;

c-2-4: if the iteration number is more than 20, or I-T_k+1-R_k+1||_FAnd | | | T_k+1-H_k+1||_FIf the value is less than the predefined threshold, stopping the iteration and determining the T at the moment_k+1And (6) outputting.

Has the advantages that: compared with the prior art, the invention has the following remarkable advantages: the method of the invention directly utilizes the original image and the gradient to carry out optimization, can obtain the image with higher contrast, and has no color distortion.

Drawings

FIG. 1 is a schematic flow diagram of one embodiment of the present invention;

FIG. 2 is a graph of the results of various stages of image processing using the method of the present invention;

fig. 3 is a diagram of the results of image processing using the method of the present invention as well as other methods.

Detailed Description

The embodiment provides a method for suppressing light reflected by a plurality of pictures based on sparse and low-rank matrix decomposition, as shown in fig. 1, including;

(1) multiple images taken of the same scene at different angles are acquired.

Setting to shoot K pictures, the pixel point of each image can be regarded as the linear combination of a reflection image and a refraction image:

P_k(p)＝T_k(p)+R_k(p),k＝0,1,.....,K-1

P_k(p) is the value of the pixel point p of the kth image, T_kIs a refractive light value, R_kThe total number of K images is the reflected light value.

(2) The multiple images are aligned using a siftflow algorithm.

And when in alignment, selecting the first image as a reference image, and aligning the K images. Since the intensity of the refracted light is greater than the intensity of the reflected light, alignment is mainly performed by refracting the light, which results in the refracted light matching each other on each image, while the reflected light differs from image to image, and the following relationship is obtained:

it is composed of

Respectively correspond to P_k(p)、T_k(p)、R_k(p) the aligned value, the value of the pixel point p of the kth image, and T (p) the value of the pixel point p of the true refracted light image to be restored.

(3) Selecting one of the aligned images as a reference image P₀And reference picture P₀Divided into a plurality of equal-sized patches.

Wherein, when dividing, assume the reference picture P₀The length of (l) is w, the width of (w) is w, and the patch is a square with the length of (patchsize), so that the original image is divided into (l/patchsize) × (w/patchsize) patches.

(4) For each patch, the following steps are performed:

A. and acquiring column vectors formed by the RGB values of the patch similar to the current patch in the current patch and other images, and forming the column vectors into an RGB matrix.

The step A specifically comprises the following steps: a-1, marking the current patch as ph₀And matching one from each of the other aligned images with ph₀The most similar patch, get a similar patch set PH₁＝{ph_j1, …, K-1}, where ph_jIs shown in picture P_jNeutral ph₀The most similar patch, K is the number of all images; a-2, adding ph₀Adding to a set PH₁In (b), a new set PH is formed₂＝{ph_j0, …, K-1 }; a-3, will gather PH₂The pixel RGB value of each patch is used as a column vector, and all the column vectors form an RGB matrix I ═ pr₀,...,pr_K-1]Wherein pr is_kWatch PH₂The column vector formed by the RGB values of all pixels of the kth patch.

B. And acquiring a column vector formed by the minimum gradient value of each pixel in the patch similar to the current patch and other images, and taking the column vector as the column vector of each column to form a minimum gradient matrix.

The step B specifically comprises the following steps: b-1, marking the current patch as ph₀And matching one from each of the other aligned images with ph₀The most similar patch, get a similar patch set PH₁＝{ph_j1, …, K-1}, where ph_jIs shown in picture P_jNeutral ph₀The most similar patch, K is the number of all images; b-2, adding ph₀Adding to a set PH₁In (b), a new set PH is formed₂＝{ph_j0, …, K-1 }; b-3, calculating the set PH₂The gradient value of each pixel of each patch in the set is obtained

Denotes patch ph_jA gradient value at a pixel position p, area representing a pixel position area of patch; b-4, for any position q, selecting a minimum value from the gradient values of the pixel positions q of all the latches

Namely, it is

Taking the minimum gradient value as the minimum gradient value of the pixel position q, repeating the step to obtain the minimum gradient values of all the pixel positions, and forming the minimum gradient values of all the pixel positions into a column vector t; b-5, forming the column vectors into a minimum gradient matrix G_min＝[t,...,t]；G_minIs equal to the number of columns of the RGB matrix.

Wherein the gradient features of the image are:

making

Representing the x or y direction of the imageTowards the gradient, one can get:

② since the reflected signal is weaker than the refracted signal, so

The salient regions in (1), i.e., the regions with stronger gradients, may be assumed to be primarily from the refracted signal. In other words, for a given pixel point p, if

Is large, either

Or either

Thus, the following relationship between gradient strengths can be obtained:

thirdly, as can be seen from the above, when the pixel point p is located at the significant edge of the refraction image, it is very likely that a plurality of aligned images have the same refraction image gradient strength at the pixel point, and the influence of the reflection image gradient strength can be ignored, that is, the pixel point p has the same refraction image gradient strength

C. And performing sparse matrix and low-rank matrix decomposition according to the recent small gradient matrix and the RGB matrix to obtain a sparse matrix corresponding to the current patch. The method specifically comprises the following steps:

c-1, establishing an objective optimization function as follows:

s.t.I＝T+R,R＝H

the meaning of this formula is: it is desirable that the reflection component T be as low rank as possible and that the gradient of the artwork be as similar as possible to the approximate gradient found above. In the formula, I represents an RGB matrix, and T represents a low-rank matrix, namely refraction components in an image; r is a sparse matrix, namely a reflection component in an image; g_minRepresenting a minimum gradient matrix; λ and τ are user-defined parameters that are used to adjust the weight of each term in the optimization objective function; h is an auxiliary variable, and the auxiliary variable is,

a gradient of H;

and C-2, solving the target optimization function by adopting an iterative augmented Lagrange function method to obtain a low-rank matrix T.

The concrete solving method comprises the following steps:

augmented Lagrangian function for obtaining target optimization function

Mixing I, G_minAs an input;

initialization of T ═ I, H ═ T

Update as follows:

T_k+1＝D_1/(2u)[(I-R_k+H_k+(Y_1k-Y_2k)/u)/2]

R_k+1＝S_λ/u[I-T_k+Y_1k/u]

Y_1(k+1)＝Y_1k+u(I-T_k-R_k)

Y_2(k+1)＝Y_2k+u(T_k-H_k)

in which the index k denotes the kth iteration, D_1/(2u)(X)＝US_1/(2u)(∑)V^*，S_1/(2u)(∑) is sng (Σ) max (Σ | -1/(2u),0), Σ is a diagonal matrix composed of eigenvalues of the matrix X, U, V^*The corresponding matrix is SVD decomposed for matrix X, i.e. X ═ U ∑ V^*U is a weight factor for the penalty term, Y₁And Y₂Is a lagrange multiplier.

C-2-4: if the iteration number is more than 20, or I-T_k+1-R_k+1||_FAnd | | | T_k+1-H_k+1||_FIf the value is less than the predefined threshold, stopping the iteration and determining the T at the moment_k+1And (6) outputting.

(5) And recombining the low-rank matrixes corresponding to all the patch to obtain an image with the reflected light suppressed.

The method of the present embodiment is used for simulation verification.

The plurality of captured images are shown in fig. 2 (a), the first image is taken as a reference, the aligned image is shown in fig. 2 (b), the gradient image calculated from the aligned image is shown in fig. 2 (c), the minimum gradient image is shown in fig. 2 (d), the minimum gradient image is an image formed by selecting a minimum gradient value for each pixel in all similar patches, and the image obtained by suppressing the reflected light by the method of the present embodiment is shown in fig. 2 (e).

To better compare the effects of Li and Brown with other methods of the prior art, the same pictures were processed and the results are shown in fig. 3, which shows that the method of the present invention is significantly superior to the Li and Brown methods.

While the invention has been described in connection with what is presently considered to be the most practical and preferred embodiment, it is to be understood that the invention is not to be limited to the disclosed embodiment, but on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims (4)

1. A multi-picture reflected light suppression method based on sparse and low-rank matrix decomposition is characterized by comprising the following steps of;

(1) acquiring a plurality of images shot for the same scene at different angles;

(2) aligning the plurality of images by utilizing a siftflow algorithm;

(3) selecting one of the aligned images as a reference image P₀And reference picture P₀Dividing the data into a plurality of patches with the same size;

(4) for each patch, the following steps are performed:

A. acquiring column vectors formed by RGB values of a patch similar to the current patch and other images, and forming an RGB matrix by the column vectors;

B. acquiring a column vector formed by the minimum gradient value of each pixel in a patch similar to the current patch and other images, and taking the column vector as the column vector of each column to form a minimum gradient matrix;

C. performing sparse matrix and low-rank matrix decomposition according to the minimum gradient matrix and the RGB matrix; the method specifically comprises the following steps:

c-1, establishing an objective optimization function as follows:

s.t.I＝T+R,R＝H

in the formula, I represents an RGB matrix, and T represents a low-rank matrix, namely refraction components in an image; r is a sparse matrix, namely a reflection component in an image; g_minRepresenting a minimum gradient matrix; λ and τ are user-defined parameters that are used to adjust the weight of each term in the optimization objective function; h is an auxiliary variable, and the auxiliary variable is,

a gradient of H;

c-2, solving the target optimization function by adopting an iterative augmented Lagrange function method to obtain a low-rank matrix T;

(5) and recombining the low-rank matrixes corresponding to all the patch to obtain an image with the reflected light suppressed.

2. The sparse-and-low-rank matrix decomposition-based multiple-picture reflected light suppression method according to claim 1, wherein: the step (4) A specifically comprises the following steps:

a-1, marking the current patch as ph₀And matching one from each of the other aligned images with ph₀The most similar patch, get a similar patch set PH₁＝{ph_j1, …, K-1}, where ph_jIs shown in picture P_jNeutral ph₀The most similar patch, K is the number of all images;

a-2, adding ph₀Adding to a set PH₁In (b), a new set PH is formed₂＝{ph_j|j＝0,…,K-1}；

A-3, will gather PH₂The pixel RGB value of each patch is used as a column vector, and all the column vectors form an RGB matrix I ═ pr₀,...,pr_K-1]Wherein pr is_kIndicates the pH₂The column vector formed by the RGB values of all pixels of the kth patch.

3. The sparse-and-low-rank matrix decomposition-based multiple-picture reflected light suppression method according to claim 1, wherein: the step (4) B specifically comprises the following steps:

b-1, marking the current patch as ph₀And matching one from each of the other aligned images with ph₀The most similar patch, get a similar patch set PH₁＝{ph_j1, …, K-1}, where ph_jIs shown in picture P_jNeutral ph₀The most similar patch, K is the number of all images;

b-2, adding ph₀Adding to a set PH₁In (b), a new set PH is formed₂＝{ph_j|j＝0,…,K-1}；

B-3, calculating the set PH₂The gradient value of each pixel of each patch in the set is obtained

Denotes patch ph_jA gradient value at a pixel position p, area representing a pixel position area of patch;

b-4, for any position q, selecting a minimum value from the gradient values of the pixel positions q of all the latches

Namely, it is

Taking the minimum gradient value as the minimum gradient value of the pixel position q, repeating the step to obtain the minimum gradient values of all the pixel positions, and forming the minimum gradient values of all the pixel positions into a column vector t;

b-5, forming the column vectors into a minimum gradient matrix G_min＝[t,...,t]；G_minIs equal to the number of columns of the RGB matrix.

4. The sparse-and-low-rank matrix decomposition-based multiple-picture reflected light suppression method according to claim 1, wherein: the step C-2 specifically comprises the following steps:

c-2-1: mixing I, G_minAs an input;

c-2-2: initialization of T ═ I, H ═ T

C-2-3: update as follows:

T_k+1＝D_1/(2u)[(I-R_k+H_k+(Y_1k-Y_2k)/u)/2]

R_k+1＝S_λ/u[I-T_k+Y_1k/u]

Y_1(k+1)＝Y_1k+u(I-T_k-R_k)

Y_2(k+1)＝Y_2k+u(T_k-H_k)

in which the index k denotes the kth iteration, D_1/(2u)(X)＝US_1/(2u)(∑)V^*，S_1/(2u)(∑) is sng (Σ) max (Σ | -1/(2u),0), Σ is a diagonal matrix composed of eigenvalues of the matrix X, U, V^*The corresponding matrix is SVD decomposed for matrix X, i.e. X ═ U ∑ V^*U is a weight factor for the penalty term, Y₁And Y₂Is a lagrange multiplier;

c-2-4: if the iteration number is more than 20, or I-T_k+1-R_k+1||_FAnd | | | T_k+1-H_k+1||_FIf the value is less than the predefined threshold, stopping the iteration and determining the T at the moment_k+1And (6) outputting.

Images