AU2020100460A4

AU2020100460A4 - Single image deraining algorithm based on multi-scale dictionary

Info

Publication number: AU2020100460A4
Application number: AU2020100460A
Authority: AU
Inventors: Shuying Huang; Yating Xu; Yong Yang
Original assignee: Xu Yating Miss
Current assignee: Xu Yating Miss
Priority date: 2020-03-26
Filing date: 2020-03-26
Publication date: 2020-04-30
Anticipated expiration: 2028-03-26

Abstract

We aim to remove the rain tracks from the rain images and retain the structure information of the original rain map to the greatest extent. Due to the complexity of the rain layer, the rainless background layer cannot be directly obtained at one time. Therefore, We according to the rain streaks of many aspects, such as sparsity, structural and directional information, proposed a new single image to the rain, which framework of the method through constant iterative update background layer, the sparse coefficient of the rain layer, the rain dictionary and a new rain layer, thereby gaining a free-rain image. Our main contribution can be divided into three parts: (I) A very effective convolutional sparse coding framework is proposed to iteratively update the rain layer and the background layer. (II) Considering the multi-scale characteristics of the noise rain layer information in the rain image taken in reality under different the depth of field, we proposed the method of learning multi dictionary, and carried out the convolution sparse coding for the raindrop information of different sizes (III) In the process of solving the rain layer, we proposed to use the multi-scale dictionary to solve the updated rain layer information, and to use the consistency of rain direction and the structure of raindrops to propose two prior constraints based on gradient, so as to obtain better results. Finally, ADMM algorithm is used to solve the model alternately to obtain the rainless image with rich details.

Description

BACKGROUND AND PURPOSE [0001] Due to bad weather, the camera captured images with low contrast image blurring detail information loss, greatly reduce the follow-up the veracity and reliability of analysis and processing, for a variety of image to the principle of rain technology characteristics and processing effects, we can go to the rain image methods according to different method and processing, the technology is divided into the image to rain to rain method based on imaging parameters to rain method based on time domain method and frequency domain to rain to rain method based on sparse domain four categories.

[0002] In recent years, with the rapid development of signal sparse representation theory, the method based on sparse representation for image of the inner structure characteristics, in order to overcome the influence of such factors as complicated background noise is becoming more and more popular, single image to the rain in the target detection of visual tracking, and other fields has been widely attention, however, the existing sparse representation method is focused on sparse constraint and no rain layer separation, and then combined with the prior knowledge of the rain or the rain line for rain and no rain dictionary classification and image reconstruction, these methods ignore the direction of the original image when reconstructing image structure information, meanwhile their often shows limited capacity in modeling textures or fine-scale details with complex patterns.

[0003] We aim to remove the rain tracks from the rain images and retain the structure information of the original rain map to the greatest extent. Due to the complexity of the rain layer, the rainless background layer cannot be directly obtained at one time. Therefore, We according to the rain streaks of many aspects, such as sparsity, structural and directional information, proposed a new single image to the rain, which framework of the method through constant iterative update background layer, the sparse coefficient of the rain layer, the rain dictionary and a new rain layer, thereby

2020100460 26 Mar 2020 gaining a free-rain image.

[0004] Our main contribution can be divided into three parts: (I) A very effective convolutional sparse coding framework is proposed to iteratively update the rain layer and the background layer. (II) Considering the multi-scale characteristics of the noise rain layer information in the rain image taken in reality under different the depth of field, we proposed the method of learning multi-dictionary, and carried out the convolution sparse coding for the raindrop information of different sizes (III) In the process of solving the rain layer, we proposed to use the multi-scale dictionary to solve the updated rain layer information, and to use the consistency of rain direction and the structure of raindrops to propose two prior constraints based on gradient, so as to obtain better results. Finally, ADMM algorithm is used to solve the model alternately to obtain the rainless image with rich details.

THE PROPOSED METHOD [0001] The traditional single image de-raining method is to treat the rainy image as the superposition of background layer and rain layer, so the rain model can be described as follows :

Y=B+R Equation 1 [0002] where Ye R’ represents the inputted rainy image, and Be R’^v is the background layer and Re R’^v is the rain layer. Single image layer separation is an ill-posed problem, and thus the closed solutions requires us to provide complete transcendental supplementary priors information. In our method, the image separation is achieved by solving the following objective function:

Μ N min JI Y-B-R ||j. +a£|| f_A . ® B ||, + β || V _yR ||, +β₂ || V % M _y - A) ||, +β₂ \\M,. ||, /=1 ./=1

Equation 2 [0003] Where II* II, is the / norm , and ^α·β are regularization parameters imposed on the background layer and the rain layer prior terms, respectively, where

2020100460 26 Mar 2020

Λ,,’{* = 1,2 } is the inner product of two horizontal gradient operators which to model the complementary subspace of signal. Where D_{S j},{ j = 1,2,3,4 ] is the rainy layer dictionary and M f j = 1,2,3,4} is its corresponding coefficient map ,where ^v _x is the difference operator in terms of the horizontal direction and the vertical direction , and “® ” denotes the convolution operation, the whole model is solved iteratively by ADMM algorithm.

DICTIONARIES LEARNING PROCESS [0001] Figure through the rain, we use a dictionary to leam the way to encode images, sparse dictionary learning is a kind of special expression of image, this paper adopts four size of [15*15,11*11,7*7,3*3] dictionary method to represent the rain layer, according to the weight distribution of different dictionaries, can be of different sizes of raindrops information express, specific rain dictionary learning process is as follows:

[0002] Stepl: the rain input layer number K and R has defined the dictionary size template (15 * 15 Identity matrix);

[0003] Step2: The rain layer is divided into patches, the step-size is land patch size is based on the size of the Initialize dictionary, and the mean block is calculated to get the difference block;

[0004] Step3: Multiply the matrix of the difference blocks by its transpose, and you get a matrix of size 225*225 [0005] Step4: By singular value decomposition SVD, the resulting U component is used as the dictionary for initialization, according to the rain streaks are nonnegative and generally have the brightest intensity in a rainy image. We extract the nonnegative data of the first K column vectors as the dictionary, and then import the dictionary into the template. In the process of de-rain, the dictionary can be regarded as the convolution filter of a filter to the input rain layer.

2020100460 26 Mar 2020

THE ALGORITHM FOR THE PROPOSED MODEL [0001] In the following chapters, we will detail the model structure mentioned in the previous section of the module. The first term is the well-known fidelity term, the second term is the prior constraint for the solution of the background layer, and the third and fourth terms are the prior constraint for the solution of the rain layer, respectively representing the sparse constraint along the y direction during the falling of the rain grain and the gradient information of the filtered image can be determined by the gradient information of the input image. The reconstructed and updated rain direction information is similar to the original rain pattern direction information. The last term is the sparse coefficient constraint of the convolution sparse coding process.

UPDATING B [0001] According to rain again on the vertical direction of gradient information, we adopt two horizontal gradient filter, in the rain of input image for a class of second order gradient operator for rapid decomposition, two filter by convolution of sparse coding process will transform into a sparse dictionary, sparse sex prior to join the background layer, and update the background layer is obtained. The initialization of the “ ⁰, The solution process of the algorithm is as follows:

M min || Y- B- R ||* +«^||./,, ® 51|, '=> Equation 3 [0002] We introduce the group of auxiliary variables L, = f_Aj ® ,m , where A is the

Lagrange variable for y , A_max and P are the parameters in the algorithm. ) denotes the soft-thresholding operator with parameter / , which is the solution for the ζ -norm approximation problem. The closed-form solution in the u-step in (4) can be efficiently solved in the FFT domain.

2020100460 26 Mar 2020 ^Ri' = ⁺⁷ ’ (^χ++—Σ -A, a );

H⁺¹=saAX⁺¹+—A);

' V Tk

L^k+' = f + μ _k(f_AiB^k+' - s^k+'y, \jfFk < Fmin’Fk+i -Fk A Equation 4

UPDATING M [0001] Fixing B and, we solve the following sub-problem to obtain M:

« Equation 5 [0002] The optimization problem in (5) is a convolutional sparse coding problem. We adopt the ADMM scheme and exploits the FFT to improve computation efficiency.

UPDATING R min || Y-B-R |£ +β || V_yR ||, +β₂1| V_v£z>_{5 y} -R) ||, Equation 6 7=1 [0001] Likely, we introduce the group of auxiliary variables ^u = (Σ ^D$,i ® ^M i ~^Rf^{v = v} y^R , Pi and P2 represent two Lagrange multipliers, here 7=1 we use it to solve the problem (6) that can be decomposed into three simple subproblems.

[0002] Because of the rain and the background layer is the process of superposition, so that we can get the background layer and initialize the dictionary to represent the rain figure, according to the convolution sparse coding process, we can get through the convolution of the sparse rain figure and the corresponding sparse coefficient, add the horizontal and vertical two directional and rain lines of sparse prior to get rain layer constraint items as follows :

2020100460 26 Mar 2020

R^k+' = inin(/?'.max(/?^il.()));

fo^k=argminfo\\ViR'-R^k)-u^k-F\2+^\Wy(R^k-P- — \\2 2 /, 2 /2 u^k = sign(y X(R'~ R^k)+max(y X(R'~ R^k)+^—\-—); _c .· π ·{ /, /, /, Equation 7 v^k = signty R^k +^-) max(| V R^k +^-1 - ^);

/2 /2 /2 p^k+' = p( + r^_x(R'-R^k+j-u^k+'y,

K⁺¹ = Pf +U(V X⁺¹ -vⁱ⁺¹);

UPDATING D [0001] With the fixed B_s R and coefficients M, we need to update the synthesis dictionary:

min/f \\V_x(^D_s/®Mj -R)\\_t Equation 8 ° ./=1 [0002]Let ^vec(D_s j ® Mj) = D_{s :/}M_s = MD_s where A is the vectorization of all the filters {D_s.j}_y=1.2.3.4 M = [M_t,M₂,M₂,M₄], and Mj is generated by collecting the patches in

Mj. We utilize a proximal gradient descent method to solve:

|[A - ^Prox||.||<_(Z'^) Equation 9 [0003] r is the step length of the gradient descent step, and^Pro^xn.n<(.) is the 4 -ball proximal operator, which makes each filter satisfy the constraint 11 f_SJ |£< 1.

Claims

1. The procedures of image fusion are as follows:

Stepl: initialization parameter [0001] Input image Y, analysis fdters {f_{A ;}}_{;=1 >2},regularization parameters a,β, γ.

Step2: Updating B [0002] According to rain again on the vertical direction of gradient information, we adopt two horizontal gradient filter, in the rain of input image for a class of second order gradient operator for rapid decomposition, two filter by convolution of sparse coding process will transform into a sparse dictionary, sparse sex prior to join the background layer, and to solve the background layer.

Step3: Initializing dictionary [0003] we use a dictionary to learn the way to encode images, sparse dictionary learning is a kind of special expression of image, this paper adopts four size of [15*15,11*11,7*7,3*3] dictionary method to represent the rain layer, according to the weight distribution of different dictionaries, can be of different sizes of raindrops information express, specific rain dictionary learning process is as follows:

[0004] 1) the rain input layer number K and R has defined the dictionary size template (15 * 15 Identity matrix);

[0005] 2) The rain layer is divided into blocks, step size is 1, size is based on the size of the dictionary, and the mean block is calculated to get the difference block;

[0006] 3) Multiply the matrix of the difference blocks by its transpose, and you get a matrix of size 225*225 [0007] 4) By singular value decomposition SVD, the resulting U component is used as the dictionary for initialization, according to the rain streaks are nonnegative and generally have the brightest intensity in a rainy image. We extract the non-negative data of the first K column vectors as the dictionary, and then import the dictionary into the template. In the process of de-rain, the dictionary can be regarded as the convolution filter of a filter to the input rain layer.

2020100460 26 Mar 2020

Step4: Update rain layer and sparsity coefficient [0008] Because of the rain and the background layer is the process of superposition, so that we can get the background layer and initialize the dictionary to represent the rain figure, according to the convolution sparse coding process, we can get through the convolution of the sparse rain figure and the corresponding sparse coefficient, add the horizontal and vertical two directional and rain lines of sparse prior to get rain layer constraint items

Step5: Update rain layer dictionary [0009] Update the dictionary according to the process of convolution sparse coding. The main algorithm flow of updating the dictionary will be introduced in detail in the next chapter.

[0010] Repeat the above steps from the second step until the whole model converges. The criterion of convergence is that the updated background layer is infinitely close to the background layer solved last time.

2. The Algorithm for The Proposed Model

Updating B [0001] The initialization of the = ⁰, The solution process of the algorithm is as follows:

m Equation 1 mm||y-5-J?||^ +a£|| f_Aj ® B ||, / = 1 [0002] We introduce the group of auxiliary variables = f_AJ ® ,m , where A is the

Lagrange variable for y , A_max and P are the parameters in the algorithm. / denotes the soft-thresholding operator with parameter , which is the solution for the I -norm approximation problem. The closed-form solution in the u-step in (4) can be efficiently solved in the FFT domain.

2020100460 26 Mar 2020 = ^+l'^>< (^χ+νΣ +—Σ A, A);

!A⁺¹ = v.(AX⁺¹ +—A);

' T_t Tk fo'=/₊p_k(.f_Afo^M-_Sff [ifRk < Rm^’Rk+ι -Rk P, Equation?

Updating M [0003] Fixing B and, we solve the following sub-problem to obtain M:

min/?₂ \N_x(D®M-R) ||, +β, \\M ||, Equation 3 [0004] The optimization problem in (5) is a convolutional sparse coding problem. We adopt the ADMM scheme and exploits the FFT to improve computation efficiency. Updating R min II Y-B-R |£ +β || V_yR ||, +β₂1| V_x(^^Ds,j® ~^R) Hi Equation 4

7=1 [0005] Likely, we introduce the group of auxiliary variables ^u = (Σ^D$,i ®^Mi ~^Rh^v = V_yR , p_t and p₂ represent two Lagrange multipliers, here ./=1 we use it to solve the problem (6) that can be decomposed into three simple subproblems.

[0006] Because of the rain and the background layer is the process of superposition, so that we can get the background layer and initialize the dictionary to represent the rain figure, according to the convolution sparse coding process, we can get through the convolution of the sparse rain figure and the corresponding sparse coefficient, add the horizontal and vertical two directional and rain lines of sparse prior to get rain layer constraint items as follows :

2020100460 26 Mar 2020

R^k+' = mm(A',max(Aⁱ⁺¹,0));

2 /, 2/ kk 0 u = sign(V _v (RR^k)+ —) max(|V _v (RR^k )+^-| - /-+);

·! Λ /1/1 Equation 5 v^k =sign(y R^k+^-)max(]\7 R^k + ^\ /2 /2/2

P^k+'=p^k +/,(V,(A'-Aⁱ⁺¹)-_M ⁱ⁺¹);

K⁺¹⁼A +/₂(VX⁺¹-vⁱ⁺¹);

Updating D [0007]Let ^vec(Dsj ® ^Mj) = Ds ,Μ, = MD_s where A is the vectorization of all the filters {D_Sj 1/=1.2.3.4 M = [M_},M₂,M₃,M₄] _f and ^Mj is generated by collecting the patches in

Mj. We utilize a proximal gradient descent method to solve:

i/r⁵=A'-^^r(L-^-O;

1[Λ'⁺¹ = ^Pro+||.||<(Equation 6 [0008] τ is the step length of the gradient descent step, and ^Pr is the l₂ -ball proximal operator, which makes each filter satisfy the constraint 11 f_SJ 11^ 1.