CN111127354A - A single image rain removal method based on multi-scale dictionary learning - Google Patents

Abstract

The invention provides a multi-scale dictionary single-image rain removing method, which comprises the following steps: step 1, taking each pair of clean images and synthesized rain-carrying images as a training sample pair, and establishing a training set; step 2, constructing a network model, wherein the network model comprises a rough rain line extraction module and a fine rain line purification module, the rough rain line extraction module comprises two convolution layers and two ReLU activation function layers, and the rough rain line extraction module is used for realizing rough rain line extraction of combined rain images; the fine rain line purification module comprises seven convolution layers and four ReLU activation functions and is used for recovering a fine rain line graph from a noisy rain line image; and 3, training the network model by using the training set constructed in the step 1, and 4, inputting the image with rain to be tested into the trained network model to obtain a corresponding rain removing image. The invention comprehensively utilizes the sparse theory and the CNN learning ability, and greatly improves the solving efficiency and the reconstruction precision of the SC problem.

Description

A single image rain removal method based on multi-scale dictionary learning

technical field

The invention belongs to the field of image processing, and particularly relates to a method of dictionary learning to remove rain from a single frame image.

Background technique

In reality, most computer vision algorithms assume that the input is clear. However, for most outdoor vision systems, such as video surveillance and autonomous driving, the rainy environment will seriously affect the imaging quality, resulting in blurred images, distortions, and poor visibility. This will greatly reduce the performance of the system. Therefore, effectively eliminating the impact of rain on images has important application value, especially for a single image to remove rain, which is the top priority of the task of rain removal. Currently, single image deraining algorithms are mainly divided into two categories: prior-based and learning-based methods.

Prior-based methods often need to observe the characteristics of rain lines in advance, and design specific a priori information, such as rain lines are mostly inclined straight lines in a certain range or have low-rank characteristics, but the rain removal performance of such algorithms largely depends on Due to the selection of prior information, it is difficult to deal with complex rainfall conditions in reality.

Learning-based methods have been a research hotspot in recent years, mainly including methods based on convolutional neural networks (CNN), which treat the image rain removal task as a pixel-by-pixel regression task, take the entire image as input, and remove the The rain map is used as output for end-to-end training and testing. Such methods do not need to design priors manually, and use the powerful learning ability of CNN to allow the network to learn the rainline features by itself. However, in such methods, there is a lack of guidance and interpretability in network design, which is not conducive to the improvement and promotion of the network.

SUMMARY OF THE INVENTION

Based on the above analysis, the object of the present invention is to provide a single image rain removal method for multi-scale dictionary learning, the method introduces sparse prior into CNN network design, which greatly improves the rain removal effect

A method for removing rain from a single image of a multi-scale dictionary provided by the present invention includes the following specific steps:

Step 1, according to the existing clean images, by adding rain lines to obtain corresponding synthetic images with rain, using each pair of clean images and synthetic images with rain as a pair of training samples to establish a training set;

Step 2, building a network model, the network model includes a rough rainline extraction module and a fine rainline purification module, wherein the rough rainline extraction module includes two convolution layers and two ReLU activation function layers, which are used to realize the synthesis of Rough rainline extraction from rain images; fine rainline purification module includes seven convolutional layers and four ReLU activation functions to recover fine rainline images from noisy rainline images;

Step 3, use the training set constructed in step 1 to train the network model,

Step 4: Input the rain image to be tested into the trained network model to obtain the corresponding rain-removed image.

Further, the specific implementation of the rough rainline extraction module in step 2 is as follows,

为卷积操作，r_∈为提取的带噪雨线图像，y为合成带雨图像。Among them, E ₁ , E ₂ are two convolutional layers,

is the convolution operation, r _∈ is the extracted noisy rainline image, and y is the synthetic rainy image.

Further, the specific implementation mode of the fine rainline purification module in step 2 is as follows,

Described fine rainline purification module comprises two parts of sparse coding solution and HR feature reconstruction, and wherein sparse coding is to solve following minimization problem by convolution mode:

Among them, f _{j, i} represents the i-th filter kernel of the j-th dictionary, f _{j, i} are three sets of filter groups, that is, j=3, which actually means three rainline dictionaries of different scales, denoted as S ₁ , S ₂ , S ₃ , the transposed dictionary is denoted as G ₁ , G ₂ , G ₃ , where S ₁ , S ₂ , S ₃ , G ₁ , G ₂ , G ₃ are implemented by the convolution layer, and c is The number of channels to be decomposed, z _{j, i} are the convolutional sparse coding to be solved, ||.|| ₁ , ||.|| ₂ represent the l ₁ norm and the l ₂ norm, respectively, and λ is the sparse penalty coefficient;

After obtaining the sparse coding, reconstruct the denoised rainline map:

E₃为一个卷积层，r为最终恢复 的精细雨线图像；in,

E3 is a convolutional layer, and r is the final restored fine _rainline image;

均初始化为r_∈，r_∈为特征提取模块提取的带噪雨线图像； 计算

分别经过S₁、S₂、S₃的和，将此和从r_∈中减去，将上述差值分别经过G₁、 G₂、G₃后再分别与

相加，得到

重复上述过程，得到

即为最终的卷积稀疏编码，其中t为迭代次数；重建时，计算

分别经过S₁、S₂、S₃的和，此和经过ReLU后再经过E₃，即恢复出精细的雨线图像。The specific process is as follows: first

are initialized to r _∈ , and r _∈ is the noisy rainline image extracted by the feature extraction module;

After passing the sum of S ₁ , S ₂ , and S ₃ respectively, the sum is subtracted from r _∈ , and the above difference is passed through G ₁ , G ₂ , and G ₃ respectively, and then added to the

add up, get

Repeat the above process to get

is the final convolutional sparse coding, where t is the number of iterations; during reconstruction, calculate

After passing through the sum of S ₁ , S ₂ , and S ₃ respectively, the sum passes through ReLU and then passes through E ₃ , that is, a fine rainline image is recovered.

Further, formula (2) is transformed into a traditional sparse coding problem using the relationship between convolution and matrix multiplication, and under the assumption of non-negative sparse coding, the ISTA algorithm is used to solve,

为中间符号，

指阈值，t为迭代次数。in,

is the middle symbol,

refers to the threshold, and t is the number of iterations.

Further, in step 3, global residual learning is introduced into the network model, and the MSE loss function is selected to minimize this loss function as the training target. The expression of the MSE loss function is as follows:

Among them, Θ refers to the network model parameters, l is the index of the training samples in the training set, y _l -r _l is the rain-removed image output by the network model, and the difference with the real clean image x _l is accumulated to obtain the final error, so that the final error Minimization implements optimization of the network model.

Further, in step 1, the number of images in the training set is increased by means of flipping, rotating, scaling, and cropping, and then rain lines are added to each clean image through Photoshop to obtain a synthetic image with rain, and the corresponding clean image and synthetic rain are added. image as a pair of training samples.

The invention provides a single image rain removal algorithm, which comprehensively utilizes sparse theory and CNN learning ability, obtains an iterative formula by solving an optimization problem in SC, and implements it with CNN, which greatly improves the solving efficiency and reconstruction accuracy of SC problem.

Description of drawings

FIG. 1 is a general flowchart of network model construction in an embodiment of the present invention.

Figure 2 is a schematic diagram of sparse coding solution.

Figure 3 is a comparison diagram of rain removal algorithms for cat images in the Rain12 dataset.

Figure 4 is a comparison diagram of each algorithm for removing rain from forest images in the Rain1200 dataset.

Detailed ways

In order to understand the present invention more clearly, the following describes the technical content of the present invention in detail.

As shown in Figure 1, a kind of single image rain removal method based on multi-scale dictionary learning provided by the invention is specifically divided into four steps:

Step 1, according to the existing clean images, by adding rain lines to obtain corresponding synthetic images with rain, using each pair of clean images and synthetic images with rain as a pair of training samples to establish a training set;

Due to the limited training dataset, data augmentation methods are required to effectively utilize the limited HR images. Data augmentation is an effective way to expand the size of data samples. Deep learning is a data-driven method, the larger the training data set, the stronger the generalization ability of the trained model. However, when collecting data in practice, it is difficult to cover all scenarios, and collecting data also requires a lot of cost, which leads to a limited training set in practice. If a variety of training data can be generated based on existing data, it will be possible to achieve better open-source and cost-effective savings, which is the purpose of data augmentation.

Commonly used data augmentation techniques are:

(1) Flip: Flip includes horizontal flip and vertical flip.

(2) Rotation: Rotation is clockwise or counterclockwise rotation. Note that when rotating, it is best to rotate 90-180°, otherwise there will be scale problems.

(3) Zoom: The image can be enlarged or reduced. When zoomed in, the enlarged image size will be larger than the original size. Most image processing architectures will crop the upscaled image to its original size.

(4) Cropping: Crop the region of interest of the picture, usually during training, randomly crop out different regions and rescale them back to the original size.

(5) Translation: Translation is to move the image along the x or y direction (or both directions). When we pan, we need to make assumptions about the background, such as assuming that it is black, etc., because part of the image is empty when panning, because objects in the picture may appear in any position, so the translation enhancement method is very useful.

(6) Adding noise: Overfitting usually occurs when the neural network learns high-frequency features (because the low-frequency features are easily learned by the neural network, and the high-frequency features can only be learned at the end) and these features It may not be helpful for the task done by the neural network, and it will affect the low frequency features, in order to eliminate the high frequency features we randomly add noise data to eliminate these features.

First, in order to train the CNN model in this embodiment, a pair of training samples needs to be constructed. Add rain lines to the above clean image by Photoshop software to obtain a composite image with rain. After obtaining clean and synthetic image pairs, the number of training sample pairs can be increased by means of flipping, rotating, scaling, and cropping, and then the model can be input for training.

Step 2, network model construction, which specifically includes two parts: a rough rainline extraction module and a fine rainline purification module.

Step 2a, the rough rainline extraction module is implemented by a simple two-layer convolutional layer and two ReLU layers, so as to realize the rough rainline extraction for images with rain;

为卷积操作，r_∈为提取的带噪雨线图像，y为合成带雨图像。Among them, E ₁ , E ₂ are two convolutional layers,

is the convolution operation, r _∈ is the extracted noisy rainline image, and y is the synthetic rainy image.

Step 2b, constructing a fine rainline purification module, including seven convolutional layers and four ReLU activation functions, for denoising the above rough (noisy) rainline image to obtain clean rainlines;

The fine rainline purification module is the core content of the present invention, and the key lies in solving three dictionaries S ₁ , S ₂ , S ₃ and their transposed dictionaries G ₁ , G ₂ , G ₃ :

Among them, f _{j, i} represents the i-th filter kernel of the j-th dictionary, f _{j, i} are three sets of filter groups, that is, j=3, which actually means three rainline dictionaries of different scales, denoted as S ₁ , S ₂ , S ₃ , the transposed dictionary is denoted as G ₁ , G ₂ , G ₃ (S ₁ , S ₂ , S ₃ , G ₁ , G ₂ , G ₃ are all implemented by the convolution layer), and c is The number of decomposed channels, z _{j, i} are the convolutional sparse coding to be solved, ||.|| ₁ , ||.|| ₂ represent the l ₁ norm and the l ₂ norm respectively, λ is the sparse penalty coefficient, this It is set to 1 in the embodiment.

After obtaining the sparse coding, the denoised rainline image can be reconstructed;

E₃为一个卷积层，r为最终恢复 的精细雨线图像；in,

E3 is a convolutional layer, and r is the final restored fine _rainline image;

For the first minimization problem, the relationship between convolution and matrix multiplication can be transformed into a traditional sparse coding problem, and under the assumption of non-negative sparse coding, the ISTA algorithm is used to solve:

为中间符号，

指阈值，t为迭代次数，S₁、S₂、S₃为不同尺度字典，对应转置字典为G₁、G₂、G₃。in,

is the middle symbol,

Refers to the threshold, t is the number of iterations, S ₁ , S ₂ , and S ₃ are dictionaries of different scales, and the corresponding transposed dictionaries are G ₁ , G ₂ , and G ₃ .

According to this, the iterative solution process of sparse coding in the fine rainline purification module can be obtained. The schematic diagram of CNN implementation is shown in Figure 2. S ₁ , S ₂ , S ₃ and G ₁ , G ₂ , G ₃ can be convolutional. layer implementation. After the rough rainline extraction module, the rough rainline image r _∈ is input into the fine rainline purification module, which is mainly divided into two steps: sparse coding solution and HR feature reconstruction. The ISTA algorithm implemented by the product will output the optimal sparse coding after a certain number of times.

均初始化为r_∈。计算

分别经过S₁、S₂、S₃的和，将此和从r_∈中减去，将上述差值分别经过G₁、G₂、G₃后再分别与

相加，得到

重复上述过程，得到

即为最终的卷积稀疏编码，其中t为迭代次数，t的取值优选为25。重建时，计算

分别经过S₁、S₂、S₃的和，此和经过ReLU 后再经过E₃即可恢复出精细的雨线图像。Finally, the entire network process is as follows: input the synthetic rain image y, and obtain the rough rain line image r _∈ after two convolutions E ₁ , E ₂ and two ReLUs.

are initialized to r _∈ . calculate

After the sum of S ₁ , S ₂ , and S ₃ respectively, the sum is subtracted from r _∈ , and the above difference is passed through G ₁ , G ₂ , and G ₃ respectively, and then added to the

add up, get

Repeat the above process to get

That is, the final convolutional sparse coding, where t is the number of iterations, and the value of t is preferably 25. When rebuilding, calculate

After the sum of S ₁ , S ₂ , and S ₃ respectively, the sum can be restored through ReLU and then through E ₃ to recover a fine rainline image.

Step 3, use the training set constructed in step 1 to train the network model. At the same time, the MSE loss function is selected in this embodiment:

Among them, Θ refers to the network model parameters, l is the index of the training data in the training set, y _l -r _l is the rain-removed image output by the network model, and the difference with the real clean image x _l is accumulated to obtain the final error. Network Optimization.

Step 4: Input the rain image to be tested into the trained network model to obtain the corresponding rain-removed image.

In the testing process, peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used as measurement standards, and the specific definitions of the two are as follows:

PSNR=10*log10(255 ² /mean(mean((XY) ² )))

SSIM=[L(X,Y) ^a ]×[C(X,Y) ^b ]×[S(X,Y) ^c ]

μ_X和μ_Y分 别代表X和Y的均值，σ_X、σ_Y和σ_XY分别代表X和Y的方差以及二者的协方差。in,

μ _X and μ _Y represent the mean of X and Y, respectively, and σ _X , σ _Y and σ _XY represent the variance and covariance of X and Y, respectively.

Among them, the higher the PSNR and SSIM values, the better the reconstruction effect.

During the test, CNN, JORDER, and DIDMDN were selected as the comparison algorithms. The visual comparison is shown in Figure 4. The patented method is easier to remove the rain lines in the image, and at the same time preserves good detail information, while the comparison algorithm has an unsatisfactory rain removal effect. Incomplete removal of rain or blurred results or even artifacts. As for quantitative indicators, two commonly used data sets (Rain12 and Rain1200) are selected as test sets. The test results are shown in Table 1. It can be seen that the method of this patent greatly improves the PSNR and SSIM of the rain removal results, which shows that effectiveness of the patented method.

Table 1 Test results

It should be understood that the above description of the embodiments is relatively detailed, and therefore should not be considered as a limitation on the protection scope of the patent of the present invention. Those of ordinary skill in the art, under the inspiration of the present invention, do not depart from the protection of the claims of the present invention. In the case of the scope of the present invention, substitutions or deformations can also be made, which all fall within the protection scope of the present invention, and the claimed protection scope of the present invention shall be subject to the appended claims.

Claims (6)

1. a single image rain removal method based on multi-scale dictionary learning, is characterized in that, comprises the steps: Step 1: According to the existing clean images, the corresponding synthetic rain images are obtained by adding rain lines, and each pair of clean images and synthetic rain images is used as a training sample pair to establish a training set; Step 2, building a network model, the network model includes a rough rainline extraction module and a fine rainline purification module, wherein the rough rainline extraction module includes two convolution layers and two ReLU activation function layers, which are used to realize pair synthesis. Rough rainline extraction from rain images; fine rainline purification module includes seven convolutional layers and four ReLU activation functions to recover fine rainline images from noisy rainline images; Step 3, use the training set constructed in step 1 to train the network model, Step 4: Input the rain image to be tested into the trained network model to obtain the corresponding rain-removed image. 2. a kind of single image rain removal method based on multi-scale dictionary learning as claimed in claim 1, is characterized in that: the concrete realization of the rough rain line extraction module in step 2 is as follows,

Among them, E ₁ , E ₂ are two convolutional layers,

is the convolution operation, r _∈ is the extracted noisy rainline image, and y is the synthetic rainy image. 3. a kind of single image rain removal method based on multi-scale dictionary learning as claimed in claim 1, is characterized in that: the concrete implementation mode of fine rain line purification module in step 2 is as follows, The fine rainline purification module includes two parts: sparse coding solution and HR feature reconstruction, wherein sparse coding solves the following minimization problem by convolution:

Among them, f _{j, i} represent the i-th filter kernel of the j-th dictionary, f _j, i are three sets of filter groups, that is, j=3, which actually means three rainline dictionaries of different scales, denoted as S ₁ , S ₂ , S ₃ , the transposed dictionary is denoted as G ₁ , G ₂ , G ₃ , where S ₁ , S ₂ , S ₃ , G ₁ , G ₂ , G ₃ are implemented by the convolution layer, and c is The number of channels to be decomposed, z _{j, i} are the convolutional sparse coding to be solved, ||.|| ₁ , ||.|| ₂ represent the l ₁ norm and the l ₂ norm, respectively, and λ is the sparse penalty coefficient; After obtaining the sparse coding, reconstruct the denoised rainline map:

in,

E3 is a convolutional layer, and r is the final restored fine _rainline image; The specific process is as follows: first

are initialized to r _∈ , and r _∈ is the noisy rainline image extracted by the feature extraction module; calculate

After the sum of S ₁ , S ₂ , and S ₃ respectively, the sum is subtracted from r _∈ , and the above difference is passed through G ₁ , G ₂ , and G ₃ respectively, and then added to the

add up, get

Repeat the above process to get

is the final convolutional sparse coding, where t is the number of iterations; during reconstruction, calculate

After the sum of S ₁ , S ₂ , and S ₃ respectively, the sum passes through ReLU and then passes through E ₃ , that is, a fine rainline image is recovered. 4. a kind of single image rain removal method based on multi-scale dictionary learning as claimed in claim 3, it is characterized in that: utilize the relation of convolution and matrix multiplication to transform formula (2) into traditional sparse coding problem, and in Under the assumption of non-negative sparse coding, the ISTA algorithm is used to solve,

in,

is the middle symbol,

refers to the threshold, and t is the number of iterations. 5. a kind of single image rain removal method based on multi-scale dictionary learning as claimed in claim 1 is characterized in that: in step 3, global residual learning is introduced into the network model, and the MSE loss function is selected to minimize This loss function is the training target, and the expression of the MSE loss function is as follows:

Among them, Θ refers to the network model parameters, l is the index of the training samples in the training set, y _l -r _l is the rain-removed image output by the network model, and the difference with the real clean image x _l is accumulated to obtain the final error, so that the final error Minimization implements optimization of the network model. 6. a kind of single image rain removal method based on multi-scale dictionary learning as claimed in claim 1 is characterized in that: adopt the means of flipping, rotating, scaling, cropping to increase the number of images in the training set in step 1, and then by Photoshop Add rain lines to each clean image to obtain a synthetic image with rain, and use the corresponding clean image and synthetic image with rain as a pair of training samples.

Images