CN111127354A - A single image rain removal method based on multi-scale dictionary learning - Google Patents

A single image rain removal method based on multi-scale dictionary learning Download PDF

Info

Publication number
CN111127354A
CN111127354A CN201911300714.3A CN201911300714A CN111127354A CN 111127354 A CN111127354 A CN 111127354A CN 201911300714 A CN201911300714 A CN 201911300714A CN 111127354 A CN111127354 A CN 111127354A
Authority
CN
China
Prior art keywords
rain
image
rainline
network model
images
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201911300714.3A
Other languages
Chinese (zh)
Other versions
CN111127354B (en
Inventor
余磊
何敬伟
袁琼雯
罗美露
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Wuhan University WHU
Original Assignee
Wuhan University WHU
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wuhan University WHU filed Critical Wuhan University WHU
Priority to CN201911300714.3A priority Critical patent/CN111127354B/en
Publication of CN111127354A publication Critical patent/CN111127354A/en
Application granted granted Critical
Publication of CN111127354B publication Critical patent/CN111127354B/en
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/28Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/73Deblurring; Sharpening
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20081Training; Learning
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20084Artificial neural networks [ANN]

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention provides a multi-scale dictionary single-image rain removing method, which comprises the following steps: step 1, taking each pair of clean images and synthesized rain-carrying images as a training sample pair, and establishing a training set; step 2, constructing a network model, wherein the network model comprises a rough rain line extraction module and a fine rain line purification module, the rough rain line extraction module comprises two convolution layers and two ReLU activation function layers, and the rough rain line extraction module is used for realizing rough rain line extraction of combined rain images; the fine rain line purification module comprises seven convolution layers and four ReLU activation functions and is used for recovering a fine rain line graph from a noisy rain line image; and 3, training the network model by using the training set constructed in the step 1, and 4, inputting the image with rain to be tested into the trained network model to obtain a corresponding rain removing image. The invention comprehensively utilizes the sparse theory and the CNN learning ability, and greatly improves the solving efficiency and the reconstruction precision of the SC problem.

Description

一种基于多尺度字典学习的单图像去雨方法A single image rain removal method based on multi-scale dictionary learning

技术领域technical field

本发明属于图像处理领域,特别涉及用字典学习的方法进行单帧图像去雨。The invention belongs to the field of image processing, and particularly relates to a method of dictionary learning to remove rain from a single frame image.

背景技术Background technique

现实中,大部分计算机视觉算法都假定输入是清晰的,然而对于大多数室外视觉系统, 例如视频监控和自动驾驶,雨天环境会严重影响成像质量,造成图像模糊、变形、可视性差 等问题,这会大大降低系统的性能,因此,有效地消除雨水对图像的影响具有重要应用价值, 特别是单张图像去雨,是去雨任务的重中之重。目前,单张图像去雨算法主要分为两类:基 于先验和基于学习的方法。In reality, most computer vision algorithms assume that the input is clear. However, for most outdoor vision systems, such as video surveillance and autonomous driving, the rainy environment will seriously affect the imaging quality, resulting in blurred images, distortions, and poor visibility. This will greatly reduce the performance of the system. Therefore, effectively eliminating the impact of rain on images has important application value, especially for a single image to remove rain, which is the top priority of the task of rain removal. Currently, single image deraining algorithms are mainly divided into two categories: prior-based and learning-based methods.

基于先验的方法往往需要事先观察雨线特性,设计特定先验信息,例如雨线在某范围内 多呈倾斜的直线或者具有低秩特性,但是此类算法的去雨性能很大程度上取决于先验信息的 选取,难以处理现实中的复杂降雨情况。Prior-based methods often need to observe the characteristics of rain lines in advance, and design specific a priori information, such as rain lines are mostly inclined straight lines in a certain range or have low-rank characteristics, but the rain removal performance of such algorithms largely depends on Due to the selection of prior information, it is difficult to deal with complex rainfall conditions in reality.

基于学习的方法是近些年的研究热点,主要包括基于卷积神经网络(CNN)的方法,此 类方法将图像去雨任务视为一个逐像素的回归任务,以整张图作为输入,去雨图作为输出, 进行端对端的训练和测试。此类方法无需手工设计先验,利用CNN强大的学习能力,让网络 自己学习雨线特征,但是此类方法中网络设计时缺少指导方案,缺少可解释性,不利于网络 的改进和提升。Learning-based methods have been a research hotspot in recent years, mainly including methods based on convolutional neural networks (CNN), which treat the image rain removal task as a pixel-by-pixel regression task, take the entire image as input, and remove the The rain map is used as output for end-to-end training and testing. Such methods do not need to design priors manually, and use the powerful learning ability of CNN to allow the network to learn the rainline features by itself. However, in such methods, there is a lack of guidance and interpretability in network design, which is not conducive to the improvement and promotion of the network.

发明内容SUMMARY OF THE INVENTION

基于上述分析,本发明的目的在于提供一种多尺度字典学习的单图像去雨方法,该方法 将稀疏先验引入CNN网络设计,大大提高了去雨效果Based on the above analysis, the object of the present invention is to provide a single image rain removal method for multi-scale dictionary learning, the method introduces sparse prior into CNN network design, which greatly improves the rain removal effect

本发明提供的一种多尺度字典单图像去雨方法,包括以下具体步骤:A method for removing rain from a single image of a multi-scale dictionary provided by the present invention includes the following specific steps:

步骤1,根据已有干净图像,通过添加雨线得到对应合成带雨图像,将每对干净图像和合 成带雨图像作为一个训练样本对,建立训练集;Step 1, according to the existing clean images, by adding rain lines to obtain corresponding synthetic images with rain, using each pair of clean images and synthetic images with rain as a pair of training samples to establish a training set;

步骤2,网络模型构建,所述网络模型包括粗糙雨线提取模块和精细雨线提纯模块,其中, 粗糙雨线提取模块包括两个卷积层和两个ReLU激活函数层,用于实现对合成带雨图像的粗 糙雨线提取;精细雨线提纯模块包括七个卷积层和四个ReLU激活函数,用于从带噪雨线图 像中恢复出精细雨线图;Step 2, building a network model, the network model includes a rough rainline extraction module and a fine rainline purification module, wherein the rough rainline extraction module includes two convolution layers and two ReLU activation function layers, which are used to realize the synthesis of Rough rainline extraction from rain images; fine rainline purification module includes seven convolutional layers and four ReLU activation functions to recover fine rainline images from noisy rainline images;

步骤3,利用步骤1中构建的训练集对网络模型进行训练,Step 3, use the training set constructed in step 1 to train the network model,

步骤4,将待测试的带雨图像输入到训练好的网络模型中,获得相应的去雨图像。Step 4: Input the rain image to be tested into the trained network model to obtain the corresponding rain-removed image.

进一步的,步骤2中的粗糙雨线提取模块的具体实现如下,Further, the specific implementation of the rough rainline extraction module in step 2 is as follows,

Figure BDA0002321719900000021
Figure BDA0002321719900000021

其中,E1,E2为两个卷积层,

Figure BDA0002321719900000022
为卷积操作,r为提取的带噪雨线图像,y为合成带雨图像。Among them, E 1 , E 2 are two convolutional layers,
Figure BDA0002321719900000022
is the convolution operation, r is the extracted noisy rainline image, and y is the synthetic rainy image.

进一步的,步骤2中精细雨线提纯模块的具体实现方式如下,Further, the specific implementation mode of the fine rainline purification module in step 2 is as follows,

所述精细雨线提纯模块包括稀疏编码求解与HR特征重建两部分,其中稀疏编码是通过卷 积方式求解如下最小化问题:Described fine rainline purification module comprises two parts of sparse coding solution and HR feature reconstruction, and wherein sparse coding is to solve following minimization problem by convolution mode:

Figure BDA0002321719900000023
Figure BDA0002321719900000023

其中,fj,i表示第j个字典的第i个滤波器核,fj,i为三组滤波器组,即j=3,实际含义为 三个不同尺度的雨线字典,记为S1、S2、S3,其转置字典记为G1、G2、G3,其中S1、S2、 S3、G1、G2、G3均由卷积层实现,c为分解的通道数,zj,i为待求解的卷积稀疏编码,||.||1, ||.||2分别表示l1范数与l2范数,λ为稀疏惩罚系数;Among them, f j, i represents the i-th filter kernel of the j-th dictionary, f j, i are three sets of filter groups, that is, j=3, which actually means three rainline dictionaries of different scales, denoted as S 1 , S 2 , S 3 , the transposed dictionary is denoted as G 1 , G 2 , G 3 , where S 1 , S 2 , S 3 , G 1 , G 2 , G 3 are implemented by the convolution layer, and c is The number of channels to be decomposed, z j, i are the convolutional sparse coding to be solved, ||.|| 1 , ||.|| 2 represent the l 1 norm and the l 2 norm, respectively, and λ is the sparse penalty coefficient;

取得稀疏编码后,重建出去噪后的雨线图:After obtaining the sparse coding, reconstruct the denoised rainline map:

Figure BDA0002321719900000024
Figure BDA0002321719900000024

其中,

Figure BDA0002321719900000025
E3为一个卷积层,r为最终恢复 的精细雨线图像;in,
Figure BDA0002321719900000025
E3 is a convolutional layer, and r is the final restored fine rainline image;

具体流程为:首先将

Figure BDA0002321719900000026
均初始化为r,r为特征提取模块提取的带噪雨线图像; 计算
Figure BDA0002321719900000027
分别经过S1、S2、S3的和,将此和从r中减去,将上述差值分别经过G1、 G2、G3后再分别与
Figure BDA0002321719900000028
相加,得到
Figure BDA0002321719900000029
重复上述过程,得到
Figure BDA00023217199000000210
即为最终的卷积稀疏编码,其中t为迭代次数;重建时,计算
Figure BDA00023217199000000211
分别经过S1、S2、S3的和,此和经过ReLU后再经过E3,即恢复出精细的雨线图像。The specific process is as follows: first
Figure BDA0002321719900000026
are initialized to r , and r is the noisy rainline image extracted by the feature extraction module;
Figure BDA0002321719900000027
After passing the sum of S 1 , S 2 , and S 3 respectively, the sum is subtracted from r , and the above difference is passed through G 1 , G 2 , and G 3 respectively, and then added to the
Figure BDA0002321719900000028
add up, get
Figure BDA0002321719900000029
Repeat the above process to get
Figure BDA00023217199000000210
is the final convolutional sparse coding, where t is the number of iterations; during reconstruction, calculate
Figure BDA00023217199000000211
After passing through the sum of S 1 , S 2 , and S 3 respectively, the sum passes through ReLU and then passes through E 3 , that is, a fine rainline image is recovered.

进一步的,利用卷积与矩阵相乘的关系将式(2)转化为传统稀疏编码问题,并且在非负 稀疏编码假设下,采用ISTA算法求解,Further, formula (2) is transformed into a traditional sparse coding problem using the relationship between convolution and matrix multiplication, and under the assumption of non-negative sparse coding, the ISTA algorithm is used to solve,

Figure BDA0002321719900000031
Figure BDA0002321719900000031

其中,

Figure BDA0002321719900000032
为中间符号,
Figure BDA0002321719900000033
指阈值,t为迭代次数。in,
Figure BDA0002321719900000032
is the middle symbol,
Figure BDA0002321719900000033
refers to the threshold, and t is the number of iterations.

进一步的,步骤3中将全局残差学习引入到网络模型中,并选择MSE损失函数,以最小 化此损失函数为训练目标,MSE损失函数的表达式如下:Further, in step 3, global residual learning is introduced into the network model, and the MSE loss function is selected to minimize this loss function as the training target. The expression of the MSE loss function is as follows:

Figure BDA0002321719900000034
Figure BDA0002321719900000034

其中,Θ是指网络模型参数,l为训练集中训练样本的索引,yl-rl为网络模型输出的去 雨图像,与真实的干净图像xl做差并累加得到最终误差,使得最终误差最小化实现网络模型 的优化。Among them, Θ refers to the network model parameters, l is the index of the training samples in the training set, y l -r l is the rain-removed image output by the network model, and the difference with the real clean image x l is accumulated to obtain the final error, so that the final error Minimization implements optimization of the network model.

进一步的,步骤1中采用翻转、旋转、缩放、裁剪的手段增加训练集中的图像数量,然 后通过Photoshop对每个干净图像添加雨线,得到合成带雨图像,将对应的干净图像和合成 带雨图像作为一个训练样本对。Further, in step 1, the number of images in the training set is increased by means of flipping, rotating, scaling, and cropping, and then rain lines are added to each clean image through Photoshop to obtain a synthetic image with rain, and the corresponding clean image and synthetic rain are added. image as a pair of training samples.

本发明提供了一种单图像去雨算法,综合利用了稀疏理论和CNN学习能力,通过求解 SC中的优化问题得到迭代公式,并用CNN实现,大大提高了SC问题的求解效率和重建精度。The invention provides a single image rain removal algorithm, which comprehensively utilizes sparse theory and CNN learning ability, obtains an iterative formula by solving an optimization problem in SC, and implements it with CNN, which greatly improves the solving efficiency and reconstruction accuracy of SC problem.

附图说明Description of drawings

图1为本发明实施例中网络模型构建的大致流程图。FIG. 1 is a general flowchart of network model construction in an embodiment of the present invention.

图2为稀疏编码求解示意图。Figure 2 is a schematic diagram of sparse coding solution.

图3为Rain12数据集中猫咪图像各算法去雨对比图。Figure 3 is a comparison diagram of rain removal algorithms for cat images in the Rain12 dataset.

图4为Rain1200数据集中森林图像各算法去雨对比图。Figure 4 is a comparison diagram of each algorithm for removing rain from forest images in the Rain1200 dataset.

具体实施方式Detailed ways

为了更清楚地了解本发明,下面具体介绍本发明技术内容。In order to understand the present invention more clearly, the following describes the technical content of the present invention in detail.

如图1所示,本发明提供的一种基于多尺度字典学习的单图像去雨方法,具体分为四个 步骤:As shown in Figure 1, a kind of single image rain removal method based on multi-scale dictionary learning provided by the invention is specifically divided into four steps:

步骤1,根据已有干净图像,通过添加雨线得到对应合成带雨图像,将每对干净图像和合 成带雨图像作为一个训练样本对,建立训练集;Step 1, according to the existing clean images, by adding rain lines to obtain corresponding synthetic images with rain, using each pair of clean images and synthetic images with rain as a pair of training samples to establish a training set;

由于训练数据集有限,需要经过数据增强方法来有效利用有限的HR图像。数据增强是扩 充数据样本规模的一种有效地方法。深度学习是一种数据驱动的方法,训练数据集越大,训 练的模型泛化能力越强。然而,实际中采集数据时,很难覆盖所有场景,而且采集数据也需 要大量成本,这就导致实际中训练集有限。如果能够根据已有数据生成各种训练数据,就能 做到更好的开源节流,这就是数据增强的目的。Due to the limited training dataset, data augmentation methods are required to effectively utilize the limited HR images. Data augmentation is an effective way to expand the size of data samples. Deep learning is a data-driven method, the larger the training data set, the stronger the generalization ability of the trained model. However, when collecting data in practice, it is difficult to cover all scenarios, and collecting data also requires a lot of cost, which leads to a limited training set in practice. If a variety of training data can be generated based on existing data, it will be possible to achieve better open-source and cost-effective savings, which is the purpose of data augmentation.

常用的数据增强技术有:Commonly used data augmentation techniques are:

(1)翻转:翻转包括水平翻转和垂直翻转。(1) Flip: Flip includes horizontal flip and vertical flip.

(2)旋转:旋转就是顺时针或者逆时针的旋转,注意在旋转的时候,最好旋转90-180°, 否则会出现尺度问题。(2) Rotation: Rotation is clockwise or counterclockwise rotation. Note that when rotating, it is best to rotate 90-180°, otherwise there will be scale problems.

(3)缩放:图像可以被放大或缩小。放大时,放大后的图像尺寸会大于原始尺寸。大多数图像处理架构会按照原始尺寸对放大后的图像进行裁切。(3) Zoom: The image can be enlarged or reduced. When zoomed in, the enlarged image size will be larger than the original size. Most image processing architectures will crop the upscaled image to its original size.

(4)裁剪:裁剪图片的感兴趣区域,通常在训练的时候,会采用随机裁剪出不同区域, 并重新放缩回原始尺寸。(4) Cropping: Crop the region of interest of the picture, usually during training, randomly crop out different regions and rescale them back to the original size.

(5)平移:平移是将图像沿着x或者y方向(或者两个方向)移动。我们在平移的时候需对背景进行假设,比如说假设为黑色等等,因为平移的时候有一部分图像是 空的,由于图片中的物体可能出现在任意的位置,所以说平移增强方法十分有用。(5) Translation: Translation is to move the image along the x or y direction (or both directions). When we pan, we need to make assumptions about the background, such as assuming that it is black, etc., because part of the image is empty when panning, because objects in the picture may appear in any position, so the translation enhancement method is very useful.

(6)添加噪声:过拟合通常发生在神经网络学习高频特征的时候(因为低频特征神经网 络很容易就可以学到,而高频特征只有在最后的时候才可以学到)而这些特征对于神经网络所做的任务可能没有帮助,而且会对低频特征产生影响,为了消除高频 特征我们随机加入噪声数据来消除这些特征。(6) Adding noise: Overfitting usually occurs when the neural network learns high-frequency features (because the low-frequency features are easily learned by the neural network, and the high-frequency features can only be learned at the end) and these features It may not be helpful for the task done by the neural network, and it will affect the low frequency features, in order to eliminate the high frequency features we randomly add noise data to eliminate these features.

首先,本实施例为了训练CNN模型,需要构造训练样本对。通过Photoshop软件对上述 干净图像添加雨线,得到合成带雨图像。在获得干净和合成图像对后,采用翻转、旋转、缩 放、裁剪的手段增加训练样本对数量,然后即可输入模型进行训练。First, in order to train the CNN model in this embodiment, a pair of training samples needs to be constructed. Add rain lines to the above clean image by Photoshop software to obtain a composite image with rain. After obtaining clean and synthetic image pairs, the number of training sample pairs can be increased by means of flipping, rotating, scaling, and cropping, and then the model can be input for training.

步骤2,网络模型构建,具体包括粗糙雨线提取模块和精细雨线提纯模块两部分。Step 2, network model construction, which specifically includes two parts: a rough rainline extraction module and a fine rainline purification module.

步骤2a,粗糙雨线提取模块由简单的两层卷积层和两个ReLU层实现,实现对带雨图像 的粗糙雨线提取;Step 2a, the rough rainline extraction module is implemented by a simple two-layer convolutional layer and two ReLU layers, so as to realize the rough rainline extraction for images with rain;

Figure BDA0002321719900000041
Figure BDA0002321719900000041

其中,E1,E2为两个卷积层,

Figure BDA0002321719900000051
为卷积操作,r为提取的带噪雨线图像,y为合成带雨图像。Among them, E 1 , E 2 are two convolutional layers,
Figure BDA0002321719900000051
is the convolution operation, r is the extracted noisy rainline image, and y is the synthetic rainy image.

步骤2b,构建精细雨线提纯模块,包括七个卷积层,四个ReLU激活函数,用于对上述 粗糙(带噪)雨线图像进行去噪处理,得到干净的雨线;Step 2b, constructing a fine rainline purification module, including seven convolutional layers and four ReLU activation functions, for denoising the above rough (noisy) rainline image to obtain clean rainlines;

精细雨线提纯模块是本发明的核心内容,关键在于求解三个字典S1、S2、S3及其转置 字典G1、G2、G3The fine rainline purification module is the core content of the present invention, and the key lies in solving three dictionaries S 1 , S 2 , S 3 and their transposed dictionaries G 1 , G 2 , G 3 :

Figure BDA0002321719900000052
Figure BDA0002321719900000052

其中,fj,i表示第j个字典的第i个滤波器核,fj,i为三组滤波器组,即j=3,实际含义为 三个不同尺度的雨线字典,记为S1、S2、S3,其转置字典记为G1、G2、G3(S1、S2、S3、 G1、G2、G3均由卷积层实现),c为分解的通道数,zj,i为待求解的卷积稀疏编码,||.||1,||.||2分别表示l1范数与l2范数,λ为稀疏惩罚系数,本实施例中设置为1。Among them, f j, i represents the i-th filter kernel of the j-th dictionary, f j, i are three sets of filter groups, that is, j=3, which actually means three rainline dictionaries of different scales, denoted as S 1 , S 2 , S 3 , the transposed dictionary is denoted as G 1 , G 2 , G 3 (S 1 , S 2 , S 3 , G 1 , G 2 , G 3 are all implemented by the convolution layer), and c is The number of decomposed channels, z j, i are the convolutional sparse coding to be solved, ||.|| 1 , ||.|| 2 represent the l 1 norm and the l 2 norm respectively, λ is the sparse penalty coefficient, this It is set to 1 in the embodiment.

取得稀疏编码后,即可重建出去噪后的雨线图像;After obtaining the sparse coding, the denoised rainline image can be reconstructed;

Figure BDA0002321719900000053
Figure BDA0002321719900000053

其中,

Figure BDA0002321719900000054
E3为一个卷积层,r为最终恢复 的精细雨线图像;in,
Figure BDA0002321719900000054
E3 is a convolutional layer, and r is the final restored fine rainline image;

针对于第一个最小化问题,利用卷积与矩阵相乘的关系可以转化为传统稀疏编码问题, 并且在非负稀疏编码假设下,采用ISTA算法求解:For the first minimization problem, the relationship between convolution and matrix multiplication can be transformed into a traditional sparse coding problem, and under the assumption of non-negative sparse coding, the ISTA algorithm is used to solve:

Figure BDA0002321719900000055
Figure BDA0002321719900000055

其中,

Figure BDA0002321719900000056
为中间符号,
Figure BDA0002321719900000057
指阈值,t为迭代次数,S1、S2、S3为不同尺度字典,对应转置字典为G1、G2、G3。in,
Figure BDA0002321719900000056
is the middle symbol,
Figure BDA0002321719900000057
Refers to the threshold, t is the number of iterations, S 1 , S 2 , and S 3 are dictionaries of different scales, and the corresponding transposed dictionaries are G 1 , G 2 , and G 3 .

据此,可以得到精细雨线提纯模块中的稀疏编码迭代求解过程,CNN实现示意图如附图 2所示,S1、S2、S3和G1、G2、G3均可通过卷积层实现。经过粗糙雨线提取模块后,粗糙 雨线图像r被输入进精细雨线提纯模块部分,该部分主要分为两步:稀疏编码求解与HR特 征重建,求解稀疏编码对应图2,通过迭代卷积实现的ISTA算法,经过一定次数后将输出最 优稀疏编码。According to this, the iterative solution process of sparse coding in the fine rainline purification module can be obtained. The schematic diagram of CNN implementation is shown in Figure 2. S 1 , S 2 , S 3 and G 1 , G 2 , G 3 can be convolutional. layer implementation. After the rough rainline extraction module, the rough rainline image r is input into the fine rainline purification module, which is mainly divided into two steps: sparse coding solution and HR feature reconstruction. The ISTA algorithm implemented by the product will output the optimal sparse coding after a certain number of times.

最终,整个网络流程为:输入合成带雨图像y,经过两个卷积E1,E2和两个ReLU得到粗糙雨线图像r,将

Figure BDA0002321719900000061
均初始化为r。计算
Figure BDA0002321719900000062
分别经过S1、S2、S3的和,将此和从r中减去,将上述差值分别经过G1、G2、G3后再分别与
Figure BDA0002321719900000063
相加,得到
Figure BDA0002321719900000064
重复上述过程,得到
Figure BDA0002321719900000065
即为最终的卷积稀疏编码,其中t为迭代次数,t的取值优选为25。重建时,计算
Figure BDA0002321719900000066
分别经过S1、S2、S3的和,此和经过ReLU 后再经过E3即可恢复出精细的雨线图像。Finally, the entire network process is as follows: input the synthetic rain image y, and obtain the rough rain line image r after two convolutions E 1 , E 2 and two ReLUs.
Figure BDA0002321719900000061
are initialized to r . calculate
Figure BDA0002321719900000062
After the sum of S 1 , S 2 , and S 3 respectively, the sum is subtracted from r , and the above difference is passed through G 1 , G 2 , and G 3 respectively, and then added to the
Figure BDA0002321719900000063
add up, get
Figure BDA0002321719900000064
Repeat the above process to get
Figure BDA0002321719900000065
That is, the final convolutional sparse coding, where t is the number of iterations, and the value of t is preferably 25. When rebuilding, calculate
Figure BDA0002321719900000066
After the sum of S 1 , S 2 , and S 3 respectively, the sum can be restored through ReLU and then through E 3 to recover a fine rainline image.

步骤3,利用步骤1中构建的训练集对网络模型进行训练。同时,本实施例中选择MSE 损失函数:Step 3, use the training set constructed in step 1 to train the network model. At the same time, the MSE loss function is selected in this embodiment:

Figure BDA0002321719900000067
Figure BDA0002321719900000067

其中,Θ是指网络模型参数,l为训练集中训练数据的索引,yl-rl为网络模型输出的去 雨图像,与真实的干净图像xl做差并累加得到最终误差,据此进行网络优化。Among them, Θ refers to the network model parameters, l is the index of the training data in the training set, y l -r l is the rain-removed image output by the network model, and the difference with the real clean image x l is accumulated to obtain the final error. Network Optimization.

步骤4,将待测试的带雨图像输入到训练好的网络模型中,获得相应的去雨图像。Step 4: Input the rain image to be tested into the trained network model to obtain the corresponding rain-removed image.

测试过程中采用峰值信噪比(PSNR)和结构相似度(SSIM)作为衡量标准,二者具体定 义如下:In the testing process, peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used as measurement standards, and the specific definitions of the two are as follows:

PSNR=10*log10(2552/mean(mean((X-Y)2)))PSNR=10*log10(255 2 /mean(mean((XY) 2 )))

SSIM=[L(X,Y)a]×[C(X,Y)b]×[S(X,Y)c]SSIM=[L(X,Y) a ]×[C(X,Y) b ]×[S(X,Y) c ]

其中,

Figure BDA0002321719900000068
μX和μY分 别代表X和Y的均值,σX、σY和σXY分别代表X和Y的方差以及二者的协方差。in,
Figure BDA0002321719900000068
μ X and μ Y represent the mean of X and Y, respectively, and σ X , σ Y and σ XY represent the variance and covariance of X and Y, respectively.

其中,PSNR与SSIM数值越高,则说明重建效果越好。Among them, the higher the PSNR and SSIM values, the better the reconstruction effect.

测试过程中,选择CNN、JORDER与DIDMDN作为对比算法,视觉对比如附图4所示, 本专利方法更容易去除图像中的雨线,同时保存良好细节信息,而对比算法去雨效果不理想,去雨不完全或产生模糊结果,甚至产生伪影。至于定量指标,则选择常用的两个数据集(Rain12 和Rain1200)作为测试集,测试结果如表1所示,可以看出:本专利的方法大幅度提升去雨 结果的PSNR和SSIM,说明了本专利方法的有效性。During the test, CNN, JORDER, and DIDMDN were selected as the comparison algorithms. The visual comparison is shown in Figure 4. The patented method is easier to remove the rain lines in the image, and at the same time preserves good detail information, while the comparison algorithm has an unsatisfactory rain removal effect. Incomplete removal of rain or blurred results or even artifacts. As for quantitative indicators, two commonly used data sets (Rain12 and Rain1200) are selected as test sets. The test results are shown in Table 1. It can be seen that the method of this patent greatly improves the PSNR and SSIM of the rain removal results, which shows that effectiveness of the patented method.

表1测试结果Table 1 Test results

Figure BDA0002321719900000071
Figure BDA0002321719900000071

应当理解的是,上述针对实施例的描述较为详细,并不能因此而认为是对本发明专利保 护范围的限制,本领域的普通技术人员在本发明的启示下,在不脱离本发明权利要求所保护 的范围情况下,还可以做出替换或变形,均落入本发明的保护范围之内,本发明的请求保护 范围应以所附权利要求为准。It should be understood that the above description of the embodiments is relatively detailed, and therefore should not be considered as a limitation on the protection scope of the patent of the present invention. Those of ordinary skill in the art, under the inspiration of the present invention, do not depart from the protection of the claims of the present invention. In the case of the scope of the present invention, substitutions or deformations can also be made, which all fall within the protection scope of the present invention, and the claimed protection scope of the present invention shall be subject to the appended claims.

Claims (6)

1.一种基于多尺度字典学习的单图像去雨方法,其特征在于,包括如下步骤:1. a single image rain removal method based on multi-scale dictionary learning, is characterized in that, comprises the steps: 步骤1,根据已有干净图像,通过添加雨线得到对应合成带雨图像,将每对干净图像和合成带雨图像作为一个训练样本对,建立训练集;Step 1: According to the existing clean images, the corresponding synthetic rain images are obtained by adding rain lines, and each pair of clean images and synthetic rain images is used as a training sample pair to establish a training set; 步骤2,网络模型构建,所述网络模型包括粗糙雨线提取模块和精细雨线提纯模块,其中,粗糙雨线提取模块包括两个卷积层和两个ReLU激活函数层,用于实现对合成带雨图像的粗糙雨线提取;精细雨线提纯模块包括七个卷积层和四个ReLU激活函数,用于从带噪雨线图像中恢复出精细雨线图;Step 2, building a network model, the network model includes a rough rainline extraction module and a fine rainline purification module, wherein the rough rainline extraction module includes two convolution layers and two ReLU activation function layers, which are used to realize pair synthesis. Rough rainline extraction from rain images; fine rainline purification module includes seven convolutional layers and four ReLU activation functions to recover fine rainline images from noisy rainline images; 步骤3,利用步骤1中构建的训练集对网络模型进行训练,Step 3, use the training set constructed in step 1 to train the network model, 步骤4,将待测试的带雨图像输入到训练好的网络模型中,获得相应的去雨图像。Step 4: Input the rain image to be tested into the trained network model to obtain the corresponding rain-removed image. 2.如权利要求1所述的一种基于多尺度字典学习的单图像去雨方法,其特征在于:步骤2中的粗糙雨线提取模块的具体实现如下,2. a kind of single image rain removal method based on multi-scale dictionary learning as claimed in claim 1, is characterized in that: the concrete realization of the rough rain line extraction module in step 2 is as follows,
Figure FDA0002321719890000011
Figure FDA0002321719890000011
其中,E1,E2为两个卷积层,
Figure FDA0002321719890000012
为卷积操作,r为提取的带噪雨线图像,y为合成带雨图像。
Among them, E 1 , E 2 are two convolutional layers,
Figure FDA0002321719890000012
is the convolution operation, r is the extracted noisy rainline image, and y is the synthetic rainy image.
3.如权利要求1所述的一种基于多尺度字典学习的单图像去雨方法,其特征在于:步骤2中精细雨线提纯模块的具体实现方式如下,3. a kind of single image rain removal method based on multi-scale dictionary learning as claimed in claim 1, is characterized in that: the concrete implementation mode of fine rain line purification module in step 2 is as follows, 所述精细雨线提纯模块包括稀疏编码求解与HR特征重建两部分,其中稀疏编码是通过卷积方式求解如下最小化问题:The fine rainline purification module includes two parts: sparse coding solution and HR feature reconstruction, wherein sparse coding solves the following minimization problem by convolution:
Figure FDA0002321719890000013
Figure FDA0002321719890000013
其中,fj,i表示第j个字典的第i个滤波器核,fj,i为三组滤波器组,即j=3,实际含义为三个不同尺度的雨线字典,记为S1、S2、S3,其转置字典记为G1、G2、G3,其中S1、S2、S3、G1、G2、G3均由卷积层实现,c为分解的通道数,zj,i为待求解的卷积稀疏编码,||.||1,||.||2分别表示l1范数与l2范数,λ为稀疏惩罚系数;Among them, f j, i represent the i-th filter kernel of the j-th dictionary, f j, i are three sets of filter groups, that is, j=3, which actually means three rainline dictionaries of different scales, denoted as S 1 , S 2 , S 3 , the transposed dictionary is denoted as G 1 , G 2 , G 3 , where S 1 , S 2 , S 3 , G 1 , G 2 , G 3 are implemented by the convolution layer, and c is The number of channels to be decomposed, z j, i are the convolutional sparse coding to be solved, ||.|| 1 , ||.|| 2 represent the l 1 norm and the l 2 norm, respectively, and λ is the sparse penalty coefficient; 取得稀疏编码后,重建出去噪后的雨线图:After obtaining the sparse coding, reconstruct the denoised rainline map:
Figure FDA0002321719890000014
Figure FDA0002321719890000014
其中,
Figure FDA0002321719890000021
E3为一个卷积层,r为最终恢复的精细雨线图像;
in,
Figure FDA0002321719890000021
E3 is a convolutional layer, and r is the final restored fine rainline image;
具体流程为:首先将
Figure FDA0002321719890000022
均初始化为r,r为特征提取模块提取的带噪雨线图像;计算
Figure FDA0002321719890000023
分别经过S1、S2、S3的和,将此和从r中减去,将上述差值分别经过G1、G2、G3后再分别与
Figure FDA0002321719890000024
相加,得到
Figure FDA0002321719890000025
重复上述过程,得到
Figure FDA0002321719890000026
即为最终的卷积稀疏编码,其中t为迭代次数;重建时,计算
Figure FDA0002321719890000027
分别经过S1、S2、S3的和,此和经过ReLU后再经过E3,即恢复出精细的雨线图像。
The specific process is as follows: first
Figure FDA0002321719890000022
are initialized to r , and r is the noisy rainline image extracted by the feature extraction module; calculate
Figure FDA0002321719890000023
After the sum of S 1 , S 2 , and S 3 respectively, the sum is subtracted from r , and the above difference is passed through G 1 , G 2 , and G 3 respectively, and then added to the
Figure FDA0002321719890000024
add up, get
Figure FDA0002321719890000025
Repeat the above process to get
Figure FDA0002321719890000026
is the final convolutional sparse coding, where t is the number of iterations; during reconstruction, calculate
Figure FDA0002321719890000027
After the sum of S 1 , S 2 , and S 3 respectively, the sum passes through ReLU and then passes through E 3 , that is, a fine rainline image is recovered.
4.如权利要求3所述的一种基于多尺度字典学习的单图像去雨方法,其特征在于:利用卷积与矩阵相乘的关系将式(2)转化为传统稀疏编码问题,并且在非负稀疏编码假设下,采用ISTA算法求解,4. a kind of single image rain removal method based on multi-scale dictionary learning as claimed in claim 3, it is characterized in that: utilize the relation of convolution and matrix multiplication to transform formula (2) into traditional sparse coding problem, and in Under the assumption of non-negative sparse coding, the ISTA algorithm is used to solve,
Figure FDA0002321719890000028
Figure FDA0002321719890000028
其中,
Figure FDA0002321719890000029
为中间符号,
Figure FDA00023217198900000210
指阈值,t为迭代次数。
in,
Figure FDA0002321719890000029
is the middle symbol,
Figure FDA00023217198900000210
refers to the threshold, and t is the number of iterations.
5.如权利要求1所述的一种基于多尺度字典学习的单图像去雨方法,其特征在于:步骤3中将全局残差学习引入到网络模型中,并选择MSE损失函数,以最小化此损失函数为训练目标,MSE损失函数的表达式如下:5. a kind of single image rain removal method based on multi-scale dictionary learning as claimed in claim 1 is characterized in that: in step 3, global residual learning is introduced into the network model, and the MSE loss function is selected to minimize This loss function is the training target, and the expression of the MSE loss function is as follows:
Figure FDA00023217198900000211
Figure FDA00023217198900000211
其中,Θ是指网络模型参数,l为训练集中训练样本的索引,yl-rl为网络模型输出的去雨图像,与真实的干净图像xl做差并累加得到最终误差,使得最终误差最小化实现网络模型的优化。Among them, Θ refers to the network model parameters, l is the index of the training samples in the training set, y l -r l is the rain-removed image output by the network model, and the difference with the real clean image x l is accumulated to obtain the final error, so that the final error Minimization implements optimization of the network model.
6.如权利要求1所述的一种基于多尺度字典学习的单图像去雨方法,其特征在于:步骤1中采用翻转、旋转、缩放、裁剪的手段增加训练集中的图像数量,然后通过Photoshop对每个干净图像添加雨线,得到合成带雨图像,将对应的干净图像和合成带雨图像作为一个训练样本对。6. a kind of single image rain removal method based on multi-scale dictionary learning as claimed in claim 1 is characterized in that: adopt the means of flipping, rotating, scaling, cropping to increase the number of images in the training set in step 1, and then by Photoshop Add rain lines to each clean image to obtain a synthetic image with rain, and use the corresponding clean image and synthetic image with rain as a pair of training samples.
CN201911300714.3A 2019-12-17 2019-12-17 Single-image rain removing method based on multi-scale dictionary learning Expired - Fee Related CN111127354B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911300714.3A CN111127354B (en) 2019-12-17 2019-12-17 Single-image rain removing method based on multi-scale dictionary learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911300714.3A CN111127354B (en) 2019-12-17 2019-12-17 Single-image rain removing method based on multi-scale dictionary learning

Publications (2)

Publication Number Publication Date
CN111127354A true CN111127354A (en) 2020-05-08
CN111127354B CN111127354B (en) 2022-07-26

Family

ID=70498195

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911300714.3A Expired - Fee Related CN111127354B (en) 2019-12-17 2019-12-17 Single-image rain removing method based on multi-scale dictionary learning

Country Status (1)

Country Link
CN (1) CN111127354B (en)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111738932A (en) * 2020-05-13 2020-10-02 合肥师范学院 A method for automatically removing rain from images taken by a vehicle-mounted camera
CN111861935A (en) * 2020-07-29 2020-10-30 天津大学 A rain removal method based on image inpainting technology
CN112712486A (en) * 2020-11-26 2021-04-27 贵州大学 Radio astronomical image reconstruction method based on activation function
CN113222888A (en) * 2021-03-19 2021-08-06 复旦大学 Textile yarn weaving size detection method based on depth texture characteristics
CN113902931A (en) * 2021-09-17 2022-01-07 淮阴工学院 Image rain removal method based on learning convolutional sparse coding
CN114119393A (en) * 2021-11-09 2022-03-01 武汉大学 A Semi-Supervised Image Rain Removal Method Based on Feature Domain Cycle Consistency
CN115131452A (en) * 2022-04-19 2022-09-30 腾讯医疗健康(深圳)有限公司 Image processing method and device for artifact removal
CN117333745A (en) * 2023-10-10 2024-01-02 南京工程学院 Precipitation quantitative estimation method for monitoring audio-visual data fusion
EP4379650A4 (en) * 2021-07-28 2024-10-30 Vivo Mobile Communication Co., Ltd. Image generation method and electronic device

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110188775A1 (en) * 2010-02-01 2011-08-04 Microsoft Corporation Single Image Haze Removal Using Dark Channel Priors
CN107133935A (en) * 2017-05-25 2017-09-05 华南农业大学 A kind of fine rain removing method of single image based on depth convolutional neural networks
CN107657593A (en) * 2017-04-20 2018-02-02 湘潭大学 A kind of removing rain based on single image method
CN108765344A (en) * 2018-05-30 2018-11-06 南京信息工程大学 A method of the single image rain line removal based on depth convolutional neural networks
CN109360155A (en) * 2018-08-17 2019-02-19 上海交通大学 A single-frame image rain removal method based on multi-scale feature fusion
CN110570351A (en) * 2019-08-01 2019-12-13 武汉大学 An image super-resolution reconstruction method based on convolutional sparse coding

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110188775A1 (en) * 2010-02-01 2011-08-04 Microsoft Corporation Single Image Haze Removal Using Dark Channel Priors
CN107657593A (en) * 2017-04-20 2018-02-02 湘潭大学 A kind of removing rain based on single image method
CN107133935A (en) * 2017-05-25 2017-09-05 华南农业大学 A kind of fine rain removing method of single image based on depth convolutional neural networks
CN108765344A (en) * 2018-05-30 2018-11-06 南京信息工程大学 A method of the single image rain line removal based on depth convolutional neural networks
CN109360155A (en) * 2018-08-17 2019-02-19 上海交通大学 A single-frame image rain removal method based on multi-scale feature fusion
CN110570351A (en) * 2019-08-01 2019-12-13 武汉大学 An image super-resolution reconstruction method based on convolutional sparse coding

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
YANYAN WEI ET.AL: "A Coarse-to-fine Multi-stream Hybrid Deraining Network for Single Image Deraining", 《ARXIV》 *
郭继昌 等: "多尺度卷积神经网络的单幅图像去雨方法", 《哈尔滨工业大学学报》 *

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111738932A (en) * 2020-05-13 2020-10-02 合肥师范学院 A method for automatically removing rain from images taken by a vehicle-mounted camera
CN111861935B (en) * 2020-07-29 2022-06-03 天津大学 Rain removing method based on image restoration technology
CN111861935A (en) * 2020-07-29 2020-10-30 天津大学 A rain removal method based on image inpainting technology
CN112712486A (en) * 2020-11-26 2021-04-27 贵州大学 Radio astronomical image reconstruction method based on activation function
CN112712486B (en) * 2020-11-26 2024-04-26 贵州大学 Radio astronomical image reconstruction method based on activation function
CN113222888A (en) * 2021-03-19 2021-08-06 复旦大学 Textile yarn weaving size detection method based on depth texture characteristics
EP4379650A4 (en) * 2021-07-28 2024-10-30 Vivo Mobile Communication Co., Ltd. Image generation method and electronic device
CN113902931A (en) * 2021-09-17 2022-01-07 淮阴工学院 Image rain removal method based on learning convolutional sparse coding
CN114119393A (en) * 2021-11-09 2022-03-01 武汉大学 A Semi-Supervised Image Rain Removal Method Based on Feature Domain Cycle Consistency
CN114119393B (en) * 2021-11-09 2025-05-02 武汉大学 A semi-supervised image deraining method based on cycle consistency in feature domain
CN115131452A (en) * 2022-04-19 2022-09-30 腾讯医疗健康(深圳)有限公司 Image processing method and device for artifact removal
WO2023202265A1 (en) * 2022-04-19 2023-10-26 腾讯科技(深圳)有限公司 Image processing method and apparatus for artifact removal, and device, product and medium
CN117333745A (en) * 2023-10-10 2024-01-02 南京工程学院 Precipitation quantitative estimation method for monitoring audio-visual data fusion

Also Published As

Publication number Publication date
CN111127354B (en) 2022-07-26

Similar Documents

Publication Publication Date Title
CN111127354B (en) Single-image rain removing method based on multi-scale dictionary learning
Wei et al. 3-D quasi-recurrent neural network for hyperspectral image denoising
Liu et al. Unpaired learning for deep image deraining with rain direction regularizer
CN110415199B (en) Multispectral remote sensing image fusion method and device based on residual learning
CN109509160A (en) Hierarchical remote sensing image fusion method utilizing layer-by-layer iteration super-resolution
CN107818555B (en) Multi-dictionary remote sensing image space-time fusion method based on maximum posterior
CN111462012A (en) SAR image simulation method for generating countermeasure network based on conditions
CN105913392A (en) Degraded image overall quality improving method in complex environment
Fu et al. Ssumamba: Spatial-spectral selective state space model for hyperspectral image denoising
CN112150354B (en) A Single Image Super-Resolution Method Based on Joint Contour Enhancement and Denoising Statistical Priors
CN110570351B (en) An image super-resolution reconstruction method based on convolutional sparse coding
CN116977208A (en) Low-illumination image enhancement method for double-branch fusion
CN112927151B (en) Image denoising method based on edge enhancement and convolutional neural network
CN118154886A (en) Infrared image denoising and small target detection method for severe weather
CN114612297A (en) Hyperspectral image super-resolution reconstruction method and device
Zhang et al. A generative adversarial network approach for removing motion blur in the automatic detection of pavement cracks
Babu et al. An efficient image dahazing using Googlenet based convolution neural networks
Xiong et al. RSHazeDiff: A unified Fourier-aware diffusion model for remote sensing image dehazing
CN105590296A (en) Dual-dictionary learning-based single-frame image super-resolution reconstruction method
CN110211064B (en) Mixed degraded text image recovery method based on edge guide
Qiao et al. Mutual channel prior guided dual-domain interaction network for single image raindrop removal
Khan et al. Phaseformer: Phase-based Attention Mechanism for Underwater Image Restoration and Beyond
Ferzo et al. Image denoising techniques using unsupervised machine learning and deep learning algorithms: A review
Zhang et al. Scene recovery: Combining visual enhancement and resolution improvement
Chopade et al. Single and multi frame image super-resolution and its performance analysis: a comprehensive survey

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
CF01 Termination of patent right due to non-payment of annual fee

Granted publication date: 20220726

CF01 Termination of patent right due to non-payment of annual fee