CN111127354A - A single image rain removal method based on multi-scale dictionary learning - Google Patents
A single image rain removal method based on multi-scale dictionary learning Download PDFInfo
- Publication number
- CN111127354A CN111127354A CN201911300714.3A CN201911300714A CN111127354A CN 111127354 A CN111127354 A CN 111127354A CN 201911300714 A CN201911300714 A CN 201911300714A CN 111127354 A CN111127354 A CN 111127354A
- Authority
- CN
- China
- Prior art keywords
- rain
- image
- rainline
- network model
- images
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 35
- 238000012549 training Methods 0.000 claims abstract description 34
- 238000000605 extraction Methods 0.000 claims abstract description 18
- 238000000746 purification Methods 0.000 claims abstract description 15
- 230000006870 function Effects 0.000 claims abstract description 14
- 230000004913 activation Effects 0.000 claims abstract description 7
- 238000005457 optimization Methods 0.000 claims description 4
- 239000011159 matrix material Substances 0.000 claims description 3
- 230000015572 biosynthetic process Effects 0.000 claims description 2
- 238000003786 synthesis reaction Methods 0.000 claims description 2
- 238000013527 convolutional neural network Methods 0.000 description 9
- 238000012360 testing method Methods 0.000 description 6
- 238000013434 data augmentation Methods 0.000 description 4
- 238000013461 design Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000013528 artificial neural network Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000013519 translation Methods 0.000 description 3
- 238000010276 construction Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 241000282326 Felis catus Species 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000004091 panning Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/28—Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/73—Deblurring; Sharpening
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Engineering & Computer Science (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
Description
技术领域technical field
本发明属于图像处理领域,特别涉及用字典学习的方法进行单帧图像去雨。The invention belongs to the field of image processing, and particularly relates to a method of dictionary learning to remove rain from a single frame image.
背景技术Background technique
现实中,大部分计算机视觉算法都假定输入是清晰的,然而对于大多数室外视觉系统, 例如视频监控和自动驾驶,雨天环境会严重影响成像质量,造成图像模糊、变形、可视性差 等问题,这会大大降低系统的性能,因此,有效地消除雨水对图像的影响具有重要应用价值, 特别是单张图像去雨,是去雨任务的重中之重。目前,单张图像去雨算法主要分为两类:基 于先验和基于学习的方法。In reality, most computer vision algorithms assume that the input is clear. However, for most outdoor vision systems, such as video surveillance and autonomous driving, the rainy environment will seriously affect the imaging quality, resulting in blurred images, distortions, and poor visibility. This will greatly reduce the performance of the system. Therefore, effectively eliminating the impact of rain on images has important application value, especially for a single image to remove rain, which is the top priority of the task of rain removal. Currently, single image deraining algorithms are mainly divided into two categories: prior-based and learning-based methods.
基于先验的方法往往需要事先观察雨线特性,设计特定先验信息,例如雨线在某范围内 多呈倾斜的直线或者具有低秩特性,但是此类算法的去雨性能很大程度上取决于先验信息的 选取,难以处理现实中的复杂降雨情况。Prior-based methods often need to observe the characteristics of rain lines in advance, and design specific a priori information, such as rain lines are mostly inclined straight lines in a certain range or have low-rank characteristics, but the rain removal performance of such algorithms largely depends on Due to the selection of prior information, it is difficult to deal with complex rainfall conditions in reality.
基于学习的方法是近些年的研究热点,主要包括基于卷积神经网络(CNN)的方法,此 类方法将图像去雨任务视为一个逐像素的回归任务,以整张图作为输入,去雨图作为输出, 进行端对端的训练和测试。此类方法无需手工设计先验,利用CNN强大的学习能力,让网络 自己学习雨线特征,但是此类方法中网络设计时缺少指导方案,缺少可解释性,不利于网络 的改进和提升。Learning-based methods have been a research hotspot in recent years, mainly including methods based on convolutional neural networks (CNN), which treat the image rain removal task as a pixel-by-pixel regression task, take the entire image as input, and remove the The rain map is used as output for end-to-end training and testing. Such methods do not need to design priors manually, and use the powerful learning ability of CNN to allow the network to learn the rainline features by itself. However, in such methods, there is a lack of guidance and interpretability in network design, which is not conducive to the improvement and promotion of the network.
发明内容SUMMARY OF THE INVENTION
基于上述分析,本发明的目的在于提供一种多尺度字典学习的单图像去雨方法,该方法 将稀疏先验引入CNN网络设计,大大提高了去雨效果Based on the above analysis, the object of the present invention is to provide a single image rain removal method for multi-scale dictionary learning, the method introduces sparse prior into CNN network design, which greatly improves the rain removal effect
本发明提供的一种多尺度字典单图像去雨方法,包括以下具体步骤:A method for removing rain from a single image of a multi-scale dictionary provided by the present invention includes the following specific steps:
步骤1,根据已有干净图像,通过添加雨线得到对应合成带雨图像,将每对干净图像和合 成带雨图像作为一个训练样本对,建立训练集;
步骤2,网络模型构建,所述网络模型包括粗糙雨线提取模块和精细雨线提纯模块,其中, 粗糙雨线提取模块包括两个卷积层和两个ReLU激活函数层,用于实现对合成带雨图像的粗 糙雨线提取;精细雨线提纯模块包括七个卷积层和四个ReLU激活函数,用于从带噪雨线图 像中恢复出精细雨线图;Step 2, building a network model, the network model includes a rough rainline extraction module and a fine rainline purification module, wherein the rough rainline extraction module includes two convolution layers and two ReLU activation function layers, which are used to realize the synthesis of Rough rainline extraction from rain images; fine rainline purification module includes seven convolutional layers and four ReLU activation functions to recover fine rainline images from noisy rainline images;
步骤3,利用步骤1中构建的训练集对网络模型进行训练,Step 3, use the training set constructed in
步骤4,将待测试的带雨图像输入到训练好的网络模型中,获得相应的去雨图像。Step 4: Input the rain image to be tested into the trained network model to obtain the corresponding rain-removed image.
进一步的,步骤2中的粗糙雨线提取模块的具体实现如下,Further, the specific implementation of the rough rainline extraction module in step 2 is as follows,
其中,E1,E2为两个卷积层,为卷积操作,r∈为提取的带噪雨线图像,y为合成带雨图像。Among them, E 1 , E 2 are two convolutional layers, is the convolution operation, r ∈ is the extracted noisy rainline image, and y is the synthetic rainy image.
进一步的,步骤2中精细雨线提纯模块的具体实现方式如下,Further, the specific implementation mode of the fine rainline purification module in step 2 is as follows,
所述精细雨线提纯模块包括稀疏编码求解与HR特征重建两部分,其中稀疏编码是通过卷 积方式求解如下最小化问题:Described fine rainline purification module comprises two parts of sparse coding solution and HR feature reconstruction, and wherein sparse coding is to solve following minimization problem by convolution mode:
其中,fj,i表示第j个字典的第i个滤波器核,fj,i为三组滤波器组,即j=3,实际含义为 三个不同尺度的雨线字典,记为S1、S2、S3,其转置字典记为G1、G2、G3,其中S1、S2、 S3、G1、G2、G3均由卷积层实现,c为分解的通道数,zj,i为待求解的卷积稀疏编码,||.||1, ||.||2分别表示l1范数与l2范数,λ为稀疏惩罚系数;Among them, f j, i represents the i-th filter kernel of the j-th dictionary, f j, i are three sets of filter groups, that is, j=3, which actually means three rainline dictionaries of different scales, denoted as S 1 , S 2 , S 3 , the transposed dictionary is denoted as G 1 , G 2 , G 3 , where S 1 , S 2 , S 3 , G 1 , G 2 , G 3 are implemented by the convolution layer, and c is The number of channels to be decomposed, z j, i are the convolutional sparse coding to be solved, ||.|| 1 , ||.|| 2 represent the l 1 norm and the l 2 norm, respectively, and λ is the sparse penalty coefficient;
取得稀疏编码后,重建出去噪后的雨线图:After obtaining the sparse coding, reconstruct the denoised rainline map:
其中,E3为一个卷积层,r为最终恢复 的精细雨线图像;in, E3 is a convolutional layer, and r is the final restored fine rainline image;
具体流程为:首先将均初始化为r∈,r∈为特征提取模块提取的带噪雨线图像; 计算分别经过S1、S2、S3的和,将此和从r∈中减去,将上述差值分别经过G1、 G2、G3后再分别与相加,得到重复上述过程,得到即为最终的卷积稀疏编码,其中t为迭代次数;重建时,计算分别经过S1、S2、S3的和,此和经过ReLU后再经过E3,即恢复出精细的雨线图像。The specific process is as follows: first are initialized to r ∈ , and r ∈ is the noisy rainline image extracted by the feature extraction module; After passing the sum of S 1 , S 2 , and S 3 respectively, the sum is subtracted from r ∈ , and the above difference is passed through G 1 , G 2 , and G 3 respectively, and then added to the add up, get Repeat the above process to get is the final convolutional sparse coding, where t is the number of iterations; during reconstruction, calculate After passing through the sum of S 1 , S 2 , and S 3 respectively, the sum passes through ReLU and then passes through E 3 , that is, a fine rainline image is recovered.
进一步的,利用卷积与矩阵相乘的关系将式(2)转化为传统稀疏编码问题,并且在非负 稀疏编码假设下,采用ISTA算法求解,Further, formula (2) is transformed into a traditional sparse coding problem using the relationship between convolution and matrix multiplication, and under the assumption of non-negative sparse coding, the ISTA algorithm is used to solve,
其中,为中间符号,指阈值,t为迭代次数。in, is the middle symbol, refers to the threshold, and t is the number of iterations.
进一步的,步骤3中将全局残差学习引入到网络模型中,并选择MSE损失函数,以最小 化此损失函数为训练目标,MSE损失函数的表达式如下:Further, in step 3, global residual learning is introduced into the network model, and the MSE loss function is selected to minimize this loss function as the training target. The expression of the MSE loss function is as follows:
其中,Θ是指网络模型参数,l为训练集中训练样本的索引,yl-rl为网络模型输出的去 雨图像,与真实的干净图像xl做差并累加得到最终误差,使得最终误差最小化实现网络模型 的优化。Among them, Θ refers to the network model parameters, l is the index of the training samples in the training set, y l -r l is the rain-removed image output by the network model, and the difference with the real clean image x l is accumulated to obtain the final error, so that the final error Minimization implements optimization of the network model.
进一步的,步骤1中采用翻转、旋转、缩放、裁剪的手段增加训练集中的图像数量,然 后通过Photoshop对每个干净图像添加雨线,得到合成带雨图像,将对应的干净图像和合成 带雨图像作为一个训练样本对。Further, in
本发明提供了一种单图像去雨算法,综合利用了稀疏理论和CNN学习能力,通过求解 SC中的优化问题得到迭代公式,并用CNN实现,大大提高了SC问题的求解效率和重建精度。The invention provides a single image rain removal algorithm, which comprehensively utilizes sparse theory and CNN learning ability, obtains an iterative formula by solving an optimization problem in SC, and implements it with CNN, which greatly improves the solving efficiency and reconstruction accuracy of SC problem.
附图说明Description of drawings
图1为本发明实施例中网络模型构建的大致流程图。FIG. 1 is a general flowchart of network model construction in an embodiment of the present invention.
图2为稀疏编码求解示意图。Figure 2 is a schematic diagram of sparse coding solution.
图3为Rain12数据集中猫咪图像各算法去雨对比图。Figure 3 is a comparison diagram of rain removal algorithms for cat images in the Rain12 dataset.
图4为Rain1200数据集中森林图像各算法去雨对比图。Figure 4 is a comparison diagram of each algorithm for removing rain from forest images in the Rain1200 dataset.
具体实施方式Detailed ways
为了更清楚地了解本发明,下面具体介绍本发明技术内容。In order to understand the present invention more clearly, the following describes the technical content of the present invention in detail.
如图1所示,本发明提供的一种基于多尺度字典学习的单图像去雨方法,具体分为四个 步骤:As shown in Figure 1, a kind of single image rain removal method based on multi-scale dictionary learning provided by the invention is specifically divided into four steps:
步骤1,根据已有干净图像,通过添加雨线得到对应合成带雨图像,将每对干净图像和合 成带雨图像作为一个训练样本对,建立训练集;
由于训练数据集有限,需要经过数据增强方法来有效利用有限的HR图像。数据增强是扩 充数据样本规模的一种有效地方法。深度学习是一种数据驱动的方法,训练数据集越大,训 练的模型泛化能力越强。然而,实际中采集数据时,很难覆盖所有场景,而且采集数据也需 要大量成本,这就导致实际中训练集有限。如果能够根据已有数据生成各种训练数据,就能 做到更好的开源节流,这就是数据增强的目的。Due to the limited training dataset, data augmentation methods are required to effectively utilize the limited HR images. Data augmentation is an effective way to expand the size of data samples. Deep learning is a data-driven method, the larger the training data set, the stronger the generalization ability of the trained model. However, when collecting data in practice, it is difficult to cover all scenarios, and collecting data also requires a lot of cost, which leads to a limited training set in practice. If a variety of training data can be generated based on existing data, it will be possible to achieve better open-source and cost-effective savings, which is the purpose of data augmentation.
常用的数据增强技术有:Commonly used data augmentation techniques are:
(1)翻转:翻转包括水平翻转和垂直翻转。(1) Flip: Flip includes horizontal flip and vertical flip.
(2)旋转:旋转就是顺时针或者逆时针的旋转,注意在旋转的时候,最好旋转90-180°, 否则会出现尺度问题。(2) Rotation: Rotation is clockwise or counterclockwise rotation. Note that when rotating, it is best to rotate 90-180°, otherwise there will be scale problems.
(3)缩放:图像可以被放大或缩小。放大时,放大后的图像尺寸会大于原始尺寸。大多数图像处理架构会按照原始尺寸对放大后的图像进行裁切。(3) Zoom: The image can be enlarged or reduced. When zoomed in, the enlarged image size will be larger than the original size. Most image processing architectures will crop the upscaled image to its original size.
(4)裁剪:裁剪图片的感兴趣区域,通常在训练的时候,会采用随机裁剪出不同区域, 并重新放缩回原始尺寸。(4) Cropping: Crop the region of interest of the picture, usually during training, randomly crop out different regions and rescale them back to the original size.
(5)平移:平移是将图像沿着x或者y方向(或者两个方向)移动。我们在平移的时候需对背景进行假设,比如说假设为黑色等等,因为平移的时候有一部分图像是 空的,由于图片中的物体可能出现在任意的位置,所以说平移增强方法十分有用。(5) Translation: Translation is to move the image along the x or y direction (or both directions). When we pan, we need to make assumptions about the background, such as assuming that it is black, etc., because part of the image is empty when panning, because objects in the picture may appear in any position, so the translation enhancement method is very useful.
(6)添加噪声:过拟合通常发生在神经网络学习高频特征的时候(因为低频特征神经网 络很容易就可以学到,而高频特征只有在最后的时候才可以学到)而这些特征对于神经网络所做的任务可能没有帮助,而且会对低频特征产生影响,为了消除高频 特征我们随机加入噪声数据来消除这些特征。(6) Adding noise: Overfitting usually occurs when the neural network learns high-frequency features (because the low-frequency features are easily learned by the neural network, and the high-frequency features can only be learned at the end) and these features It may not be helpful for the task done by the neural network, and it will affect the low frequency features, in order to eliminate the high frequency features we randomly add noise data to eliminate these features.
首先,本实施例为了训练CNN模型,需要构造训练样本对。通过Photoshop软件对上述 干净图像添加雨线,得到合成带雨图像。在获得干净和合成图像对后,采用翻转、旋转、缩 放、裁剪的手段增加训练样本对数量,然后即可输入模型进行训练。First, in order to train the CNN model in this embodiment, a pair of training samples needs to be constructed. Add rain lines to the above clean image by Photoshop software to obtain a composite image with rain. After obtaining clean and synthetic image pairs, the number of training sample pairs can be increased by means of flipping, rotating, scaling, and cropping, and then the model can be input for training.
步骤2,网络模型构建,具体包括粗糙雨线提取模块和精细雨线提纯模块两部分。Step 2, network model construction, which specifically includes two parts: a rough rainline extraction module and a fine rainline purification module.
步骤2a,粗糙雨线提取模块由简单的两层卷积层和两个ReLU层实现,实现对带雨图像 的粗糙雨线提取;Step 2a, the rough rainline extraction module is implemented by a simple two-layer convolutional layer and two ReLU layers, so as to realize the rough rainline extraction for images with rain;
其中,E1,E2为两个卷积层,为卷积操作,r∈为提取的带噪雨线图像,y为合成带雨图像。Among them, E 1 , E 2 are two convolutional layers, is the convolution operation, r ∈ is the extracted noisy rainline image, and y is the synthetic rainy image.
步骤2b,构建精细雨线提纯模块,包括七个卷积层,四个ReLU激活函数,用于对上述 粗糙(带噪)雨线图像进行去噪处理,得到干净的雨线;Step 2b, constructing a fine rainline purification module, including seven convolutional layers and four ReLU activation functions, for denoising the above rough (noisy) rainline image to obtain clean rainlines;
精细雨线提纯模块是本发明的核心内容,关键在于求解三个字典S1、S2、S3及其转置 字典G1、G2、G3:The fine rainline purification module is the core content of the present invention, and the key lies in solving three dictionaries S 1 , S 2 , S 3 and their transposed dictionaries G 1 , G 2 , G 3 :
其中,fj,i表示第j个字典的第i个滤波器核,fj,i为三组滤波器组,即j=3,实际含义为 三个不同尺度的雨线字典,记为S1、S2、S3,其转置字典记为G1、G2、G3(S1、S2、S3、 G1、G2、G3均由卷积层实现),c为分解的通道数,zj,i为待求解的卷积稀疏编码,||.||1,||.||2分别表示l1范数与l2范数,λ为稀疏惩罚系数,本实施例中设置为1。Among them, f j, i represents the i-th filter kernel of the j-th dictionary, f j, i are three sets of filter groups, that is, j=3, which actually means three rainline dictionaries of different scales, denoted as S 1 , S 2 , S 3 , the transposed dictionary is denoted as G 1 , G 2 , G 3 (S 1 , S 2 , S 3 , G 1 , G 2 , G 3 are all implemented by the convolution layer), and c is The number of decomposed channels, z j, i are the convolutional sparse coding to be solved, ||.|| 1 , ||.|| 2 represent the l 1 norm and the l 2 norm respectively, λ is the sparse penalty coefficient, this It is set to 1 in the embodiment.
取得稀疏编码后,即可重建出去噪后的雨线图像;After obtaining the sparse coding, the denoised rainline image can be reconstructed;
其中,E3为一个卷积层,r为最终恢复 的精细雨线图像;in, E3 is a convolutional layer, and r is the final restored fine rainline image;
针对于第一个最小化问题,利用卷积与矩阵相乘的关系可以转化为传统稀疏编码问题, 并且在非负稀疏编码假设下,采用ISTA算法求解:For the first minimization problem, the relationship between convolution and matrix multiplication can be transformed into a traditional sparse coding problem, and under the assumption of non-negative sparse coding, the ISTA algorithm is used to solve:
其中,为中间符号,指阈值,t为迭代次数,S1、S2、S3为不同尺度字典,对应转置字典为G1、G2、G3。in, is the middle symbol, Refers to the threshold, t is the number of iterations, S 1 , S 2 , and S 3 are dictionaries of different scales, and the corresponding transposed dictionaries are G 1 , G 2 , and G 3 .
据此,可以得到精细雨线提纯模块中的稀疏编码迭代求解过程,CNN实现示意图如附图 2所示,S1、S2、S3和G1、G2、G3均可通过卷积层实现。经过粗糙雨线提取模块后,粗糙 雨线图像r∈被输入进精细雨线提纯模块部分,该部分主要分为两步:稀疏编码求解与HR特 征重建,求解稀疏编码对应图2,通过迭代卷积实现的ISTA算法,经过一定次数后将输出最 优稀疏编码。According to this, the iterative solution process of sparse coding in the fine rainline purification module can be obtained. The schematic diagram of CNN implementation is shown in Figure 2. S 1 , S 2 , S 3 and G 1 , G 2 , G 3 can be convolutional. layer implementation. After the rough rainline extraction module, the rough rainline image r ∈ is input into the fine rainline purification module, which is mainly divided into two steps: sparse coding solution and HR feature reconstruction. The ISTA algorithm implemented by the product will output the optimal sparse coding after a certain number of times.
最终,整个网络流程为:输入合成带雨图像y,经过两个卷积E1,E2和两个ReLU得到粗糙雨线图像r∈,将均初始化为r∈。计算分别经过S1、S2、S3的和,将此和从r∈中减去,将上述差值分别经过G1、G2、G3后再分别与相加,得到重复上述过程,得到即为最终的卷积稀疏编码,其中t为迭代次数,t的取值优选为25。重建时,计算分别经过S1、S2、S3的和,此和经过ReLU 后再经过E3即可恢复出精细的雨线图像。Finally, the entire network process is as follows: input the synthetic rain image y, and obtain the rough rain line image r ∈ after two convolutions E 1 , E 2 and two ReLUs. are initialized to r ∈ . calculate After the sum of S 1 , S 2 , and S 3 respectively, the sum is subtracted from r ∈ , and the above difference is passed through G 1 , G 2 , and G 3 respectively, and then added to the add up, get Repeat the above process to get That is, the final convolutional sparse coding, where t is the number of iterations, and the value of t is preferably 25. When rebuilding, calculate After the sum of S 1 , S 2 , and S 3 respectively, the sum can be restored through ReLU and then through E 3 to recover a fine rainline image.
步骤3,利用步骤1中构建的训练集对网络模型进行训练。同时,本实施例中选择MSE 损失函数:Step 3, use the training set constructed in
其中,Θ是指网络模型参数,l为训练集中训练数据的索引,yl-rl为网络模型输出的去 雨图像,与真实的干净图像xl做差并累加得到最终误差,据此进行网络优化。Among them, Θ refers to the network model parameters, l is the index of the training data in the training set, y l -r l is the rain-removed image output by the network model, and the difference with the real clean image x l is accumulated to obtain the final error. Network Optimization.
步骤4,将待测试的带雨图像输入到训练好的网络模型中,获得相应的去雨图像。Step 4: Input the rain image to be tested into the trained network model to obtain the corresponding rain-removed image.
测试过程中采用峰值信噪比(PSNR)和结构相似度(SSIM)作为衡量标准,二者具体定 义如下:In the testing process, peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used as measurement standards, and the specific definitions of the two are as follows:
PSNR=10*log10(2552/mean(mean((X-Y)2)))PSNR=10*log10(255 2 /mean(mean((XY) 2 )))
SSIM=[L(X,Y)a]×[C(X,Y)b]×[S(X,Y)c]SSIM=[L(X,Y) a ]×[C(X,Y) b ]×[S(X,Y) c ]
其中,μX和μY分 别代表X和Y的均值,σX、σY和σXY分别代表X和Y的方差以及二者的协方差。in, μ X and μ Y represent the mean of X and Y, respectively, and σ X , σ Y and σ XY represent the variance and covariance of X and Y, respectively.
其中,PSNR与SSIM数值越高,则说明重建效果越好。Among them, the higher the PSNR and SSIM values, the better the reconstruction effect.
测试过程中,选择CNN、JORDER与DIDMDN作为对比算法,视觉对比如附图4所示, 本专利方法更容易去除图像中的雨线,同时保存良好细节信息,而对比算法去雨效果不理想,去雨不完全或产生模糊结果,甚至产生伪影。至于定量指标,则选择常用的两个数据集(Rain12 和Rain1200)作为测试集,测试结果如表1所示,可以看出:本专利的方法大幅度提升去雨 结果的PSNR和SSIM,说明了本专利方法的有效性。During the test, CNN, JORDER, and DIDMDN were selected as the comparison algorithms. The visual comparison is shown in Figure 4. The patented method is easier to remove the rain lines in the image, and at the same time preserves good detail information, while the comparison algorithm has an unsatisfactory rain removal effect. Incomplete removal of rain or blurred results or even artifacts. As for quantitative indicators, two commonly used data sets (Rain12 and Rain1200) are selected as test sets. The test results are shown in Table 1. It can be seen that the method of this patent greatly improves the PSNR and SSIM of the rain removal results, which shows that effectiveness of the patented method.
表1测试结果Table 1 Test results
应当理解的是,上述针对实施例的描述较为详细,并不能因此而认为是对本发明专利保 护范围的限制,本领域的普通技术人员在本发明的启示下,在不脱离本发明权利要求所保护 的范围情况下,还可以做出替换或变形,均落入本发明的保护范围之内,本发明的请求保护 范围应以所附权利要求为准。It should be understood that the above description of the embodiments is relatively detailed, and therefore should not be considered as a limitation on the protection scope of the patent of the present invention. Those of ordinary skill in the art, under the inspiration of the present invention, do not depart from the protection of the claims of the present invention. In the case of the scope of the present invention, substitutions or deformations can also be made, which all fall within the protection scope of the present invention, and the claimed protection scope of the present invention shall be subject to the appended claims.
Claims (6)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911300714.3A CN111127354B (en) | 2019-12-17 | 2019-12-17 | Single-image rain removing method based on multi-scale dictionary learning |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911300714.3A CN111127354B (en) | 2019-12-17 | 2019-12-17 | Single-image rain removing method based on multi-scale dictionary learning |
Publications (2)
Publication Number | Publication Date |
---|---|
CN111127354A true CN111127354A (en) | 2020-05-08 |
CN111127354B CN111127354B (en) | 2022-07-26 |
Family
ID=70498195
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201911300714.3A Expired - Fee Related CN111127354B (en) | 2019-12-17 | 2019-12-17 | Single-image rain removing method based on multi-scale dictionary learning |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN111127354B (en) |
Cited By (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111738932A (en) * | 2020-05-13 | 2020-10-02 | 合肥师范学院 | A method for automatically removing rain from images taken by a vehicle-mounted camera |
CN111861935A (en) * | 2020-07-29 | 2020-10-30 | 天津大学 | A rain removal method based on image inpainting technology |
CN112712486A (en) * | 2020-11-26 | 2021-04-27 | 贵州大学 | Radio astronomical image reconstruction method based on activation function |
CN113222888A (en) * | 2021-03-19 | 2021-08-06 | 复旦大学 | Textile yarn weaving size detection method based on depth texture characteristics |
CN113902931A (en) * | 2021-09-17 | 2022-01-07 | 淮阴工学院 | Image rain removal method based on learning convolutional sparse coding |
CN114119393A (en) * | 2021-11-09 | 2022-03-01 | 武汉大学 | A Semi-Supervised Image Rain Removal Method Based on Feature Domain Cycle Consistency |
CN115131452A (en) * | 2022-04-19 | 2022-09-30 | 腾讯医疗健康(深圳)有限公司 | Image processing method and device for artifact removal |
CN117333745A (en) * | 2023-10-10 | 2024-01-02 | 南京工程学院 | Precipitation quantitative estimation method for monitoring audio-visual data fusion |
EP4379650A4 (en) * | 2021-07-28 | 2024-10-30 | Vivo Mobile Communication Co., Ltd. | Image generation method and electronic device |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20110188775A1 (en) * | 2010-02-01 | 2011-08-04 | Microsoft Corporation | Single Image Haze Removal Using Dark Channel Priors |
CN107133935A (en) * | 2017-05-25 | 2017-09-05 | 华南农业大学 | A kind of fine rain removing method of single image based on depth convolutional neural networks |
CN107657593A (en) * | 2017-04-20 | 2018-02-02 | 湘潭大学 | A kind of removing rain based on single image method |
CN108765344A (en) * | 2018-05-30 | 2018-11-06 | 南京信息工程大学 | A method of the single image rain line removal based on depth convolutional neural networks |
CN109360155A (en) * | 2018-08-17 | 2019-02-19 | 上海交通大学 | A single-frame image rain removal method based on multi-scale feature fusion |
CN110570351A (en) * | 2019-08-01 | 2019-12-13 | 武汉大学 | An image super-resolution reconstruction method based on convolutional sparse coding |
-
2019
- 2019-12-17 CN CN201911300714.3A patent/CN111127354B/en not_active Expired - Fee Related
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20110188775A1 (en) * | 2010-02-01 | 2011-08-04 | Microsoft Corporation | Single Image Haze Removal Using Dark Channel Priors |
CN107657593A (en) * | 2017-04-20 | 2018-02-02 | 湘潭大学 | A kind of removing rain based on single image method |
CN107133935A (en) * | 2017-05-25 | 2017-09-05 | 华南农业大学 | A kind of fine rain removing method of single image based on depth convolutional neural networks |
CN108765344A (en) * | 2018-05-30 | 2018-11-06 | 南京信息工程大学 | A method of the single image rain line removal based on depth convolutional neural networks |
CN109360155A (en) * | 2018-08-17 | 2019-02-19 | 上海交通大学 | A single-frame image rain removal method based on multi-scale feature fusion |
CN110570351A (en) * | 2019-08-01 | 2019-12-13 | 武汉大学 | An image super-resolution reconstruction method based on convolutional sparse coding |
Non-Patent Citations (2)
Title |
---|
YANYAN WEI ET.AL: "A Coarse-to-fine Multi-stream Hybrid Deraining Network for Single Image Deraining", 《ARXIV》 * |
郭继昌 等: "多尺度卷积神经网络的单幅图像去雨方法", 《哈尔滨工业大学学报》 * |
Cited By (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111738932A (en) * | 2020-05-13 | 2020-10-02 | 合肥师范学院 | A method for automatically removing rain from images taken by a vehicle-mounted camera |
CN111861935B (en) * | 2020-07-29 | 2022-06-03 | 天津大学 | Rain removing method based on image restoration technology |
CN111861935A (en) * | 2020-07-29 | 2020-10-30 | 天津大学 | A rain removal method based on image inpainting technology |
CN112712486A (en) * | 2020-11-26 | 2021-04-27 | 贵州大学 | Radio astronomical image reconstruction method based on activation function |
CN112712486B (en) * | 2020-11-26 | 2024-04-26 | 贵州大学 | Radio astronomical image reconstruction method based on activation function |
CN113222888A (en) * | 2021-03-19 | 2021-08-06 | 复旦大学 | Textile yarn weaving size detection method based on depth texture characteristics |
EP4379650A4 (en) * | 2021-07-28 | 2024-10-30 | Vivo Mobile Communication Co., Ltd. | Image generation method and electronic device |
CN113902931A (en) * | 2021-09-17 | 2022-01-07 | 淮阴工学院 | Image rain removal method based on learning convolutional sparse coding |
CN114119393A (en) * | 2021-11-09 | 2022-03-01 | 武汉大学 | A Semi-Supervised Image Rain Removal Method Based on Feature Domain Cycle Consistency |
CN114119393B (en) * | 2021-11-09 | 2025-05-02 | 武汉大学 | A semi-supervised image deraining method based on cycle consistency in feature domain |
CN115131452A (en) * | 2022-04-19 | 2022-09-30 | 腾讯医疗健康(深圳)有限公司 | Image processing method and device for artifact removal |
WO2023202265A1 (en) * | 2022-04-19 | 2023-10-26 | 腾讯科技(深圳)有限公司 | Image processing method and apparatus for artifact removal, and device, product and medium |
CN117333745A (en) * | 2023-10-10 | 2024-01-02 | 南京工程学院 | Precipitation quantitative estimation method for monitoring audio-visual data fusion |
Also Published As
Publication number | Publication date |
---|---|
CN111127354B (en) | 2022-07-26 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN111127354B (en) | Single-image rain removing method based on multi-scale dictionary learning | |
Wei et al. | 3-D quasi-recurrent neural network for hyperspectral image denoising | |
Liu et al. | Unpaired learning for deep image deraining with rain direction regularizer | |
CN110415199B (en) | Multispectral remote sensing image fusion method and device based on residual learning | |
CN109509160A (en) | Hierarchical remote sensing image fusion method utilizing layer-by-layer iteration super-resolution | |
CN107818555B (en) | Multi-dictionary remote sensing image space-time fusion method based on maximum posterior | |
CN111462012A (en) | SAR image simulation method for generating countermeasure network based on conditions | |
CN105913392A (en) | Degraded image overall quality improving method in complex environment | |
Fu et al. | Ssumamba: Spatial-spectral selective state space model for hyperspectral image denoising | |
CN112150354B (en) | A Single Image Super-Resolution Method Based on Joint Contour Enhancement and Denoising Statistical Priors | |
CN110570351B (en) | An image super-resolution reconstruction method based on convolutional sparse coding | |
CN116977208A (en) | Low-illumination image enhancement method for double-branch fusion | |
CN112927151B (en) | Image denoising method based on edge enhancement and convolutional neural network | |
CN118154886A (en) | Infrared image denoising and small target detection method for severe weather | |
CN114612297A (en) | Hyperspectral image super-resolution reconstruction method and device | |
Zhang et al. | A generative adversarial network approach for removing motion blur in the automatic detection of pavement cracks | |
Babu et al. | An efficient image dahazing using Googlenet based convolution neural networks | |
Xiong et al. | RSHazeDiff: A unified Fourier-aware diffusion model for remote sensing image dehazing | |
CN105590296A (en) | Dual-dictionary learning-based single-frame image super-resolution reconstruction method | |
CN110211064B (en) | Mixed degraded text image recovery method based on edge guide | |
Qiao et al. | Mutual channel prior guided dual-domain interaction network for single image raindrop removal | |
Khan et al. | Phaseformer: Phase-based Attention Mechanism for Underwater Image Restoration and Beyond | |
Ferzo et al. | Image denoising techniques using unsupervised machine learning and deep learning algorithms: A review | |
Zhang et al. | Scene recovery: Combining visual enhancement and resolution improvement | |
Chopade et al. | Single and multi frame image super-resolution and its performance analysis: a comprehensive survey |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant | ||
CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20220726 |
|
CF01 | Termination of patent right due to non-payment of annual fee |