CN113673590A - Rain removing method, system and medium based on multi-scale hourglass dense connection network - Google Patents

Rain removing method, system and medium based on multi-scale hourglass dense connection network Download PDF

Info

Publication number
CN113673590A
CN113673590A CN202110929946.6A CN202110929946A CN113673590A CN 113673590 A CN113673590 A CN 113673590A CN 202110929946 A CN202110929946 A CN 202110929946A CN 113673590 A CN113673590 A CN 113673590A
Authority
CN
China
Prior art keywords
rain
scale
hourglass
network
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110929946.6A
Other languages
Chinese (zh)
Other versions
CN113673590B (en
Inventor
吴梦华
罗玉
凌捷
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangdong University of Technology
Original Assignee
Guangdong University of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangdong University of Technology filed Critical Guangdong University of Technology
Priority to CN202110929946.6A priority Critical patent/CN113673590B/en
Publication of CN113673590A publication Critical patent/CN113673590A/en
Application granted granted Critical
Publication of CN113673590B publication Critical patent/CN113673590B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformation in the plane of the image
    • G06T3/40—Scaling the whole image or part thereof
    • G06T3/4038—Scaling the whole image or part thereof for image mosaicing, i.e. plane images composed of plane sub-images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/40—Analysis of texture
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/32—Indexing scheme for image data processing or generation, in general involving image mosaicing

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • General Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Molecular Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Health & Medical Sciences (AREA)
  • Evolutionary Biology (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

The invention provides a rain removing method based on a multi-scale hourglass dense connection network, which comprises the steps of firstly, utilizing a multi-scale hourglass parallel structure to sample extracted and input feature images for multiple times to obtain multi-scale rain streak mapping features; secondly, adding a space and channel double-attention module which is connected in parallel into the multi-scale hourglass structure, and fusing multi-level characteristics by residual jump connection to realize self-adaptive recalibration for identifying and extracting the rainprint layer characteristic information; the invention intensively connects the multi-scale hourglass attention residual error modules to carry out mutual intensive forward transmission and characteristic fusion on the characteristics extracted among different depth network layers; finally, a global rich rainprint layer characteristic diagram is obtained through multi-level characteristic aggregation, and a clear rain-free background layer image is obtained by utilizing a rain diagram linear superposition model. The method comprehensively utilizes multi-scale feature fusion, a double-attention mechanism and hierarchical feature polymerization, improves the learning accuracy of the model features, effectively extracts and removes the rain streak layer features, and obtains clear rain removing images.

Description

Rain removing method, system and medium based on multi-scale hourglass dense connection network
Technical Field
The invention relates to the technical field of computer vision image processing, in particular to a rain removing method, system and medium based on a multi-scale hourglass dense connection network.
Background
The rain stripes in the image shot in the rainy scene often block or obscure key information of the background content of the image, and the image quality is seriously reduced, so that the working performance of other follow-up computer vision systems is further hindered, and the practical application is influenced. Therefore, the image rain-removing task is an important pre-processing work of an outdoor computer vision system, and attracts extensive attention and attention of researchers. The effective rain removing technology can usually recover the blurred rained image into a clear high-quality rainless image, so that a more accurate detection or identification result is effectively provided, the performance of a computer vision system is greatly promoted and improved, the application range is wide, and the application prospect is wide.
In recent years, various researches on a single image rain removing algorithm are proposed, and the researches are mainly divided into two types, wherein the first type is a model-based method, and further can be divided into a priori-based method and a filter-based method. The prior-based method comprises methods such as sparse discrimination, a Gaussian mixture model and low-rank representation. The filter-based approach uses edge and physical property filters to get a rainless clear background image. The second category is based on data-driven deep neural network model methods. Unlike model-based methods, data-driven methods represent the image rain removal problem as a process of learning a nonlinear function and finding appropriate parameters to map the rain print portion to the background scene. Driven by deep learning, researchers have constructed mapping functions using Convolutional Neural Networks (CNNs) or generative countermeasure networks (GANs). The CNN method directly obtains a deterministic mapping function from a rain image to a clear background, and the GANs can generate a rain-free image according to an input rain image and synthesize a clear image with visual attractiveness. Fu proposes a rain removing method named DerainNet, which inputs the high-frequency part of the image into a convolutional neural network for training and learning, but the method causes the loss of the image color and luster, and the residual trace of rain stripes still exists in the background area. Zhang et al proposed a single image de-raining algorithm based on condition-generated confrontation network (CGAN), but the visual effect of the restoration was unnatural. The RESCAN iteratively repeats rain removal in multiple stages by taking the output of the previous stage of the network as the input of the next stage, but reduces the rain removal effect because the network needs to perform conversion between images and features for multiple times.
Through the analysis, the existing rain removing algorithm has the problems of rain streak residue or excessive rain removal, unnatural recovery of detail textures and the like, so that the actual application of the single image rain removing algorithm is greatly influenced. In order to solve the problems, the invention provides a single-image rain removing algorithm based on a multi-scale hourglass dense connection network. A multi-scale hourglass dense connection network called MHAR-Net is established by densely connecting a plurality of layers of attention mechanism residual modules with different scales, wherein the multi-scale hourglass structure extracts rain streak characteristic information with various scales, and simultaneously, a channel and a space double-attention module are introduced for parallel connection, so that the multi-scale information and the characteristic attention are better utilized. Residual error jump connection in the multi-scale hourglass double attention module unit realizes shallow layer feature fusion, and rain feature representation capability is improved; in addition, the multiple multi-scale hourglass attention modules are densely connected, the characteristics of the images are fully utilized, the deep layer network can utilize the characteristics of the shallow layer, the information flow is increased, the multi-level characteristics are reused and aggregated, the fact that the details and the key information of more images are extracted quickly is guaranteed, the rain removing processing in a complex rain environment is achieved, the details and the space information of the images are restored as far as possible, and the problems of color distortion, brightness loss, excessive rain removing and the like are reduced.
Disclosure of Invention
The invention aims to solve the defects of the existing image rain removing technology, and provides a single image rain removing method based on end-to-end mapping of a multi-scale hourglass dense connection network.
The invention provides a rain removing method based on a multi-scale hourglass dense connection network, which comprises the following steps:
acquiring a synthesized image rain removal sample data set to obtain a data set for training a network model;
constructing a single image rain removal network model of end-to-end mapping of the multi-scale hourglass dense connection network, wherein the network model comprises three convolution layers with convolution kernel size of 3 and four multi-scale hourglass attention residual error module units;
preprocessing the data set to obtain a training data set, training a single image rain removal network model of end-to-end mapping of the multi-scale hourglass dense connection network by using the training data set based on a Pythrch deep learning frame, and obtaining a single image rain removal network model of end-to-end mapping of the trained multi-scale hourglass dense connection network;
inputting the rain removing image into a trained single image rain removing network model for end-to-end mapping of the multi-scale hourglass dense connection network to obtain a rain removing image.
In the scheme, the method for constructing the single-image rain-removing network model of the end-to-end mapping of the multi-scale hourglass dense connection network specifically comprises the following steps:
the conversion from the image space to the feature space is realized by forming an input layer by a convolution layer with a convolution kernel size of 3 and a LeakyReLU activation function, and the input layer is used for extracting the rain image features of a shallow layer, and the process is expressed as follows:
F0＝σ(Conv3×3(Orain))，
wherein, OrainA rain image representing the input network; conv3×3(. cndot.) represents convolution operation with convolution kernel size of 3 × 3, which is used to transform the image to feature space, F0 represents shallow feature image extracted after convolution and nonlinear operation of the image, and σ (·) is nonlinear activation function leak ReLU;
further collecting different-scale rain streak characteristics from the extracted shallow characteristic F0 by using a multi-scale hourglass structure, wherein the multi-scale hourglass structure consists of a down-sampling layer, an up-sampling layer, a pooling layer, a convolution layer and a deconvolution layer, and the mathematical expression of the multi-scale hourglass structure is as follows:
Figure BDA0003210199850000031
wherein X represents data information input to the network, Fu(X；ηu) An upsampling process is performed on the data. Fd(X；ηd) A down sampling process is performed on the input data. i, j denotes the number of layers of up and down sampling, ηdAnd ηuRepresenting the network parameters of downsampling and upsampling, respectively, HiRepresenting characteristic information obtained after hourglass network processing;
extracting and transmitting different scales of rain streak characteristics by using a multi-scale hourglass residual error structure, and introducing a space and channel double attention module which is connected in parallel to recalibrate the space context characteristic dependence and weight the channel important characteristic information at the same time so as to inhibit useless channel information;
the channel attention module extracts and processes the channel characteristic information of the characteristic diagram to obtain the correlation characteristic information and the importance degree among channels, and the channel attention characteristic diagram is shown as follows:
Figure BDA0003210199850000041
wherein AvgPool refers to an average pooling operation, MaxPool refers to a maximum pooling operation, and MLP denotes a multi-layered perceptron constituting a shared network generation channel attention map Mapc; f is the characteristic information of the input network,
Figure BDA0003210199850000042
and
Figure BDA0003210199850000043
respectively representing the feature maps after the average pooling and maximum pooling operations in the channel attention module, W1And W2Representing a network weight parameter, wherein sigma represents a sigmoid nonlinear activation function;
the spatial attention module extracts the spatial context information distribution condition of the attention image, and the spatial attention characteristic diagram is represented as follows:
Figure BDA0003210199850000044
wherein F ∈ R1×H×WIs the spatial feature distribution, F (i) is the feature of the corresponding point, C represents the corresponding channel number; f7 × 7 represents a convolution operation with a filter size of 7 × 7;
Figure BDA0003210199850000045
and
Figure BDA0003210199850000046
the characteristic diagram obtained by carrying out average pooling and maximum pooling operations on spatial attention is shown, and the size of the characteristic diagram is 1 multiplied by H multiplied by W; sigma represents a sigmoid function;
the feature images obtained by the channel attention module and the spatial attention module are merged and fused, and the process can be further expressed as follows:
Fout＝Concat([Mapc(F)，Maps(F)])，
wherein Concat is the splicing operation, Mapc(F) And Maps(F) Respectively representing the characteristic information obtained after processing by the channel attention module and the spatial attention module, FoutRepresenting output characteristic information obtained after network processing of the whole multi-scale hourglass residual error attention module unit;
the multi-scale hourglass structure and the double attention modules are connected with residual jump to form a multi-scale hourglass attention residual module unit, a residual network transfers the characteristic jump extracted from the shallow layer to a deeper network layer in a cross-layer mode, and the MHAR module is expressed as follows:
Fi＝FMHARi([L0，......，Ld]，[W0，......，Wd])，
wherein [ L ]0，......，Ld]Is the input of the convolution layer corresponding to the network, [ W ]0，......，Wd]Is a parameter of each layer of the MHAR module, FMHARiRepresenting the mapping process of the ith MHAR module;
adding a dense connection mechanism into a single-image rain-removing network model of end-to-end mapping of a multi-scale hourglass dense connection network, wherein the specific implementation is as follows:
F1＝FMHAR1(F0)+F0，
F2＝FMHAR2(F1)+F1+F0，
F3＝FMHAR3(F2)+F2+F1+F0，
F4＝FMHAR4(F3)+F3+F2+F1+F0。
conv with convolution kernel sizes of 1 × 1 and 3 × 3 using a LeakyReLU activation function1×1And Conv3×3The convolutional layer fuses the aggregated depth features, represented as:
R＝σ(Conv3×3(σ(Conv1×1(MHAR4(F3)+F3+F2+F1+F0))))，
wherein, sigma is a nonlinear activation function LeakyReLU, and R is a raingrain layer obtained by extraction and separation.
In the scheme, the training data set is used for training a single image rain removal network model mapped end to end of the multi-scale hourglass dense connection network based on the Pythrch deep learning framework, and the method specifically comprises the following steps:
inputting training data sets into the constructed single image rain-removing network model of end-to-end mapping of the multi-scale hourglass dense connection network in batches for training, and finally obtaining the pre-trained single image rain-removing network model of end-to-end mapping of the multi-scale hourglass dense connection network;
B＝O-R，
wherein, O is a rain image input into the network, R is a rain print layer image extracted and separated by the network, and B is a rain-free image generated by the rain removing network processing.
In the scheme, a training data set is used for training a single image rain removal network model of end-to-end mapping of the multi-scale hourglass dense connection network based on a Pythrch deep learning framework, and the single image rain removal network model specifically comprises reconstruction loss, SSIM loss and perception loss, and specifically comprises the following steps:
reconstruction loss the reconstruction loss is constructed based on mean square error as defined below:
Figure BDA0003210199850000061
wherein L isMSERepresenting the mean square error, H, W representing the size, i.e., height and width, of the image, C being the number of channels of the image, G (R)i(ii) a P) represents the ith rainless image output by the rain removing model, P is a network training parameter, and P is { w ═ w1，w2，......，wn；b1，b2，......，bn}，w1，...wnIs a weight parameter; b1，...，bnAs an offset vector, BiRepresenting the corresponding ith real rain-free image; f is a Frobenius norm;
the SSIM loss is calculated by the structural similarity of the images, which is expressed by the formula:
Figure BDA0003210199850000062
wherein, muxWhich represents the mean value of the image,
Figure BDA0003210199850000063
representing the variance, σ, of the imagexyRepresenting the covariance of the two images, C1And C2Is an empirical constant of the equation, the SSIM loss function is defined as:
LSSIM＝-SSIM(G(R；P)，B)，
wherein L isSSIMIs a negative function;
the perceptual loss is defined as:
Figure BDA0003210199850000071
where V represents a non-linear neural network transformation, with the goal of minimizing the distance between high-level features;
the final blending error loss function can be expressed as:
Lloss＝λ1LMSE+λ2LSSIM+λ3LP，
wherein λ is1、λ2、λ3Are parameters of the relative weights on the three components reconstruction loss, SSIM loss and perceptual loss.
In this scheme, λ1、λ2、λ3Are set to 1, 0.2, 0.04, respectively.
In a second aspect, the invention provides a rain removing system based on a multi-scale hourglass dense connection network, comprising: a memory and a processor, wherein the memory includes a program of a rain removing method based on a multi-scale hourglass dense connection network, and the program of the rain removing method based on the multi-scale hourglass dense connection network realizes the following steps when being executed by the processor:
acquiring a synthesized image rain removal sample data set to obtain a data set for training a network model;
constructing a single image rain removal network model of end-to-end mapping of the multi-scale hourglass dense connection network, wherein the network model comprises three convolution layers with convolution kernel size of 3 and four multi-scale hourglass attention residual error module units;
preprocessing the data set to obtain a training data set, training a single image rain removal network model of end-to-end mapping of the multi-scale hourglass dense connection network by using the training data set based on a Pythrch deep learning frame, and obtaining a single image rain removal network model of end-to-end mapping of the trained multi-scale hourglass dense connection network;
inputting the rain removing image into a trained single image rain removing network model for end-to-end mapping of the multi-scale hourglass dense connection network to obtain a rain removing image.
In the scheme, the training data set is used for training a single image rain removal network model mapped end to end of the multi-scale hourglass dense connection network based on the Pythrch deep learning framework, and the method specifically comprises the following steps:
inputting training data sets into the constructed single image rain-removing network model of end-to-end mapping of the multi-scale hourglass dense connection network in batches for training, and finally obtaining the pre-trained single image rain-removing network model of end-to-end mapping of the multi-scale hourglass dense connection network;
B＝O-R，
wherein, O is a rain image input into the network, R is a rain print layer image extracted and separated by the network, and B is a rain-free image generated by the rain removing network processing.
In the scheme, a training data set is used for training a single image rain removal network model of end-to-end mapping of the multi-scale hourglass dense connection network based on a Pythrch deep learning framework, and the single image rain removal network model specifically comprises reconstruction loss, SSIM loss and perception loss, and specifically comprises the following steps:
reconstruction loss the reconstruction loss is constructed based on mean square error as defined below:
Figure BDA0003210199850000081
wherein L isMSERepresenting the mean square error, H, W representing the size, i.e., height and width, of the image, C being the number of channels of the image, G (R)i(ii) a P) represents the ith rainless image output by the rain removing model, P is a network training parameter, and P is { w ═ w1，w2，......，wn；b1，b2，......，bn}，w1，...wnIs a weight parameter; b1，...，bnAs an offset vector, BiRepresenting the corresponding ith real rain-free image; f is a Frobenius norm;
the SSIM loss is calculated by the structural similarity of the images, which is expressed by the formula:
Figure BDA0003210199850000082
wherein, muxWhich represents the mean value of the image,
Figure BDA0003210199850000083
representing the variance, σ, of the imagexyRepresenting the covariance of the two images, C1And C2Is an empirical constant of the equation, the SSIM loss function is defined as:
LSSIM＝-SSIM(G(R；P)，B)，
wherein L isSSIMIs a negative function;
the perceptual loss is defined as:
Figure BDA0003210199850000091
where V represents a non-linear neural network transformation, with the goal of minimizing the distance between high-level features;
the final blending error loss function can be expressed as:
Lloss＝λ1LMSE+λ2LSSIM+λ3LP，
wherein λ is1、λ2、λ3Are parameters of the relative weights on the three components reconstruction loss, SSIM loss and perceptual loss.
In this scheme, λ1、λ2、λ3Are set to 1, 0.2, 0.04, respectively.
The third aspect of the invention provides a readable storage medium, wherein the readable storage medium comprises a rain removing method program based on a multi-scale hourglass dense connection network, and when the rain removing method program based on the multi-scale hourglass dense connection network is executed by a processor, the steps of the rain removing method based on the multi-scale hourglass dense connection network are realized.
The invention provides a single image rain removing method based on a multi-scale hourglass dense connection network, which comprises the steps of firstly, utilizing a multi-scale hourglass parallel structure to sample extracted and input feature images for multiple times, extracting feature images with different scales and obtaining multi-scale rain streak mapping features; secondly, adding space and channel double-attention modules which are connected in parallel into the multi-scale hourglass structure, and fusing multi-level characteristics by residual jump connection to form a whole multi-scale hourglass attention residual error module unit (MHAR module) so as to realize self-adaptive recalibration for identifying and extracting the rainprint layer characteristic information; in addition, the invention intensively connects the multi-scale hourglass attention residual error modules to carry out mutual intensive forward transmission and characteristic fusion on the characteristics extracted among different depth network layers; finally, a global rich rainprint layer characteristic diagram is obtained through multi-level characteristic aggregation, and a clear rain-free background layer image is obtained by utilizing a rain diagram linear superposition model. The method comprehensively utilizes multi-scale feature fusion, a double-attention mechanism and hierarchical feature polymerization, improves the learning accuracy of the model features, effectively extracts and removes the rain streak layer features, and obtains clear rain removing images.
Drawings
FIG. 1 is a single image rain removal network model framework diagram based on end-to-end mapping of a multi-scale hourglass dense connection network of the present invention;
FIG. 2 is a structural diagram of a multi-scale hourglass attention residual module unit MHAR of the rain removing method based on the multi-scale hourglass dense connection network of the invention;
FIG. 3 is a multi-scale hourglass structure diagram of the rain removal method based on the multi-scale hourglass dense connection network of the present invention;
FIG. 4 is a detail view of a channel attention structure CAM of the rain removing method based on the multi-scale hourglass dense connection network of the invention;
FIG. 5 is a detail view of a spatial attention structure SAM of the rain removing method based on the multi-scale hourglass dense connection network of the invention;
FIG. 6 is a flow chart of the rain removal method based on the multi-scale hourglass dense connection network of the present invention;
fig. 7 is a frame diagram of a rain removal system based on a multi-scale hourglass dense connection network of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The invention provides a novel single-image rain removing method based on a multi-scale hourglass dense connection network. In order to more accurately acquire and utilize the characteristics extracted from the shallow layer in the network training process, the correlation between the characteristics and the more effective multiplexing transmissibility of the characteristics, the method provides a multi-scale hourglass residual error attention module, and the modules are densely connected with one another to form the whole single-image rain removal network model architecture.
As shown in fig. 6, the invention discloses a rain removing method based on a multi-scale hourglass dense connection network, which comprises the following steps:
s102, acquiring a synthesized image rain removal sample data set to obtain a data set for training a network model;
s104, constructing a single-image rain removal network model of end-to-end mapping of the multi-scale hourglass dense connection network, wherein the network model comprises three convolution layers with convolution kernel size of 3 and four multi-scale hourglass attention residual error module units as shown in FIG. 1, and the multi-scale hourglass attention residual error module units are shown in FIG. 2;
s106, preprocessing the data set to obtain a training data set, training a single image rain removing network model of end-to-end mapping of the multi-scale hourglass dense connection network by using the training data set based on a Pythrch deep learning frame, and obtaining a single image rain removing network model of end-to-end mapping of the trained multi-scale hourglass dense connection network;
and S108, inputting the rain removing image into the trained single image rain removing network model for end-to-end mapping of the multi-scale hourglass dense connection network to obtain a rain removing image.
First, a convolution layer Conv with a convolution kernel size of 3 × 3 and a LeakyReLU activation function is used3×3Forming an input layer, converting the rain-bearing image from an image space to a feature space, extracting features of the rain image in a shallow layer,the process can be expressed as:
F0＝σ(Conv3×3(Orain))，
wherein, OrainA rain image representing the input network; conv3×3(. The) represents convolution layer with convolution kernel size of 3 x 3, and is used for transforming image into feature space and extracting shallow layer feature information F0And σ is the nonlinear activation function LeakyReLU.
The multi-scale hourglass structure is used for extracting feature graphs of different scales from an input feature image, the multi-scale hourglass structure is shown in figure 3, specifically, the feature graphs are respectively sampled for multiple times by utilizing a plurality of parallel structures of an hourglass network to obtain multi-scale mapping features, the features of different network layers are fused without being constrained by the shapes and sizes of rain marks, the feature information extracted by the network can be abstracted quickly and efficiently, and a model can learn the differences among the scales more easily. The mathematical expression for the multi-scale hourglass network module structure can be expressed as:
Figure BDA0003210199850000121
wherein X represents input data, Fu(X；ηu) An upsampling process is performed on the data. Fd(X；ηd) A down sampling process is performed on the input data. EtadAnd ηuRepresenting the network parameters of downsampling and upsampling, respectively, HiRepresenting the characteristic information obtained after the hourglass network processing. X represents the characteristic information of the input network MHAR module. For different MHAR modules, the input is the feature information extracted by the corresponding previous network. i and j represent the number of layers of up-down sampling, i.e. the number of times up-down sampling is performed. The up-down sampling times are correspondingly equal, so that the sizes of the rain image finally input into the network and the rain-free image output by the network are ensured to be unchanged.
The downsampling operation is to downsample the original input characteristic image to obtain characteristic maps with different scales, and the downsampling structure is composed of Convolution layer conversion and Batch Normalization and ReLU activation functions with Convolution kernel sizes of 3 × 3, 3 × 3 and 1 × 1 respectively. The downsampling operation may be expressed as:
Fi＝Poolingi(Convi×i(ReLU(Batchnorm(F0)))，
where i is the convolution kernel size, F0 represents shallow feature data input into the hourglass structure, and Fi is a feature image obtained by convolving with i × i convolution kernel size and downsampling Poolingi.
An upsampling operation (Upscale) upsamples the downsampled reduced-size deep feature map to recover the original-size rain image features, which can be expressed by the formula:
Figure BDA0003210199850000122
wherein the content of the first and second substances,
Figure BDA0003210199850000123
illustrating the process of the upsampling operation in the multi-scale hourglass structure,
Figure BDA0003210199850000124
representing convolution and upsampling by a convolution kernel of size i x i
Figure BDA0003210199850000125
And obtaining the characteristic image.
Finally will be
Figure BDA0003210199850000126
Input the fusion module, namely:
Figure BDA0003210199850000127
wherein, FoutA characteristic diagram showing the fused output of the multi-scale hourglass structure,
Figure BDA0003210199850000131
representing the size of the convolved kernel as i x i and upsampling
Figure BDA0003210199850000132
The resulting signature graph, H (-) represents a series of operations consisting of two sets of 3 × 3 and one set of 1 × 1 convolutions.
A double attention mechanism is introduced into the multi-scale hourglass structure, the extracted multi-scale rain streak features are utilized to pass through the double attention mechanism, and an attention module is introduced from two layers of space and a channel to omit a large amount of unimportant feature information, so that more important and useful hierarchical features are enhanced and extracted, and the calculation cost of network model training is reduced.
The multi-scale rain print features learn a weight for each feature channel through a channel attention module, wherein the weight represents the importance of a feature map, which enables a network to filter out useless feature information and focus on useful information, the channel attention module is as shown in fig. 4, and for an extracted feature map, firstly, the maximum and average pixel values on each channel of the extracted feature map are obtained to represent the channel, then the maximum and average pixel values are used as the input of a two-layer multilayer perceptron network, the weights of different channels are adaptively adjusted, then feature fusion is carried out through addition, and finally all values are normalized between 0 and 1 through a sigmoid function, so that the channel attention map is obtained:
Figure BDA0003210199850000133
wherein, MLP represents to form the shared network and generate the channel attention MapcThe multilayer sensor of (1); f is the characteristic information of the input network,
Figure BDA0003210199850000134
and
Figure BDA0003210199850000135
a feature map of the channel attention module after the average pooling and maximum pooling operations is shown, respectively. W1And W2Representing the network weight parameter. σ denotes sigmoid nonlinear activation function. Pooling is the removal of the garbage information that is,key information is retained. The pooling layer is a down-sampling layer and is mainly used for feature dimension reduction, data compression and parameter quantity reduction, overfitting is reduced, and meanwhile the fault tolerance of the model is improved. AvgPool refers to average pooling operation; average pooling is the averaging of feature points in the neighborhood of the feature map. MaxPool refers to maximal pooling operation; maximum pooling is the maximization of feature points in the neighborhood of the feature map.
The multi-scale rainprint features are processed by a spatial attention module, as shown in fig. 5, which focuses the network on a position where there is rainfall according to the difference of the correlation degree of the spatial pixel values of each feature map, so as to facilitate the network to identify and extract the rainprints in the image. In the space attention module, global averaging and maximum pooling are carried out to capture the dependency between the spatial context information of different features, and the relationship between the hierarchical features is mined by using the feature context information. The spatial attention feature map is:
Figure BDA0003210199850000141
wherein F ∈ R1×H×WIs the spatial feature distribution, F (i) is the feature of the corresponding point, C represents the corresponding channel number; f. of7 ×7Represents a convolution operation with a filter size of 7 x 7;
Figure BDA0003210199850000142
and
Figure BDA0003210199850000143
the characteristic diagram obtained by carrying out average pooling and maximum pooling operations on spatial attention is shown, and the size of the characteristic diagram is 1 multiplied by H multiplied by W; σ denotes a sigmoid function.
And further splicing the feature images obtained by the channel attention module and the spatial attention module to obtain more important information, wherein the process can be further expressed as follows:
Fout＝Concat([Mapc(F)，Maps(F)])，
wherein Concat isSplicing operation, Mapc(F) And Maps(F) Respectively representing the characteristic information obtained after processing by the channel attention module and the spatial attention module, FoutAnd the output characteristic information obtained after the network processing of the whole multi-scale hourglass residual attention module unit is represented.
Meanwhile, in order to better acquire local shallow layer information, residual dense connecting branches are added in the multi-scale hourglass structure, the original input image is subjected to feature expansion by using a convolution core of 1x1, then an amplified feature diagram after deconvolution is fused, and the multi-scale and multi-level features are fused by combining global information and local information. The residual connection enables the forward-propagating characteristics to be preserved, namely, the result of each layer is obtained on the basis of the previous result, and the connection mode not only can prevent the disappearance of the network gradient, but also is beneficial to the convergence of the network, and effectively fuses the characteristics of different sources into a comprehensive characteristic diagram. The multi-scale hourglass residual error structure and the double attention module form a whole multi-scale hourglass attention residual error module unit, namely an MHAR module, and the specific implementation can be expressed as follows:
Fi＝FMHARi([L0，......，Ld]，[W0，......，Wd])，
wherein [ L ]0，......，Ld]Is the input of the convolution layer corresponding to the network, [ W ]0，......，Wd]Is a parameter of each layer of the MHAR module, FMHARiIndicating the mapping process of the ith MHAR module.
A dense connection mechanism is added in a single image rain-removing network model mapped end to end in a multi-scale hourglass dense connection network, namely four MHAR modules are stacked and densely connected to form a deeper network to extract higher-layer abstract features and aggregate multi-layer rain print features. The specific implementation can be expressed as:
F1＝MHAR1(F0)+F0，
F2＝MHAR2(F1)+F1+F0，
F3＝MHAR3(F2)+F2+F1+F0，
F4＝MHAR4(F3)+F3+F2+F1+F0because the number of layers of the convolution depth network is deepened, rain print characteristics are lost through multi-layer repeated up-down sampling and transmission, a multi-scale hourglass notices dense connection among residual modules, in addition, shallow layer characteristics extracted from the initial convolution layer and characteristics processed by various MHRA modules are fused, so that deep layer characteristics can utilize the shallow layer characteristics, the characteristics extracted by various hierarchical networks are fully used for realizing characteristic reuse, and the flowing of information is increased to prevent the defect of key information.
Finally, Conv with convolution kernel sizes of 1 × 1 and 3 × 3 with LeakyReLU activation function is used1×1And Conv3×3And the convolutional layer fuses the polymerized depth features, and then the converged depth features are decoded and restored into a clear image without rain.
R＝σ(Conv3×3(σ(Conv1×1(MHAR4(F3)+F3+F2+F1+F0) ))) disclosed herein, a rain removal system based on a multi-scale hourglass dense connection network is disclosed, the system comprising: a memory 71 and a processor 72, wherein the memory includes a program of a rain removing method based on a multi-scale hourglass dense connection network, and when the program of the rain removing method based on the multi-scale hourglass dense connection network is executed by the processor, the following steps are realized:
acquiring a synthesized image rain removal sample data set to obtain a data set for training a network model;
constructing a single image rain removal network model of end-to-end mapping of the multi-scale hourglass dense connection network, wherein the network model comprises three convolution layers with convolution kernel size of 3 and four multi-scale hourglass attention residual error module units;
preprocessing the data set to obtain a training data set, training a single image rain removal network model of end-to-end mapping of the multi-scale hourglass dense connection network by using the training data set based on a Pythrch deep learning frame, and obtaining a single image rain removal network model of end-to-end mapping of the trained multi-scale hourglass dense connection network;
inputting the rain removing image into a trained single image rain removing network model for end-to-end mapping of the multi-scale hourglass dense connection network to obtain a rain removing image.
First, a convolution layer Conv with a convolution kernel size of 3 × 3 and a LeakyReLU activation function is used3×3And constituting an input layer, converting the rain-bearing image from an image space to a feature space, and extracting features of the rain image in a shallow layer, wherein the process can be expressed as:
F0＝σ(Conv3×3(Orain))，
wherein, OrainA rain image representing the input network; conv3×3(. The) represents convolution layer with convolution kernel size of 3 x 3, and is used for transforming image into feature space and extracting shallow layer feature information F0And σ is the nonlinear activation function LeakyReLU.
The method comprises the steps of extracting feature graphs of different scales from an input feature image by using a multi-scale hourglass structure, specifically, carrying out multiple sampling on the feature graphs by using a plurality of parallel structures of an hourglass network to obtain multi-scale mapping features, and when the features of different network layers are fused, the method is not restricted by the shapes and the sizes of rain marks, can quickly and efficiently abstract feature information extracted by the network, so that a model can learn the differences among the scales more easily. The mathematical expression for the multi-scale hourglass network module structure can be expressed as:
Figure BDA0003210199850000171
wherein X represents input data, Fu(X；ηu) An upsampling process is performed on the data. Fd(X；ηd) A down sampling process is performed on the input data. EtadAnd ηuRepresenting the network parameters of downsampling and upsampling, respectively, HiRepresenting the characteristic information obtained after the hourglass network processing. X represents the characteristic information of the input network MHAR module. For different MHAR modules, the inputs areThe characteristic information extracted from the corresponding previous network. i and j represent the number of layers of up-down sampling, i.e. the number of times up-down sampling is performed. The up-down sampling times are correspondingly equal, so that the sizes of the rain image finally input into the network and the rain-free image output by the network are ensured to be unchanged.
The downsampling operation is to downsample the original input characteristic image to obtain characteristic maps with different scales, and the downsampling structure is composed of Convolution layer conversion and Batch Normalization and ReLU activation functions with Convolution kernel sizes of 3 × 3, 3 × 3 and 1 × 1 respectively. The downsampling operation may be expressed as:
Fi＝Poolingi(Convi×i(ReLU(Batchnorm(F0)))，
where i is the convolution kernel size, F0Shallow characteristic data representing the input hourglass structure, FiConvolving and downsampling Pooling with convolution kernel size of i × iiAnd obtaining the characteristic image.
An upsampling operation (Upscale) upsamples the downsampled reduced-size deep feature map to recover the original-size rain image features, which can be expressed by the formula:
Figure BDA0003210199850000181
wherein the content of the first and second substances,
Figure BDA0003210199850000182
illustrating the process of the upsampling operation in the multi-scale hourglass structure,
Figure BDA0003210199850000183
representing convolution and upsampling by a convolution kernel of size i x i
Figure BDA0003210199850000184
And obtaining the characteristic image.
Finally will be
Figure BDA0003210199850000185
Input fusionA module, namely:
Figure BDA0003210199850000186
wherein, FoutA characteristic diagram showing the fused output of the multi-scale hourglass structure,
Figure BDA0003210199850000187
representing the size of the convolved kernel as i x i and upsampling
Figure BDA0003210199850000188
The resulting signature graph, H (-) represents a series of operations consisting of two sets of 3 × 3 and one set of 1 × 1 convolutions.
A double attention mechanism is introduced into the multi-scale hourglass structure, the extracted multi-scale rain streak features are utilized to pass through the double attention mechanism, and an attention module is introduced from two layers of space and a channel to omit a large amount of unimportant feature information, so that more important and useful hierarchical features are enhanced and extracted, and the calculation cost of network model training is reduced.
The multi-scale rain print features learn a weight for each feature channel through a channel attention module, the weight represents the importance of a feature map, the network filters useless feature information and focuses on useful information, the channel attention module firstly obtains the maximum and average pixel values on each channel of the extracted feature map to represent the channel, then uses the maximum and average pixel values as the input of a two-layer multilayer perceptron network to adaptively adjust the weights of different channels, then performs feature fusion through addition, and finally normalizes all values between 0 and 1 through a sigmoid function to obtain the channel attention map as:
Figure BDA0003210199850000189
wherein, MLP represents to form the shared network and generate the channel attention MapcThe multilayer sensor of (1); f is the characteristic information of the input networkIn the form of a capsule, the particles,
Figure BDA0003210199850000191
and
Figure BDA0003210199850000192
a feature map of the channel attention module after the average pooling and maximum pooling operations is shown, respectively. W1And W2Representing the network weight parameter. σ denotes sigmoid nonlinear activation function. Pooling is the removal of garbage and the retention of critical information. The pooling layer is a down-sampling layer and is mainly used for feature dimension reduction, data compression and parameter quantity reduction, overfitting is reduced, and meanwhile the fault tolerance of the model is improved. AvgPool refers to average pooling operation; average pooling is the averaging of feature points in the neighborhood of the feature map. MaxPool refers to maximal pooling operation; maximum pooling is the maximization of feature points in the neighborhood of the feature map.
The multi-scale rain strip features are focused on positions with rainfall by the network through the spatial attention module according to different degrees of correlation of spatial pixel values of each feature map, so that the identification and extraction of the rain strips in the images by the network are facilitated. In the space attention module, global averaging and maximum pooling are carried out to capture the dependency between the spatial context information of different features, and the relationship between the hierarchical features is mined by using the feature context information. The spatial attention feature map is:
Figure BDA0003210199850000193
wherein F ∈ R1×H×WIs the spatial feature distribution, F (i) is the feature of the corresponding point, C represents the corresponding channel number; f. of7 ×7Represents a convolution operation with a filter size of 7 x 7;
Figure BDA0003210199850000194
and
Figure BDA0003210199850000195
to show spatial attentionThe size of the characteristic diagram obtained by average pooling and maximum pooling operations is 1 × H × W; σ denotes a sigmoid function.
And further splicing the feature images obtained by the channel attention module and the spatial attention module to obtain more important information, wherein the process can be further expressed as follows:
Fout＝Concat([Mapc(F)，Maps(F)])，
wherein Concat is the splicing operation, Mapc(F) And Maps(F) Respectively representing the characteristic information obtained after processing by the channel attention module and the spatial attention module, FoutAnd the output characteristic information obtained after the network processing of the whole multi-scale hourglass residual attention module unit is represented.
Meanwhile, in order to better acquire local shallow layer information, residual dense connecting branches are added in the multi-scale hourglass structure, the original input image is subjected to feature expansion by using a convolution core of 1x1, then an amplified feature diagram after deconvolution is fused, and the multi-scale and multi-level features are fused by combining global information and local information. The residual connection enables the forward-propagating characteristics to be preserved, namely, the result of each layer is obtained on the basis of the previous result, and the connection mode not only can prevent the disappearance of the network gradient, but also is beneficial to the convergence of the network, and effectively fuses the characteristics of different sources into a comprehensive characteristic diagram. The multi-scale hourglass residual error structure and the double attention module form a whole multi-scale hourglass attention residual error module unit, namely an MHAR module, and the specific implementation can be expressed as follows:
Fi＝FMHARi([L0，......，Ld]，[W0，......，Wd])，
wherein [ L ]0，......，Ld]Is the input of the convolution layer corresponding to the network, [ W ]0，......，Wd]Is a parameter of each layer of the MHAR module, FMHARiIndicating the mapping process of the ith MHAR module.
A dense connection mechanism is added in a single image rain-removing network model mapped end to end in a multi-scale hourglass dense connection network, namely four MHAR modules are stacked and densely connected to form a deeper network to extract higher-layer abstract features and aggregate multi-layer rain print features. The specific implementation can be expressed as:
F1＝MHAR1(F0)+F0，
F2＝MHAR2(F1)+F1+F0，
F3＝MHAR3(F2)+F2+F1+F0，
F4＝MHAR4(F3)+F3+F2+F1+F0，
because the number of layers of the convolution depth network is deepened, rain print characteristics are lost through multi-layer repeated up-down sampling and transmission, a multi-scale hourglass notices dense connection among residual modules, in addition, shallow layer characteristics extracted from the initial convolution layer and characteristics processed by various MHRA modules are fused, so that deep layer characteristics can utilize the shallow layer characteristics, the characteristics extracted by various hierarchical networks are fully used for realizing characteristic reuse, and the flowing of information is increased to prevent the defect of key information.
Finally, Conv with convolution kernel sizes of 1 × 1 and 3 × 3 with LeakyReLU activation function is used1×1And Conv3×3And the convolutional layer fuses the polymerized depth features, and then the converged depth features are decoded and restored into a clear image without rain.
R＝σ(Conv3×3(σ(Conv1×1(MHAR4(F3)+F3+F2+F1+F0) And) is a nonlinear activation function LeakyReLU, and R is a raingrain layer obtained by extraction and separation.
Finally, a pre-trained end-to-end rain removing network model can be obtained, and a rain-free clear background layer image output by the model is obtained by utilizing a rain chart linear superposition model. The rain map linear overlay model can be represented as:
B＝O-R，
wherein, O is a rain image input into the network, R is a rain print layer image extracted and separated by the network, and B is a rain-free image generated by the rain removing network processing.
The above are the main process steps of the method of the present invention, and the loss optimization method employed in the present invention will be described in detail below.
The method of the invention provides a mixed optimization objective function: the reconstruction loss, the SSIM loss and the perception loss are combined, the similarity of each pixel is kept while the minimum image reconstruction loss is ensured, and in addition, the high similarity of the global image structure is kept to the greatest extent, so that the rain removing result is more accurate. Specifically, it can be defined as: l isloss＝LMSE+LSSIM+LP，
Wherein L islossIs the overall objective function of the method of the present invention. Fidelity term using mean square error LMSEAs reconstruction loss to constrain the error between the real rainless image and the network output rainless image, the accuracy of the network output rainless image is measured; l isSSIMIs a structural similarity loss function, L, for measuring the similarity of two imagesPIs a perceptual loss function that minimizes the perceptual difference between the output rain-removed image of the model and the input true rain-bearing image. λ is a scaling parameter used to adjust the relative weights on the perceptual loss component. In the experimental verification process, the method proves that the mixed optimization target function has better effect than other forms of target functions through loss function ablation research. The reconstruction loss, SSIM loss and perceptual loss are specifically defined as:
1) reconstruction loss:
Figure BDA0003210199850000221
wherein, P is a network training parameter, and P ═ { w ═ w1，w2，......，wn；b1，b2，......，bn}，w1，...wnIs a weight parameter; b1，...，bnIs an offset vector. Reconstructing the loss to constrain the error between the real rainless image and the network output rainless image, measuring the accuracy of the network output rainless image,
2) the loss of SSIM is defined as follows:
Figure BDA0003210199850000222
LSSIM＝-SSIM(G(R；P)，B)，
wherein SSIM (·) is an SSIM function for calculating the similarity between two images, G (R; P) is a rain-removing image, and B is a real rain-free image. The goal of SSIM loss is to maximize the SSIM value of the image as much as possible, so LSSIMIs a negative function.
3) Loss of perception:
Figure BDA0003210199850000223
perceptual loss to normalize the similarity between high-dimensional features of an image. Wherein G (R) is a rain-removed image obtained by the method of the invention, and B is a corresponding real rain-free image. V (-) represents a nonlinear CNN transformation operation. In the method of the present invention, the feature loss in the pre-trained VGG16 model is calculated.
Therefore, to canonically constrain the fitting ability of the network model to achieve optimization of the network model, the final hybrid error loss function can be expressed as:
Lloss＝λ1LMSE+λ2LSSIM+λ3LP，
wherein λ is1、λ2、λ3Are parameters for adjusting the relative weights on the three components reconstruction loss, SSIM loss and perceptual loss. Usually lambda1，λ2，λ3Are set to 1, 0.2, 0.04, respectively.
The third aspect of the invention provides a readable storage medium, wherein the readable storage medium comprises a rain removing method program based on a multi-scale hourglass dense connection network, and when the rain removing method program based on the multi-scale hourglass dense connection network is executed by a processor, the steps of the rain removing method based on the multi-scale hourglass dense connection network are realized.
The invention provides a single image rain removing method based on a multi-scale hourglass dense connection network, which comprises the steps of firstly, utilizing a multi-scale hourglass parallel structure to sample extracted and input feature images for multiple times, extracting feature images with different scales and obtaining multi-scale rain streak mapping features; secondly, adding space and channel double-attention modules which are connected in parallel into the multi-scale hourglass structure, and fusing multi-level characteristics by residual jump connection to form a whole multi-scale hourglass attention residual error module unit (MHAR module) so as to realize self-adaptive recalibration for identifying and extracting the rainprint layer characteristic information; in addition, the invention intensively connects the multi-scale hourglass attention residual error modules to carry out mutual intensive forward transmission and characteristic fusion on the characteristics extracted among different depth network layers; finally, a global rich rainprint layer characteristic diagram is obtained through multi-level characteristic aggregation, and a clear rain-free background layer image is obtained by utilizing a rain diagram linear superposition model. The method comprehensively utilizes multi-scale feature fusion, a double-attention mechanism and hierarchical feature polymerization, improves the learning accuracy of the model features, effectively extracts and removes the rain streak layer features, and obtains clear rain removing images.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.

Claims (10)

1. A rain removing method based on a multi-scale hourglass dense connection network is characterized by comprising the following steps:
acquiring a synthesized image rain removal sample data set to obtain a data set for training a network model;
constructing a single image rain removal network model of end-to-end mapping of the multi-scale hourglass dense connection network, wherein the network model comprises three convolution layers with convolution kernel size of 3 and four multi-scale hourglass attention residual error module units;
preprocessing the data set to obtain a training data set, training a single image rain removal network model of end-to-end mapping of the multi-scale hourglass dense connection network by using the training data set based on a Pythrch deep learning frame, and obtaining a single image rain removal network model of end-to-end mapping of the trained multi-scale hourglass dense connection network;
inputting the rain removing image into a trained single image rain removing network model for end-to-end mapping of the multi-scale hourglass dense connection network to obtain a rain removing image.
2. The rain removing method based on the multi-scale hourglass dense connection network according to claim 1, wherein the single image rain removing network model for end-to-end mapping of the multi-scale hourglass dense connection network is constructed, and specifically comprises the following steps:
the conversion from the image space to the feature space is realized by forming an input layer by a convolution layer with a convolution kernel size of 3 and a LeakyReLU activation function, and the input layer is used for extracting the rain image features of a shallow layer, and the process is expressed as follows:
F0＝σ(Conv3×3(Orain))，
wherein, OrainA rain image representing the input network; conv3×3(. -) convolution operation with convolution kernel size 3 x 3 to transform image to feature space, F0The shallow layer characteristic image is extracted after the image is subjected to convolution and nonlinear operation, and sigma (-) is a nonlinear activation function LeakyReLU;
extraction of shallow features F from an extracted using a multi-scale hourglass structure0Further gather different yardstick rain streak characteristics in, the multiscale hourglass structure comprises down-sampling, upsampling, pooling layer, convolution layer and deconvolution layer, and its mathematical expression is:
Figure FDA0003210199840000021
wherein X represents data information input to the network, Fu(X；ηu) An upsampling process is performed on the data. Fd(X；ηd) A down sampling process is performed on the input data. i, j denotes the number of layers of up and down sampling, ηdAnd ηuRepresenting the network parameters of downsampling and upsampling, respectively, HiRepresenting characteristic information obtained after hourglass network processing;
extracting and transmitting different scales of rain streak characteristics by using a multi-scale hourglass residual error structure, and introducing a space and channel double attention module which is connected in parallel to recalibrate the space context characteristic dependence and weight the channel important characteristic information at the same time so as to inhibit useless channel information;
the channel attention module extracts and processes the channel characteristic information of the characteristic diagram to obtain the correlation characteristic information and the importance degree among channels, and the channel attention characteristic diagram is shown as follows:
Figure FDA0003210199840000022
wherein, AvgPool refers to average pooling operation, MaxPool refers to maximum pooling operation, MLP indicates generation of channel attention Map for constituting shared networkcThe multilayer sensor of (1); f is the characteristic information of the input network,
Figure FDA0003210199840000023
and
Figure FDA0003210199840000024
respectively representing the feature maps after the average pooling and maximum pooling operations in the channel attention module, W1And W2Representing a network weight parameter, wherein sigma represents a sigmoid nonlinear activation function;
the spatial attention module extracts the spatial context information distribution condition of the attention image, and the spatial attention characteristic diagram is represented as follows:
Figure FDA0003210199840000025
wherein the content of the first and second substances,F∈R1×H×Wis the spatial feature distribution, F (i) is the feature of the corresponding point, C represents the corresponding channel number; f. of7×7Represents a convolution operation with a filter size of 7 x 7;
Figure FDA0003210199840000031
and
Figure FDA0003210199840000032
the characteristic diagram obtained by carrying out average pooling and maximum pooling operations on spatial attention is shown, and the size of the characteristic diagram is 1 multiplied by H multiplied by W; sigma represents a sigmoid function;
the feature images obtained by the channel attention module and the spatial attention module are merged and fused, and the process can be further expressed as follows:
Fout＝Concat([Mapc(F),Maps(F)])，
wherein Concat is the splicing operation, Mapc(F) And Maps(F) Respectively representing the characteristic information obtained after processing by the channel attention module and the spatial attention module, FoutRepresenting output characteristic information obtained after network processing of the whole multi-scale hourglass residual error attention module unit;
the multi-scale hourglass structure and the double attention modules are connected with residual jump to form a multi-scale hourglass attention residual module unit, a residual network transfers the characteristic jump extracted from the shallow layer to a deeper network layer in a cross-layer mode, and the MHAR module is expressed as follows:
Fi＝FMHARi([L0,……,Ld],[W0,……,Wd])，
wherein [ L ]0,……,Ld]Is the input of the convolution layer corresponding to the network, [ W ]0,……,Wd]Is a parameter of each layer of the MHAR module, FMHARiRepresenting the mapping process of the ith MHAR module;
adding a dense connection mechanism into a single-image rain-removing network model of end-to-end mapping of a multi-scale hourglass dense connection network, wherein the specific implementation is as follows:
F1＝FMHAR1(F0)+F0，
F2＝FMHAR2(F1)+F1+F0，
F3＝FMHAR3(F2)+F2+F1+F0，
F4＝FMHAR4(F3)+F3+F2+F1+F0。
conv with convolution kernel sizes of 1 × 1 and 3 × 3 using a LeakyReLU activation function1×1And Conv3×3The convolutional layer fuses the aggregated depth features, represented as:
R＝σ(Conv3×3(σ(Conv1×1(MHAR4(F3)+F3+F2+F1+F0))))，
wherein, sigma is a nonlinear activation function LeakyReLU, and R is a raingrain layer obtained by extraction and separation.
3. The method according to claim 1, wherein a Pytorch deep learning framework is used to train a single-image rain removing network model of end-to-end mapping of the multi-scale hourglass dense connection network with the training data set, and specifically comprises:
inputting training data sets into the constructed single image rain-removing network model of end-to-end mapping of the multi-scale hourglass dense connection network in batches for training, and finally obtaining the pre-trained single image rain-removing network model of end-to-end mapping of the multi-scale hourglass dense connection network;
B＝O-R，
wherein, O is a rain image input into the network, R is a rain print layer image extracted and separated by the network, and B is a rain-free image generated by the rain removing network processing.
4. The method according to claim 1, wherein a Pytorch deep learning framework is used to train a single-image rain removing network model of end-to-end mapping of the multi-scale hourglass dense connection network with the training data set, and the single-image rain removing network model specifically comprises reconstruction loss, SSIM loss, and perceptual loss, and specifically comprises:
reconstruction loss the reconstruction loss is constructed based on mean square error as defined below:
Figure FDA0003210199840000041
wherein L isMSERepresenting the mean square error, H, W representing the size, i.e., height and width, of the image, C being the number of channels of the image, G (R)i(ii) a P) represents the ith rainless image output by the rain removing model, P is a network training parameter, and P is { w ═ w1,w2,……，wn；b1,b2,……，bn},w1,…wnIs a weight parameter; b1,…，bnAs an offset vector, BiRepresenting the corresponding ith real rain-free image; f is a Frobenius norm;
the SSIM loss is calculated by the structural similarity of the images, which is expressed by the formula:
Figure FDA0003210199840000051
wherein, muxWhich represents the mean value of the image,
Figure FDA0003210199840000052
representing the variance, σ, of the imagexyRepresenting the covariance of the two images, C1And C2Is an empirical constant of the equation, the SSIM loss function is defined as:
LSSIM＝-SSIM(G(R；P),B)，
wherein L isSSIMIs a negative function;
the perceptual loss is defined as:
Figure FDA0003210199840000053
where V represents a non-linear neural network transformation, with the goal of minimizing the distance between high-level features;
the final blending error loss function can be expressed as:
Lloss＝λ1LMSE+λ2LSSIM+λ3LP，
wherein λ is1、λ2、λ3Are parameters of the relative weights on the three components reconstruction loss, SSIM loss and perceptual loss.
5. The rain removing method based on the multi-scale hourglass dense connection network as claimed in claim 4, wherein λ1、λ2、λ3Are set to 1, 0.2, 0.04, respectively.
6. A rain removal system based on a multi-scale hourglass dense connection network, the system comprising: a memory and a processor, wherein the memory includes a program of a rain removing method based on a multi-scale hourglass dense connection network, and the program of the rain removing method based on the multi-scale hourglass dense connection network realizes the following steps when being executed by the processor:
acquiring a synthesized image rain removal sample data set to obtain a data set for training a network model;
constructing a single image rain removal network model of end-to-end mapping of the multi-scale hourglass dense connection network, wherein the network model comprises three convolution layers with convolution kernel size of 3 and four multi-scale hourglass attention residual error module units;
preprocessing the data set to obtain a training data set, training a single image rain removal network model of end-to-end mapping of the multi-scale hourglass dense connection network by using the training data set based on a Pythrch deep learning frame, and obtaining a single image rain removal network model of end-to-end mapping of the trained multi-scale hourglass dense connection network;
inputting the rain removing image into a trained single image rain removing network model for end-to-end mapping of the multi-scale hourglass dense connection network to obtain a rain removing image.
7. The rain removing system based on the multi-scale hourglass dense connection network of claim 6, wherein a Pytorch-based deep learning framework is used to train a single-image rain removing network model of end-to-end mapping of the multi-scale hourglass dense connection network with the training data set, and specifically comprises:
inputting training data sets into the constructed single image rain-removing network model of end-to-end mapping of the multi-scale hourglass dense connection network in batches for training, and finally obtaining the pre-trained single image rain-removing network model of end-to-end mapping of the multi-scale hourglass dense connection network;
B＝O-R，
wherein, O is a rain image input into the network, R is a rain print layer image extracted and separated by the network, and B is a rain-free image generated by the rain removing network processing.
8. The rain removing system based on the multi-scale hourglass dense connection network of claim 6, wherein a single-image rain removing network model for end-to-end mapping of the multi-scale hourglass dense connection network is trained by using the training data set based on a Pythrch deep learning framework, and specifically comprises reconstruction loss, SSIM loss, and sensing loss, and specifically comprises:
reconstruction loss the reconstruction loss is constructed based on mean square error as defined below:
Figure FDA0003210199840000061
wherein L isMSERepresenting the mean square error, H, W representing the size, i.e., height and width, of the image, C being the number of channels of the image, G (R)i(ii) a P) represents the ith rainless image output by the rain removing model, P is a network training parameter, and P is { w ═ w1,w2,……，wn；b1,b2,……，bn},w1,…wnIs a weight parameter; b1,…，bnAs an offset vector, BiRepresenting the corresponding ith real rain-free image; f is a Frobenius norm;
the SSIM loss is calculated by the structural similarity of the images, which is expressed by the formula:
Figure FDA0003210199840000071
wherein, muxWhich represents the mean value of the image,
Figure FDA0003210199840000072
representing the variance, σ, of the imagexyRepresenting the covariance of the two images, C1And C2Is an empirical constant of the equation, the SSIM loss function is defined as:
LSSIM＝-SSIM(G(R；P),B)，
wherein L isSSIMIs a negative function;
the perceptual loss is defined as:
Figure FDA0003210199840000073
where V represents a non-linear neural network transformation, with the goal of minimizing the distance between high-level features;
the final blending error loss function can be expressed as:
Lloss＝λ1LMSE+λ2LSSIM+λ3LP，
wherein λ is1、λ2、λ3Are parameters of the relative weights on the three components reconstruction loss, SSIM loss and perceptual loss.
9. The method of claim 8, wherein the method comprises a step of removing a dense connection network based on a multi-scale hourglassRain system, characterised by λ1、λ2、λ3Are set to 1, 0.2, 0.04, respectively.
10. Readable storage medium, characterized in that the readable storage medium comprises a rain removing method program based on multi-scale hourglass dense connection network, when the rain removing method program based on multi-scale hourglass dense connection network is executed by a processor, the steps of the rain removing method based on multi-scale hourglass dense connection network according to any one of claims 1 to 5 are realized.
CN202110929946.6A 2021-08-13 2021-08-13 Rain removing method, system and medium based on multi-scale hourglass dense connection network Active CN113673590B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110929946.6A CN113673590B (en) 2021-08-13 2021-08-13 Rain removing method, system and medium based on multi-scale hourglass dense connection network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110929946.6A CN113673590B (en) 2021-08-13 2021-08-13 Rain removing method, system and medium based on multi-scale hourglass dense connection network

Publications (2)

Publication Number Publication Date
CN113673590A true CN113673590A (en) 2021-11-19
CN113673590B CN113673590B (en) 2022-12-23

Family

ID=78542716

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110929946.6A Active CN113673590B (en) 2021-08-13 2021-08-13 Rain removing method, system and medium based on multi-scale hourglass dense connection network

Country Status (1)

Country Link
CN (1) CN113673590B (en)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114066959A (en) * 2021-11-25 2022-02-18 天津工业大学 Single-stripe image depth estimation method based on Transformer
CN114492522A (en) * 2022-01-24 2022-05-13 四川大学 Automatic modulation classification method based on improved stacked hourglass neural network
CN114638768A (en) * 2022-05-19 2022-06-17 武汉大学 Image rain removing method, system and equipment based on dynamic association learning network
CN114677306A (en) * 2022-03-29 2022-06-28 中国矿业大学 Context aggregation image rain removing method based on edge information guidance
CN114972105A (en) * 2022-06-10 2022-08-30 江苏海洋大学 Single image rain removing method based on multi-scale fusion residual error network
CN115937049A (en) * 2023-02-23 2023-04-07 华中科技大学 Rain removal model lightweight method, system, device and medium
WO2024040973A1 (en) * 2022-08-22 2024-02-29 南京邮电大学 Multi-scale fused dehazing method based on stacked hourglass network
CN114066959B (en) * 2021-11-25 2024-05-10 天津工业大学 Single fringe image depth estimation method based on transducer

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108710830A (en) * 2018-04-20 2018-10-26 浙江工商大学 A kind of intensive human body 3D posture estimation methods for connecting attention pyramid residual error network and equidistantly limiting of combination
CN111340738A (en) * 2020-03-24 2020-06-26 武汉大学 Image rain removing method based on multi-scale progressive fusion
CN111652812A (en) * 2020-04-30 2020-09-11 南京理工大学 Image defogging and rain removing algorithm based on selective attention mechanism
CN112907479A (en) * 2021-03-05 2021-06-04 西安电子科技大学 Residual single image rain removing method based on attention mechanism

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108710830A (en) * 2018-04-20 2018-10-26 浙江工商大学 A kind of intensive human body 3D posture estimation methods for connecting attention pyramid residual error network and equidistantly limiting of combination
CN111340738A (en) * 2020-03-24 2020-06-26 武汉大学 Image rain removing method based on multi-scale progressive fusion
CN111652812A (en) * 2020-04-30 2020-09-11 南京理工大学 Image defogging and rain removing algorithm based on selective attention mechanism
CN112907479A (en) * 2021-03-05 2021-06-04 西安电子科技大学 Residual single image rain removing method based on attention mechanism

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
徐爱生 等: "注意力残差网络的单图像去雨方法研究", 《小型微型计算机系统》 *
李金晶: "基于卷积神经网络的单幅图像去雨研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》 *
马婧婧 等: "多尺度沙漏结构的单幅图像去雨算法研究", 《小型微型计算机系统》 *

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114066959A (en) * 2021-11-25 2022-02-18 天津工业大学 Single-stripe image depth estimation method based on Transformer
CN114066959B (en) * 2021-11-25 2024-05-10 天津工业大学 Single fringe image depth estimation method based on transducer
CN114492522A (en) * 2022-01-24 2022-05-13 四川大学 Automatic modulation classification method based on improved stacked hourglass neural network
CN114677306A (en) * 2022-03-29 2022-06-28 中国矿业大学 Context aggregation image rain removing method based on edge information guidance
CN114638768A (en) * 2022-05-19 2022-06-17 武汉大学 Image rain removing method, system and equipment based on dynamic association learning network
CN114638768B (en) * 2022-05-19 2022-08-09 武汉大学 Image rain removing method, system and equipment based on dynamic association learning network
CN114972105A (en) * 2022-06-10 2022-08-30 江苏海洋大学 Single image rain removing method based on multi-scale fusion residual error network
CN114972105B (en) * 2022-06-10 2024-03-29 江苏海洋大学 Single image rain removing method based on multi-scale fusion residual error network
WO2024040973A1 (en) * 2022-08-22 2024-02-29 南京邮电大学 Multi-scale fused dehazing method based on stacked hourglass network
CN115937049A (en) * 2023-02-23 2023-04-07 华中科技大学 Rain removal model lightweight method, system, device and medium

Also Published As

Publication number Publication date
CN113673590B (en) 2022-12-23

Similar Documents

Publication Publication Date Title
CN113673590B (en) Rain removing method, system and medium based on multi-scale hourglass dense connection network
CN110111366B (en) End-to-end optical flow estimation method based on multistage loss
Engin et al. Cycle-dehaze: Enhanced cyclegan for single image dehazing
CN106952228B (en) Super-resolution reconstruction method of single image based on image non-local self-similarity
CN112507997B (en) Face super-resolution system based on multi-scale convolution and receptive field feature fusion
CN111915530B (en) End-to-end-based haze concentration self-adaptive neural network image defogging method
CN111127374B (en) Pan-sharing method based on multi-scale dense network
CN112184577B (en) Single image defogging method based on multiscale self-attention generation countermeasure network
CN111709895A (en) Image blind deblurring method and system based on attention mechanism
Zhang et al. LR-Net: Low-rank spatial-spectral network for hyperspectral image denoising
CN109872305B (en) No-reference stereo image quality evaluation method based on quality map generation network
CN112634159B (en) Hyperspectral image denoising method based on blind noise estimation
CN114782298B (en) Infrared and visible light image fusion method with regional attention
CN112070688A (en) Single image defogging method for generating countermeasure network based on context guidance
CN114266957A (en) Hyperspectral image super-resolution restoration method based on multi-degradation mode data augmentation
Singh et al. A review of image fusion: Methods, applications and performance metrics
CN115578262A (en) Polarization image super-resolution reconstruction method based on AFAN model
CN115861076A (en) Unsupervised hyperspectral image super-resolution method based on matrix decomposition network
Babu et al. An efficient image dahazing using Googlenet based convolution neural networks
Deng et al. Multiple frame splicing and degradation learning for hyperspectral imagery super-resolution
CN113628143A (en) Weighted fusion image defogging method and device based on multi-scale convolution
CN113781375B (en) Vehicle-mounted vision enhancement method based on multi-exposure fusion
Liu et al. Dual UNet low-light image enhancement network based on attention mechanism
CN114118199A (en) Image classification method and system for fault diagnosis of intelligent pump cavity endoscope
Li et al. ConvFormerSR: Fusing Transformers and Convolutional Neural Networks for Cross-sensor Remote Sensing Imagery Super-resolution

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant