CN112884073B - Image rain removing method, system, terminal and storage medium - Google Patents

Image rain removing method, system, terminal and storage medium Download PDF

Info

Publication number
CN112884073B
CN112884073B CN202110302518.0A CN202110302518A CN112884073B CN 112884073 B CN112884073 B CN 112884073B CN 202110302518 A CN202110302518 A CN 202110302518A CN 112884073 B CN112884073 B CN 112884073B
Authority
CN
China
Prior art keywords
image
rain
layer
model
network
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110302518.0A
Other languages
Chinese (zh)
Other versions
CN112884073A (en
Inventor
王永芳
黎梦瑶
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
University of Shanghai for Science and Technology
Original Assignee
University of Shanghai for Science and Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by University of Shanghai for Science and Technology filed Critical University of Shanghai for Science and Technology
Priority to CN202110302518.0A priority Critical patent/CN112884073B/en
Publication of CN112884073A publication Critical patent/CN112884073A/en
Application granted granted Critical
Publication of CN112884073B publication Critical patent/CN112884073B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computational Linguistics (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)

Abstract

The invention provides an image rain removing method and system, wherein an image rain removing model based on a recursive residual cavity space pyramid pool network is constructed; constructing a training sample set; training the image rain removal model by adopting the training sample set to obtain a rain-free image generation model; and inputting the rain pictures with different sizes into the rain-free image generation model to obtain a corresponding rain-removing image. A corresponding terminal and storage medium are also provided. The method is based on the recursive residual error cavity space pyramid pool network, so that the problem of image restoration in rainy days is effectively solved; by extracting and fusing multi-scale information in the rain map, the quality of the recovered rain-free image is higher, and a long-term and short-term memory network module is introduced to enhance the dependency between stages; a mixing loss function is introduced, so that the details of the recovered image are finer, and the edges are clearer; the structural similarity and the peak signal-to-noise ratio of the rain-removing reconstructed image can be effectively improved, and a better effect is achieved subjectively.

Description

Image rain removing method, system, terminal and storage medium
Technical Field
The invention relates to the technical field of image processing and reconstruction, in particular to an image rain removing method, system, terminal and storage medium based on a recursive residual empty space pyramid pool network.
Background
With the development of science and technology, the human society is entering the information-based society. Images have become an important source of information for human beings because of their large information content and high transmission speed. In recent years, computer vision systems are being widely used in various fields of society. In many algorithms of computer vision, such as image segmentation, target recognition, behavior detection, etc., it is necessary to utilize effective information in the image for implementation. However, under the condition of poor weather conditions, such as rain, the phenomena of contrast reduction, image blurring, color distortion and the like of images acquired by an outdoor imaging system can be caused, and the extraction of image features is directly influenced, so that the accuracy of computer vision algorithms such as target identification, behavior detection and the like is reduced. Image rain removal belongs to an image processing technology, namely, a rain-free image is recovered from a rain image. Image rain removal has important applications for the stable operation of outdoor vision systems.
The traditional image rain removing algorithm mainly depends on the statistical analysis of rain stripes and background scenes. These algorithms are optimized by constructing priors in the background and rain layers and then using a cost function. According to a priori extraction mode, the traditional image rain removing algorithm is mainly divided into a sparse coding algorithm and a Gaussian mixture model.
In recent years, Convolutional Neural Networks (CNNs) have made remarkable progress in the computer vision field such as object detection and image segmentation, and there are more and more people applying CNNs to the image rain-removing field. The deep learning based algorithm has surpassed the traditional method. These methods automatically extract hierarchical features using deep networks so that they can learn more complex mappings from rain maps to clean images. In 2017, Yang et al, Yang, Wenhan, Robby T.Tan, Jianshi Feng, Jianying Liu, Zongming Guo, and Shuicheng Yan. "Deep joint rain detection and removal from a single image," In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp.1357-1366.2017, applied CNN In the field of image rain removal, proposed a combined rain detection and removal network that detects the location of rain by predicting a binary rain mask and took a recursive framework to gradually remove rain streaks. In the same year, Fu et al In Fu, Xueyang, Jianbin Huang, Delu Zeng, Yue Huang, Xinghao Ding, and John Paisley, "Removing chain from single images via a depth detail network," In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp.3855-3863.2017, propose a depth detail network which, after decomposition, enters high frequency details of the image into the network and makes training of the network easier and more stable by predicting the residual between the rain map and the clean image. The advanced deep learning image rain removal algorithm solves the problems that the traditional image rain removal algorithm is complex in calculation and the like. However, these networks often fail to recover a good rain-free image when they process some real rain patterns that have never been seen in training.
Through search, the following results are found:
in the Chinese patent application CN110390654A, which is published as 2019, 10 and 29, a post-processing method for multi-stage iterative collaborative representation of a rain-removing image, part of details lost in the rain removing process can be recovered by learning the regression relationship of a training set formed by training image pairs based on the training image pairs, namely the rain-removing image and a corresponding rain-free image. The invention provides an iterative collaborative representation method, namely a multilevel mapping model between a rain removing image and a corresponding rain-free image characteristic pair is learned in a training phase, and then the rain removing image is continuously optimized in corresponding iteration by utilizing the multilevel mapping model, so that the visual quality of the rain removing image is effectively improved. The method needs to extract features through a plurality of preprocessing processes, and trains a model on the extracted features, so that the calculated amount is large, and the time for processing one picture is possibly long.
Chinese patent application publication No. CN110717527A, published as 2020, 1, 21, the method for determining a target detection model in combination with a void space pyramid structure, includes inputting an input feature map into k void convolution layer branches, performing a void convolution operation on the input feature map at each void convolution layer branch to obtain each output feature map, fusing the input feature map and k output feature maps to construct a void space pyramid structure, and fusing the void space pyramid structure into a target detection model based on a convolution network, wherein the void space pyramid structure resamples the input feature map by using void convolutions with different specific expansion rates to obtain output feature maps with different receptive fields, and then performs feature fusion of input and output to obtain multi-scale information, and fuses the void space pyramid structure into a target detection model based on deep learning, the detection capability of the target detection model based on deep learning on the multi-scale target can be improved. If the method is applied to an image rain removal technology, a target detection model of the method focuses more on high-level semantic features and position information of an image, and the image rain removal focuses more on low-level features at a pixel level, so that the loss of detailed information of a reconstruction result can be caused by directly applying the method to the image rain removal.
At present, no explanation or report of the similar technology of the invention is found, and similar data at home and abroad are not collected.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides an image rain removing method, system, terminal and storage medium based on a recursive residual error hollow space pyramid pool network.
According to an aspect of the present invention, there is provided an image rain removing method, including:
constructing an image rain removal model based on a recursive residual error hollow space pyramid pool network;
constructing a training sample set;
training the image rain removal model by adopting the training sample set, and constraining a reconstruction result by a mixed loss function by utilizing a self-adaptive moment estimation optimization method to obtain a rain-free image generation model;
and inputting the rain pictures with different sizes into the rain-free image generation model to obtain a corresponding rain-removing image.
Preferably, the constructing of the image rain removing model based on the recursive residual empty space pyramid pool network includes:
adopting a rolling block to construct a preliminary feature extraction layer for extracting the features of the input image;
constructing a correlation connection layer by adopting a long and short term memory volume block;
adopting a residual block to construct a preliminary feature fusion layer for fusing the features of the input image;
constructing a multi-scale feature extraction layer by adopting a residual empty space pyramid pool network, and performing feature extraction, fusion and cascade on the fused input image features through parallel expansion convolution with different expansion rates to obtain multi-scale features;
constructing a multi-scale feature fusion layer by adopting a residual block, wherein the multi-scale feature fusion layer is used for fusing the multi-scale features;
adopting a rolling block to construct an output layer for outputting a reconstructed rain-free image;
sequentially connecting the preliminary feature extraction layer, the correlation connection layer, the preliminary feature fusion layer, the multi-scale feature extraction layer, the multi-scale feature fusion layer and the output layer to form an image rain removal reconstruction module of one stage;
the image rain removing reconstruction modules in multiple stages are sequentially connected to form a recursive network structure, the correlation among the stages is increased through the correlation connecting layer, network parameters are shared among the multiple stages, and finally the image rain removing model with the multi-stage structure and based on the recursive residual error hollow space pyramid pool network is formed.
Preferably, the constructing a training sample set includes:
transforming the training pictures by adopting an image augmentation method to form a training sample pair;
and cutting the training sample pair to construct and form a training sample set.
Preferably, the image augmentation method adopts a flipping method to flip and transform the training picture.
Preferably, the clipping the training sample pair includes:
and cutting the images in the training sample pair into image blocks with the resolution of 100x 100.
Preferably, the constraining the reconstruction result by the mixture loss function using the adaptive moment estimation optimization method includes:
and constraining the reconstructed rain-free image from the structural and semantic angles through a mixed loss function, wherein the negative structural similarity loss is constrained from the structural angle, and the perceptual loss is constrained from the semantic angle, and is represented as follows:
Figure GDA0003583944070000041
wherein, the lambda is a coefficient of the term,
Figure GDA0003583944070000042
the negative structural similarity loss is used for restraining the structural similarity of the reconstructed rain-free image and the standard rain-free image and helping the image to reconstruct structural information;
Figure GDA0003583944070000043
the model recovers a better visual result by utilizing the advanced characteristics of the loss network trained in advance;
loss of negative structural similarity
Figure GDA0003583944070000044
The definition is as follows:
Figure GDA0003583944070000045
wherein SSIM (. cndot.) is structural similarity, XoutputIs the final reconstructed rain-free image, XgtIs a corresponding standard rain-free image.
The loss of perception
Figure GDA0003583944070000046
The definition is as follows:
Figure GDA0003583944070000047
wherein phi isj(X) is a feature map of layer j of the VGG-16 network pre-trained on ImageNet; cj，Hj，WjAre respectively phij(X) channel, height and width.
Preferably, λ is 0.04.
According to another aspect of the present invention, there is provided an image rain removing system comprising:
the image rain removing model building module is used for building an image rain removing model based on the recursive residual empty space pyramid pool network;
a training sample set constructing module, which is used for constructing a training sample set;
the rain-free image generation module is used for training the image rain removal model by adopting the training sample set, and constraining a reconstruction result by a mixed loss function by using a self-adaptive moment estimation optimization method to obtain a rain-free image generation model; and inputting the rain pictures with different sizes into the rain-free image generation model to obtain a corresponding rain-removing image.
Preferably, the image rain removing model adopts a multi-stage image rain removing reconstruction module structure, wherein the image rain removing reconstruction module of each stage comprises:
a preliminary feature extraction layer, which is constructed by a convolution block and is used for extracting the features of the input image;
a correlation connection layer, which is constructed by adopting a long-short term memory convolution block;
a preliminary feature fusion layer, constructed using a residual block, for fusing the input image features;
a multi-scale feature extraction layer which is constructed by adopting a residual empty space pyramid pool network and performs feature extraction, fusion and cascade connection on the fused input image features through parallel expansion convolution with different expansion rates to obtain multi-scale features;
a multi-scale feature fusion layer, which is constructed by adopting a residual block and is used for fusing the multi-scale features;
an output layer, constructed using a convolution block, for outputting the reconstructed rain-free image;
the image rain removing reconstruction modules in the multiple stages are sequentially connected to form a recursive network structure, the correlation among the stages is increased through the correlation connecting layer, network parameters are shared among the multiple stages, and finally the image rain removing model with the multi-stage structure and based on the recursive residual error hollow space pyramid pool network is formed.
According to a third aspect of the present invention, there is provided a terminal comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor being operable to perform any of the methods described above when executing the program.
According to a fourth aspect of the invention, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, is operable to perform the method of any of the above.
Due to the adoption of the technical scheme, compared with the prior art, the invention has the following beneficial effects:
the image rain removing method, the system, the terminal and the storage medium provided by the invention effectively solve the problem of image recovery in rainy days on the basis of the recursive residual error hollow space pyramid pool network.
The image rain removing method, the system, the terminal and the storage medium provided by the invention have the advantages that the quality of the recovered rain-free image is higher through the extraction and fusion of multi-scale information in the rain map, and the long-term and short-term memory network module is introduced to enhance the dependency between stages.
The image rain removing method, the system, the terminal and the storage medium provided by the invention introduce the mixing loss function, so that the details of the recovered image are finer and the edges are clearer.
The image rain removing method, the system, the terminal and the storage medium provided by the invention can effectively improve the structural similarity and the peak signal-to-noise ratio of the rain removing reconstruction image and also obtain better effect subjectively.
The image rain removing method, the system, the terminal and the storage medium provided by the invention fully consider the multi-scale characteristic of the image, and provide a single image rain removing technology based on the recursive residual empty space pyramid pool network based on the characteristic.
The image rain removing method, the system, the terminal and the storage medium provided by the invention adopt the mixed loss function to constrain the reconstructed rain-free image from the two angles of the structural similarity and the content.
The image rain removing method, the system, the terminal and the storage medium provided by the invention adopt a multi-stage recursive network structure, and parameters are shared among different stages, so that the training difficulty and the parameters of the model are effectively reduced.
The image rain removing method, the system, the terminal and the storage medium directly use the original image to train the network model, provide a simple end-to-end rain removing model and have high image processing speed.
According to the image rain removing method, the system, the terminal and the storage medium, a simple convolutional neural network is provided according to tasks, and a rain-free image is effectively reconstructed by combining multi-scale information according to the characteristics of rain.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:
fig. 1 is a flowchart of an image rain removing method according to an embodiment of the present invention.
FIG. 2 is a schematic diagram of an image rain removal model based on a recursive residual hollow space pyramid pool network according to a preferred embodiment of the present invention.
FIG. 3 is a comparison graph of the results of whether perceptual loss is used in accordance with a preferred embodiment of the present invention.
FIG. 4 is a comparison of the reconstruction of the 27 th image in Rain200H of a preferred embodiment of the present invention and other methods.
Fig. 5 is a comparison graph of the reconstruction results of a real rain map according to a preferred embodiment of the present invention and other methods.
FIG. 6 is a graph comparing the output results of stages 1 to 6 in a preferred embodiment of the present invention.
Fig. 7 is a schematic structural diagram of an image rain removing system according to an embodiment of the present invention.
Detailed Description
The following examples illustrate the invention in detail: the embodiment is implemented on the premise of the technical scheme of the invention, and a detailed implementation mode and a specific operation process are given. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention.
Fig. 1 is a flowchart of an image rain removing method according to an embodiment of the present invention.
As shown in fig. 1, the image rain removing method provided by this embodiment may include the following steps:
s100, constructing an image rain removal model based on a recursive residual error hollow space pyramid pool network;
s200, constructing a training sample set;
s300, training an image rain removal model by adopting the training sample set, and constraining a reconstruction result by a mixed loss function by using a self-adaptive moment estimation optimization method to obtain a rain-free image generation model;
s400, inputting the rain pictures with different sizes into the rain-free image generation model to obtain corresponding rain-removed images.
In S100 of this embodiment, constructing an image rain removing model based on the recursive residual hollow space pyramid pool network may include the following steps:
s101, constructing a preliminary feature extraction layer by adopting a rolling block, wherein the preliminary feature extraction layer is used for extracting features of an input image;
s102, constructing a correlation connection layer by adopting a long-term and short-term memory volume block;
s103, constructing a preliminary feature fusion layer by adopting a residual block, wherein the preliminary feature fusion layer is used for fusing the features of the input image;
s104, constructing a multi-scale feature extraction layer by adopting a residual empty space pyramid pool network, and performing feature extraction, fusion and cascade connection on the fused input image features through parallel expansion convolution with different expansion rates to obtain multi-scale features;
s105, constructing a multi-scale feature fusion layer by adopting a residual block, wherein the multi-scale feature fusion layer is used for fusing multi-scale features;
s106, adopting a rolling block to construct an output layer for outputting the reconstructed rain-free image;
s107, sequentially connecting the primary feature extraction layer, the correlation connection layer, the primary feature fusion layer, the multi-scale feature extraction layer, the multi-scale feature fusion layer and the output layer to form a one-stage image rain removal reconstruction module;
and S108, sequentially connecting the image rain removal reconstruction modules in multiple stages to form a recursive network structure, increasing the correlation among the stages through a correlation connecting layer, sharing network parameters among the multiple stages, and finally forming an image rain removal model based on the recursive residual error empty space pyramid pool network and having a multi-stage structure.
In S200 of this embodiment, constructing the training sample set may include the following steps:
s201, converting a training picture by adopting an image augmentation method to form a training sample pair;
and S202, cutting the training sample pair to construct a training sample set.
In a specific application example of the embodiment, the image augmentation method may adopt a flipping method to flip the training picture.
In a specific application example of this embodiment, the clipping the training sample pair may include:
the images in the training sample pair are cropped into image blocks with a resolution of 100x 100.
In S300 of this embodiment, constraining the reconstruction result by the mixture loss function using the adaptive moment estimation optimization method includes:
and (3) constraining the reconstructed rain-free image from the structural and semantic angles through a mixed loss function, wherein the negative structural similarity loss is constrained from the structural angle, and the perceptual loss is constrained from the semantic angle and is represented as follows:
Figure GDA0003583944070000071
wherein λ is a coefficient, and in a specific application example, λ may be 0.04,
Figure GDA0003583944070000072
the negative structural similarity loss is used for restraining the structural similarity of the reconstructed rain-free image and the standard rain-free image and helping the image to reconstruct structural information;
Figure GDA0003583944070000073
the model recovers a better visual result by utilizing the advanced characteristics of the loss network trained in advance;
loss of negative structural similarity
Figure GDA0003583944070000074
The definition is as follows:
Figure GDA0003583944070000081
wherein SSIM (. cndot.) is structural similarity, XoutputIs the final reconstructed rain-free image, XgtIs a corresponding standard rain-free image.
Loss of perception
Figure GDA0003583944070000082
The definition is as follows:
Figure GDA0003583944070000083
wherein phij(X) is a feature map of layer j of the VGG-16 network pre-trained on ImageNet; cj，Hj，WjAre respectively phij(X) channel, height and width.
The pre-trained VGG-16 network level j profile on ImageNet is set to compute perceptual loss. ImageNet is a large data set, VGG-16 is a classical network structure, and the perceptual function is calculated by pre-training the parameters of VGG-16 on ImageNet.
According to the image rain removing method provided by the embodiment of the invention, the important function of the multi-scale features on the rain removal of the single image is fully considered, the multi-scale features in the image are effectively extracted through the residual empty space pyramid pool network, and the extracted multi-scale features are fused by adopting the residual block. In addition, in order to reduce the parameter quantity of the network, a recursive structure is adopted, network parameters are shared among multiple stages, and Long Short-Term Memory volume (LSTM) blocks are introduced to increase the correlation among the stages. In order to enable the reconstructed image detail texture to be finer, a mixed loss function is adopted, and the reconstructed rain-free image is constrained from two angles of structural similarity and content.
A preferred embodiment of the present invention provides an image rain removing method, and the following describes the technical solution provided by the preferred embodiment in further detail with reference to the accompanying drawings.
The image rain removing method provided by the preferred embodiment is to design an image rain removing model according to a convolutional neural network structure, and comprises a feature extraction part, a fusion part and a reconstruction part. Then, the original rain map and the pair of rain-free images are cut into small pieces with the resolution of 100x100 as a training set. Finally, the image pair is sent into a recursive convolutional neural network for gradual reconstruction, the rain-free image recovered through the constraint of a mixing loss function is trained by using an adaptive moment estimation optimizer (refer to Kingma, Diederik P., and Jimmy Ba. 'Adam: A method for stochastic optimization.' arXiv preprintiv: 1412.6980 (2014)), and a model for reconstructing the rain image into the rain-free image, namely an image rain removing model based on the recursive residual hollow space pyramid network is obtained.
According to the concept, the preferred embodiment adopts the following technical scheme:
an image rain removing method comprises the following steps:
step 1, establishing an image rain removal model based on a recursive residual error hollow space pyramid pool network: designing an image rain removal model according to the convolutional neural network structure, including feature extraction, fusion and reconstruction;
step 2, image augmentation: the training of the deep learning model depends heavily on a large-scale data set, and the image augmentation technology generates similar training samples by performing a series of simple transformations on training pictures, so that the scale of the training samples is improved. The large-scale data set can effectively improve the robustness and generalization capability of the model, and the scale of the image sample is enlarged by using a turning method;
step 3, training set establishment: cutting the large-scale image sample pairs obtained in the step 2, cutting the image into image blocks with the resolution of 100x100, and constructing a training set by using the image block pairs;
step 4, training a recurrent convolution neural network model: training a single image rain removal model on the training set obtained in the step 3, using a self-adaptive moment estimation optimization method, constraining a reconstruction result through a mixing loss function, and obtaining a model for recovering a rain image into a rain-free image after training is finished;
step 5, rain-free image reconstruction: and (4) inputting rain pictures with different sizes in the model obtained by training in the step 4 to obtain a corresponding rain-free image.
In the preferred embodiment, a schematic structural diagram of an image rain removal model based on a recursive residual void space pyramid pool network is shown in fig. 2. The method can be realized in the program simulation of Ubuntu 16.04 and PyTorch environment. Firstly, a single image rain removal model is designed according to a convolutional neural network structure, and comprises a characteristic extraction part, a fusion part and a reconstruction part. Then, the original image pair is subjected to data amplification, and a training set required by the model is formed by slicing the amplified image pair. And finally, training the network model by using an adaptive moment optimizer, and constraining the reconstructed rain-free image from two angles of structural similarity and content through a mixed loss function to obtain a model for reconstructing a rain map into a rain-free image, namely an image rain removal model based on a recursive residual empty space pyramid pool network.
As a preferred embodiment, in step 1, a recursive residual hollow space pyramid pool single image rain removal model is proposed, which includes feature extraction, fusion, and reconstruction, and a network structure is shown in fig. 2. The model adopts a multi-stage structure, and a rain-free image is reconstructed step by step from coarse to fine. In order to avoid the increase of the number of network parameters caused by multiple stages, a recursive structure is adopted, and parameters are shared among multiple stages. For each phase, the original rain map and the output of the previous phase are combined as input, wherein for the first phase, the original rain map and the original rain map are combined as input.
Regarding the single stage in the model, firstly, a 3 × 3 convolution block is used to extract the features of the input image, and in order to increase the correlation between stages, a long and short term memory convolution block (refer to Shi, Xingjian, Zhoouring Chen, Hao Wang, Dit-Yan Yueung, Wai-Kin Wong, and Wang-chun Wo. "connected LSTM network: A machine learning approach for prediction and non-prediction." Advances in the neural information processing systems 28(2015):802-tOutput gate otForgetting door ftAnd state ctThe structure of the utility model is shown as follows,
it＝σ(Wyi*yt+Whi*ht-1+bi) (1)
ft＝σ(Wyf*yt+Whf*ht-1+bf) (2)
ot＝σ(Wyo*yt+Who*ht-1+bo) (3)
ct＝ft⊙ct-1+it⊙tanh(Wyc*yt+Whc*ht-1+bc) (4)
ht＝ot⊙tanh(ct) (5)
wherein, is a convolution operation, the | _ is a matrix multiplication, σ is a sigmoid function, ytIs a feature of input layer generation, htIs the output characteristic of the long and short term memory convolutional layer, W and b are the corresponding convolutional matrix and offset, respectively, and all convolutional layers are 3 × 3 in size. Next, a Residual block (ResBlock) containing two 3 × 3 convolutional layers and a ReLU active layer fuses the features obtained at the previous stage. To enlarge the receptive field, Residual hole space pyramid (Residual error) modulespatial pyramid discharging block, ResASPP, is introduced with reference to Wang, Longguang, Yingqian Wang, Zhengfa Liang, Zaiping Lin, Junging Yang, Wei An, and Yuan Guo e. "Learning parallel orientation for stereo image super-resolution" In Proceedings of the IEEE Conference on Computer Vision and Pattern registration, pp.12250-12259.2019 "). The module extracts features through three parallel expansion convolutions with expansion rates of 1, 4 and 8, uses features obtained by fusion of 1 × 1 convolution blocks, and concatenates the parallel expansion convolutions and the 1 × 1 convolution blocks three times in a residual error manner to obtain final output, wherein the specific structure is shown in fig. 2. And then the extracted multi-scale features are fused through the residual blocks, the operation is alternated for several times, and finally, a reconstructed rain-free image is obtained through an output layer consisting of 3 multiplied by 3 convolution. Since the number of network stages in the present invention is 6, the finally obtained image without rain is the image generated in stage 6.
As a preferred embodiment, In step 2, the training set images used are 1800 images In Rain200H (referred to by Yang, Wenhan, ribbon t. tan, Jiashi Feng, jiang Liu, zong Guo, and shuiching yan, "Deep joint image detection and removal from a single image," In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.1357-1366.2017 "), and the training set images used by Rain200L (referred to by Yang, Wenhan, row t. tan, Jiashi feed, jiang Liu, zong Guo, and shuiching yan Yang," Deep joint image detection and removal feed "registration, pp.1357-1366.2017), respectively, are obtained by inverting the image data of the image set, In a number greater than the original image data, map 1366.2017.
In step 3, the augmented image pair obtained in step 2 is subjected to a block segmentation process, and the original image is segmented into 100 × 100 image blocks, wherein a plurality of 100 × 100 image block pairs constitute a training set used by the training model.
As a preferred embodiment, in step 4, a single-image rain removing model is trained on the training set obtained in step 3, and an Adaptive moment estimation algorithm (Adaptive Mom) is used as an optimization algorithment Estimation), the batch size is set to 6, the initial learning rate is 1 × 10-3Every 30 epochs, the learning rate is decreased by multiplying by 0.2. The reconstructed rain-free image is constrained from two aspects of pixel value and content by a mixing loss function, and is represented as follows:
Figure GDA0003583944070000111
wherein lambda is set to 0.04,
Figure GDA0003583944070000112
is a negative loss of structural similarity and,
Figure GDA0003583944070000113
is the loss of perception. The negative structural similarity loss is used to constrain the structural similarity of the reconstructed rain-free image and the standard rain-free image, and helps the image to reconstruct structural information. The loss is defined as follows:
Figure GDA0003583944070000114
wherein XoutputIs the final reconstructed rain-free image, XgtIs a corresponding standard rain-free image.
Perceptual loss utilizes advanced features of a pre-trained loss network to enable the model to recover visually better results. The loss is defined as follows:
Figure GDA0003583944070000115
wherein phij(X) is a feature map of layer j of the VGG-16 network pre-trained on ImageNet. Cj，Hj，WjAre respectively phij(X) channel, height and width.
After training is completed, a model for restoring the rain image to the rain-free image can be obtained.
In step 5, a rain-free image which can be recovered can be obtained by inputting a rain map of an arbitrary size into the model trained in step 4.
The extensive capability and advancement of the image Rain removal method based on the recursive residual void space pyramid network proposed In the present invention were evaluated using the test set of the public data set Rain200H, Rain200L, and data set Rain12 (see Li, Yu, Robby t.tan, Xiaojie Guo, Jiangbo Lu, and Michael s.brown. "Rain stream removal using layer printers." In Proceedings of the IEEE con on computer vision and pattern recognition, pp.2736-2744.2016), and the real Rain map. The number of images in the test set of the data sets Rain200H and Rain200L is 200, and the number of images in the data set Rain12 is 12. The real rain picture is an image on the network, and only a subjective result is shown because no corresponding original picture exists. The environment of the experiment is an Ubuntu 16.04 operating system, the memory is 16GB, the GPU is GeForce 1080, and the deep learning framework is a PyTorch platform. Peak Signal to Noise Ratio (PSNR) and Structural Similarity coefficient (SSIM) were used as evaluation indexes. The larger the PSNR value is, the closer the SSIM is to 1 indicates that the rainless image restored by the model is closer to the original image, and the higher the restoration quality is. Fig. 3 compares the comparison of results on test pictures with the use of perceptual loss, fig. 4-5 compare the comparison of recovery results on test pictures with different algorithms, and fig. 6 shows the output results of different models from 1 to 6 in the number of stages.
TABLE 1
Figure GDA0003583944070000116
Figure GDA0003583944070000121
TABLE 2
Method PSNR SSIM
Negative structural similarity function 30.32 0.9151
Negative structural similarity function + perceptual function 30.40 0.9158
Table 1 shows the PSNR and SSIM values of the present invention compared to other advanced methods, where the highest experimental index is represented by bold red font and the second experimental index is represented by bold blue font. From table 1, it can be seen that the method of the present invention has better robustness and accuracy on three databases, and thus, the method of the present invention has better robustness and accuracy in the aspect of image rain removal. Table 2 shows the influence of the mixing loss function provided by the present invention on image rain removal, and it can be seen that the mixing loss function provided by the present invention can improve the quality of the reconstructed image. The experiment shows that the recursive residual error void space pyramid pool image rain removing algorithm provided by the invention has the advantages of advancement in objective indexes and visual effects, low calculation complexity and better real-time property.
Another embodiment of the present invention provides an image rain removing system, as shown in fig. 7, which may include: the system comprises an image rain removing model building module, a training sample set building module and a rain-free image generating module; wherein:
the image rain removing model building module is used for building an image rain removing model based on the recursive residual empty space pyramid pool network;
a training sample set constructing module, which is used for constructing a training sample set;
the rain-free image generation module is used for training the image rain removal model by adopting a training sample set, and restraining a reconstruction result by using a self-adaptive moment estimation optimization method through a mixed loss function to obtain a rain-free image generation model; and inputting the rain pictures with different sizes into the rain-free image generation model to obtain a corresponding rain-removing image.
As a preferred embodiment, the image rain removing model adopts a multi-stage image rain removing reconstruction module structure, wherein the image rain removing reconstruction module of each stage comprises:
a preliminary feature extraction layer, which is constructed by a convolution block and is used for extracting the features of the input image;
a correlation connection layer, which is constructed by adopting a long-short term memory convolution block;
a preliminary feature fusion layer, which is constructed by using a residual block and is used for fusing the features of the input image;
a multi-scale feature extraction layer which is constructed by adopting a residual empty space pyramid pool network and performs feature extraction, fusion and cascade connection on the fused input image features through parallel expansion convolution with different expansion rates to obtain multi-scale features;
the multi-scale feature fusion layer is constructed by adopting a residual block and is used for fusing multi-scale features;
an output layer, constructed using a convolution block, for outputting the reconstructed rain-free image;
the image rain removing reconstruction modules in the multiple stages are sequentially connected to form a recursive network structure, the correlation among the stages is increased through a correlation connecting layer, network parameters are shared among the multiple stages, and finally the image rain removing model with the multi-stage structure and based on the recursive residual error hollow space pyramid pool network is formed.
A third embodiment of the present invention provides a terminal comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor is operable to execute the method of any one of the above embodiments of the present invention when executing the program.
A fourth embodiment of the invention provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, is operable to perform the method of any one of the above-described embodiments of the invention.
The image rain removing method, system, terminal and storage medium provided by the above embodiments of the present invention are based on a recursive residual empty space pyramid pool network, and first, multi-scale rain information is utilized by alternately cascading residual empty space pyramid pool blocks and residual blocks. In addition, a long and short term memory volume block is introduced into the rain removing network in consideration of the cross-stage dependency relationship of the depth feature. For each stage in the recursive network, the output of the previous stage is concatenated with the original rain image as input. Furthermore, the proposed network is trained with a mixed penalty consisting of a negative structural similarity penalty and a perceptual penalty. Experiments are carried out on a synthetic rain data set and a real rain data set, and the experimental results show that the method has higher robustness and accuracy.
It should be noted that, the steps in the method provided by the present invention may be implemented by using corresponding modules, devices, units, and the like in the system, and those skilled in the art may implement the composition of the system by referring to the technical solution of the method, that is, the embodiment in the method may be understood as a preferred example for constructing the system, and will not be described herein again.
Those skilled in the art will appreciate that, in addition to implementing the system and its various devices provided by the present invention in purely computer readable program code means, the method steps can be fully programmed to implement the same functions by implementing the system and its various devices in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Therefore, the system and various devices thereof provided by the present invention can be regarded as a hardware component, and the devices included in the system and various devices thereof for realizing various functions can also be regarded as structures in the hardware component; means for performing the functions may also be regarded as structures within both software modules and hardware components for performing the methods.
The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes and modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention.

Claims (8)

1. An image rain removing method, comprising:
constructing an image rain removal model based on a recursive residual error hollow space pyramid pool network;
constructing a training sample set;
training the image rain removal model by adopting the training sample set, and constraining a reconstruction result by a mixed loss function by utilizing a self-adaptive moment estimation optimization method to obtain a rain-free image generation model;
inputting the rain pictures with different sizes into the rain-free image generation model to obtain corresponding rain-removing images;
the method for constructing the image rain removal model based on the recursive residual error hollow space pyramid pool network comprises the following steps:
adopting a rolling block to construct a preliminary feature extraction layer for extracting the features of the input image;
constructing a correlation connection layer by adopting a long and short term memory volume block;
adopting a residual block to construct a preliminary feature fusion layer for fusing the features of the input image;
constructing a multi-scale feature extraction layer by adopting a residual empty space pyramid pool network, and performing feature extraction, fusion and cascade on the fused input image features through parallel expansion convolution with different expansion rates to obtain multi-scale features;
constructing a multi-scale feature fusion layer by adopting a residual block, wherein the multi-scale feature fusion layer is used for fusing the multi-scale features;
adopting a rolling block to construct an output layer for outputting a reconstructed rain-free image;
sequentially connecting the preliminary feature extraction layer, the correlation connection layer, the preliminary feature fusion layer, the multi-scale feature extraction layer, the multi-scale feature fusion layer and the output layer to form an image rain removal reconstruction module of one stage;
the image rain removing reconstruction modules in multiple stages are sequentially connected to form a recursive network structure, the correlation among the stages is increased through the correlation connecting layer, network parameters are shared among the multiple stages, and finally the image rain removing model with the multi-stage structure and based on the recursive residual error hollow space pyramid pool network is formed.
2. The image rain removal method according to claim 1, wherein the constructing a training sample set comprises:
transforming the training pictures by adopting an image augmentation method to form a training sample pair;
and cutting the training sample pair to construct and form a training sample set.
3. The image rain removing method according to claim 2, wherein the image augmenting method adopts a flipping method to flip the training picture.
4. The image rain removal method of claim 2, wherein the cropping the training sample pair comprises:
and cutting the images in the training sample pair into image blocks with the resolution of 100x 100.
5. The image rain removing method according to claim 1, wherein the constraining the reconstruction result by the mixture loss function using the adaptive moment estimation optimization method comprises:
and constraining the reconstructed rain-free image from the structural and semantic angles through a mixed loss function, wherein the negative structural similarity loss is constrained from the structural angle, and the perceptual loss is constrained from the semantic angle, and is represented as follows:
Figure FDA0003553577490000021
wherein, the lambda is a coefficient of the term,
Figure FDA0003553577490000022
the negative structural similarity loss is used for restraining the structural similarity of the reconstructed rain-free image and the standard rain-free image and helping the image to reconstruct structural information;
Figure FDA0003553577490000023
the model recovers a better visual result by utilizing the advanced characteristics of the loss network trained in advance;
loss of negative structural similarity
Figure FDA0003553577490000024
The definition is as follows:
Figure FDA0003553577490000025
wherein SSIM (. cndot.) is structural similarity, XoutputIs the final reconstructed rain-free image, XgtIs a corresponding standard no-rain image;
the loss of perception
Figure FDA0003553577490000026
The definition is as follows:
Figure FDA0003553577490000027
wherein phij(X) is a feature map of layer j of the VGG-16 network pre-trained on ImageNet; cj，Hj，WjAre respectively phij(X) channel, height and width.
6. An image rain removal system, comprising:
the image rain removing model building module is used for building an image rain removing model based on the recursive residual empty space pyramid pool network;
a training sample set constructing module, which is used for constructing a training sample set;
the rain-free image generation module is used for training the image rain removal model by adopting the training sample set, and constraining a reconstruction result by a mixed loss function by using a self-adaptive moment estimation optimization method to obtain a rain-free image generation model; inputting the rain pictures with different sizes into the rain-free image generation model to obtain corresponding rain-removing images;
the image rain removing model adopts a multi-stage image rain removing reconstruction module structure, wherein the image rain removing reconstruction module of each stage comprises the following components in sequence:
a preliminary feature extraction layer, which is constructed by a convolution block and is used for extracting the features of the input image;
a correlation connection layer, which is constructed by adopting a long-short term memory convolution block;
a preliminary feature fusion layer, constructed using a residual block, for fusing the input image features;
a multi-scale feature extraction layer which is constructed by adopting a residual empty space pyramid pool network and performs feature extraction, fusion and cascade connection on the fused input image features through parallel expansion convolution with different expansion rates to obtain multi-scale features;
a multi-scale feature fusion layer, which is constructed by adopting a residual block and is used for fusing the multi-scale features;
an output layer, constructed using a convolution block, for outputting the reconstructed rain-free image;
the image rain removing reconstruction modules in the multiple stages are sequentially connected to form a recursive network structure, the correlation among the stages is increased through the correlation connecting layer, network parameters are shared among the multiple stages, and finally the image rain removing model with the multi-stage structure and based on the recursive residual error hollow space pyramid pool network is formed.
7. A terminal comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor, when executing the program, is operative to perform the method of any of claims 1-5.
8. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, is adapted to carry out the method of any one of claims 1 to 5.
CN202110302518.0A 2021-03-22 2021-03-22 Image rain removing method, system, terminal and storage medium Active CN112884073B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110302518.0A CN112884073B (en) 2021-03-22 2021-03-22 Image rain removing method, system, terminal and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110302518.0A CN112884073B (en) 2021-03-22 2021-03-22 Image rain removing method, system, terminal and storage medium

Publications (2)

Publication Number Publication Date
CN112884073A CN112884073A (en) 2021-06-01
CN112884073B true CN112884073B (en) 2022-05-27

Family

ID=76041633

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110302518.0A Active CN112884073B (en) 2021-03-22 2021-03-22 Image rain removing method, system, terminal and storage medium

Country Status (1)

Country Link
CN (1) CN112884073B (en)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113344825B (en) * 2021-07-02 2022-04-26 南昌航空大学 Image rain removing method and system
CN113450288B (en) * 2021-08-04 2022-09-06 广东工业大学 Single image rain removing method and system based on deep convolutional neural network and storage medium
CN113807367B (en) * 2021-09-17 2023-06-16 平安科技（深圳）有限公司 Image feature extraction method, device, equipment and storage medium
CN114627005B (en) * 2022-02-16 2024-04-12 武汉大学 Rain density classification guided double-stage single image rain removing method
CN115546076B (en) * 2022-12-05 2023-04-07 耕宇牧星(北京)空间科技有限公司 Remote sensing image thin cloud removing method based on convolutional network

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110503613A (en) * 2019-08-13 2019-11-26 电子科技大学 Based on the empty convolutional neural networks of cascade towards removing rain based on single image method
CN112241939A (en) * 2020-10-15 2021-01-19 天津大学 Light-weight rain removing method based on multi-scale and non-local

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111462013B (en) * 2020-04-03 2022-03-22 西安交通大学 Single-image rain removing method based on structured residual learning
CN111639692B (en) * 2020-05-25 2022-07-22 南京邮电大学 Shadow detection method based on attention mechanism
CN112070687A (en) * 2020-08-20 2020-12-11 武汉大学 Image rain removing method and system based on team recursive feedback mechanism

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110503613A (en) * 2019-08-13 2019-11-26 电子科技大学 Based on the empty convolutional neural networks of cascade towards removing rain based on single image method
CN112241939A (en) * 2020-10-15 2021-01-19 天津大学 Light-weight rain removing method based on multi-scale and non-local

Also Published As

Publication number Publication date
CN112884073A (en) 2021-06-01

Similar Documents

Publication Publication Date Title
CN112884073B (en) Image rain removing method, system, terminal and storage medium
Van Den Oord et al. Pixel recurrent neural networks
CN111325165B (en) Urban remote sensing image scene classification method considering spatial relationship information
Guo et al. Mambair: A simple baseline for image restoration with state-space model
CN112541864A (en) Image restoration method based on multi-scale generation type confrontation network model
CN112241939B (en) Multi-scale and non-local-based light rain removal method
CN113298734B (en) Image restoration method and system based on mixed hole convolution
Ahn et al. EAGNet: Elementwise attentive gating network-based single image de-raining with rain simplification
CN116051396A (en) Image denoising method based on feature enhancement network and GRU network
Quan et al. Image desnowing via deep invertible separation
CN114331913A (en) Motion blurred image restoration method based on residual attention block
Li et al. Dual mixture model based cnn for image denoising
CN116030537B (en) Three-dimensional human body posture estimation method based on multi-branch attention-seeking convolution
CN111126166A (en) Remote sensing image road extraction method and system
WO2022252519A1 (en) Image processing method and apparatus, terminal, medium, and program
Jia et al. Single-image snow removal based on an attention mechanism and a generative adversarial network
CN114943655A (en) Image restoration system for generating confrontation network structure based on cyclic depth convolution
Shi et al. Progressive with purpose: Guiding progressive inpainting DNNs through context and structure
Hu et al. Enhancing Rainy Weather Driving: Deep Unfolding Network with PGD Algorithm for Single Image Deraining
Nie et al. Image restoration from patch-based compressed sensing measurement
Wu et al. RDS-denoiser: a detail-preserving convolutional neural network for image denoising
Latifi et al. Image denoising using convolutional neural network
Zhang et al. DAResNet Based on double-layer residual block for restoring industrial blurred images
CN115631115B (en) Dynamic image restoration method based on recursion transform
Qiu et al. Learning mean progressive scattering using binomial truncated loss for image dehazing

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant