CN116091350A - Image rain removing method and system based on multi-cascade progressive convolution structure - Google Patents
Image rain removing method and system based on multi-cascade progressive convolution structure Download PDFInfo
- Publication number
- CN116091350A CN116091350A CN202310053373.4A CN202310053373A CN116091350A CN 116091350 A CN116091350 A CN 116091350A CN 202310053373 A CN202310053373 A CN 202310053373A CN 116091350 A CN116091350 A CN 116091350A
- Authority
- CN
- China
- Prior art keywords
- cascade
- progressive
- convolution
- image
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000000750 progressive effect Effects 0.000 title claims abstract description 143
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000012549 training Methods 0.000 claims abstract description 44
- 125000004122 cyclic group Chemical group 0.000 claims abstract description 35
- 238000012545 processing Methods 0.000 claims abstract description 28
- 238000010276 construction Methods 0.000 claims description 20
- 238000000605 extraction Methods 0.000 claims description 18
- 238000012360 testing method Methods 0.000 claims description 16
- 238000004364 calculation method Methods 0.000 claims description 10
- 230000004913 activation Effects 0.000 claims description 6
- 230000003416 augmentation Effects 0.000 claims description 6
- 230000002194 synthesizing effect Effects 0.000 claims description 6
- 230000015654 memory Effects 0.000 claims description 4
- 230000000694 effects Effects 0.000 description 12
- 230000015556 catabolic process Effects 0.000 description 7
- 238000006731 degradation reaction Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000005457 optimization Methods 0.000 description 3
- 238000009825 accumulation Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000007774 longterm Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000006403 short-term memory Effects 0.000 description 2
- 239000013585 weight reducing agent Substances 0.000 description 2
- 230000002411 adverse Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/73—Deblurring; Sharpening
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Processing (AREA)
Abstract
The application discloses an image rain removing method and system based on a multi-cascade progressive convolution structure, wherein the method comprises the following steps: acquiring a rainy day environment image dataset, and training and optimizing the multi-cascade progressive convolution operator based on the rainy day environment image dataset to obtain an optimized multi-cascade progressive convolution operator; constructing a multi-cascade progressive convolution operator module based on the optimized multi-cascade progressive convolution operator; constructing a progressive cyclic rain removal network based on a multi-cascade progressive convolution operator module; and acquiring an image to be processed, and processing the image to be processed based on a progressive cyclic rain removing network to obtain a processed image. The method can effectively enlarge the receptive field of the convolution layer under the condition of slightly increasing the parameter quantity, and improves the capability of extracting the detail features of the image and the capability of analyzing the global features; the raindrops or rain lines with different sizes and shapes can be captured more carefully, the raindrops or rain lines can be removed, the background details of the original rainless pictures can be effectively reserved, and the rainless picture has wide application prospect.
Description
Technical Field
The application relates to the field of image processing, in particular to an image rain removing method and system based on a multi-cascade progressive convolution structure.
Background
With research and development of artificial intelligence and deep learning, computer vision has been successfully applied to various actual scenes, such as target detection and recognition, unmanned driving, industrial detection, virtual reality technology, etc., as an important research direction in the field. The wide application scene of computer vision also puts forward higher requirements on the preprocessing technology of images, and the completion degree and precision of visual tasks can be effectively improved by acquiring clear and complete images, but in the actual image acquisition process, the camera acquisition place is always outdoors, so that the situation of weather which can cause shielding to the images, such as rainy days or foggy days, is inevitably encountered. If the acquired image is not clear, uncontrollable adverse effects such as reduced accuracy, reduced precision, inability to converge the network and the like can be caused to the computer vision task. Therefore, the acquired rainy day image is recovered, and the removal of raindrops to obtain a clean rainless image is quite significant.
For the current rain removal task, the current main research flow still adopts a deep learning method to construct a proper neural network architecture, trains the network by using a large data set, and enables the network to capture rain mark characteristics and remove the rain mark characteristics by self. However, the existing network often only considers the image characteristic information of the shallow single-scale rainy day, and does not deeply mine the multi-scale and fine characteristics contained in the global characteristic map. The size, the direction and the shape of the rain trace and the rain line in the rain image have great difference, the rain removing effect obtained by adopting only single-scale processing in the network learning process is limited, and the rain trace and the rain line are difficult to remove cleanly.
Disclosure of Invention
The application provides an image rain removing method and system based on a multi-cascade progressive convolution structure, wherein a progressive cyclic rain removing network is constructed by training a multi-cascade progressive convolution operator and based on the multi-cascade progressive convolution operator structure, and then the image to be processed is subjected to repeated cyclic processing based on the progressive cyclic rain removing network, so that the rain removing of the image is completed.
To achieve the above object, the present application provides the following solutions:
an image rain removing method based on a multi-cascade progressive convolution structure comprises the following steps:
acquiring a rainy day environment image dataset, and training and optimizing a multi-cascade progressive convolution operator based on the rainy day environment image dataset to obtain an optimized multi-cascade progressive convolution operator;
constructing a multi-cascade progressive convolution operator module based on the optimized multi-cascade progressive convolution operator;
constructing a progressive cyclic rain removal network based on the multi-cascade progressive convolution operator module;
and acquiring an image to be processed, and processing the image to be processed based on the progressive cyclic rain removing network to obtain a processed image.
Preferably, the method for acquiring the rainy day environment image dataset comprises the following steps:
synthesizing the rainy day environment image dataset based on a linear superposition model;
and dividing the rainy day environment image data set into a training set and a testing set according to a preset proportion.
Preferably, the training method of the optimized multi-cascade progressive convolution operator comprises the following steps:
inputting the training set to a first-layer dimension-lifting convolution operation in the multi-cascade progressive convolution module to perform dimension augmentation of the feature map so as to obtain a global feature map;
based on a four-layer ladder-type convolution structure in the multi-cascade progressive convolution module, performing convolution calculation from top to bottom on the global feature map to obtain a first output result;
splicing the first output results, and integrating characteristic information to obtain characteristic extraction results;
and comparing the feature extraction result with the test set and completing training to obtain an optimized multi-cascade progressive convolution operator.
Preferably, the construction method of the multi-cascade progressive convolution operator module comprises the following steps:
and connecting a plurality of optimized multi-cascade progressive convolution operators in series in a residual jump connection mode to obtain the multi-cascade progressive convolution operator module.
Preferably, the progressive cycle rain removal network construction method comprises the following steps:
constructing a first layer network based on the first convolution layer and the ReLU activation layer;
constructing a second-layer network based on the long-period memory network layer;
constructing a third layer network based on the multi-cascade progressive convolution module;
constructing a fourth layer network based on the second convolution layer;
and constructing the progressive cycle rain removing network based on the first layer network, the second layer network, the third layer network and the fourth layer network.
Preferably, the method for obtaining the processed image includes:
and inputting the image to be processed into the progressive cyclic rain removing network for repeated cyclic processing to obtain a processed image.
The application also provides an image rain removing system based on the multi-cascade progressive convolution structure, which comprises: the device comprises a training unit, an operator module construction unit, a network construction unit and a processing unit;
the training unit is used for acquiring a rainy day environment image data set, and training and optimizing the multi-cascade progressive convolution operator based on the rainy day environment image data set to obtain an optimized multi-cascade progressive convolution operator;
the operator module construction unit is used for constructing a multi-cascade progressive convolution operator module based on the optimized multi-cascade progressive convolution operator;
the network construction unit is used for constructing a progressive cyclic rain removal network based on the multi-cascade progressive convolution operator module;
the processing unit is used for collecting an image to be processed, and processing the image to be processed based on the progressive cycle rain removing network to obtain a processed image.
Preferably, the method for acquiring the rainy day environment image dataset comprises the following steps:
synthesizing the rainy day environment image dataset based on a linear superposition model;
and dividing the rainy day environment image data set into a training set and a testing set according to a preset proportion.
Preferably, the training method of the optimized multi-cascade progressive convolution operator comprises the following steps:
inputting the training set to a first-layer dimension-lifting convolution operation in the multi-cascade progressive convolution module to perform dimension augmentation of the feature map so as to obtain a global feature map;
based on a four-layer ladder-type convolution structure in the multi-cascade progressive convolution module, performing convolution calculation from top to bottom on the global feature map to obtain a first output result;
splicing the first output results, and integrating characteristic information to obtain characteristic extraction results;
and comparing the feature extraction result with the test set and completing training to obtain an optimized multi-cascade progressive convolution operator.
The beneficial effects of this application are:
(1) The receptive field of the convolution layer can be effectively enlarged under the condition of increasing the parameter quantity by a small amount, and the capability of extracting the detail features of the image and the capability of analyzing the global features are improved;
(2) Rain marks or rain lines with different sizes and shapes can be captured more carefully, the rain marks or the rain lines are removed, and the background details of the original rainless picture can be effectively reserved;
(3) The aim of reducing the network parameter quantity is achieved by reasonably adjusting the channel number, so that the learning time and the learning cost are shortened, the network is lighter, and the network can be portable and carried into various vehicle-mounted image processing equipment, and has wide application prospects.
Drawings
For a clearer description of the technical solutions of the present application, the drawings that are required to be used in the embodiments are briefly described below, it being evident that the drawings in the following description are only some embodiments of the present application, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a method according to a first embodiment of the present application;
FIG. 2 is a schematic diagram of a multi-cascade progressive convolution operator structure according to a first embodiment of the present application;
FIG. 3 is a schematic diagram of a multi-cascade progressive convolution operator with reduced parameters according to an embodiment of the present application;
fig. 4 is a block diagram of a progressive cycle rain removal network according to an embodiment of the present application;
fig. 5 is a detailed structural diagram of a progressive cycle rain removal network according to the first embodiment of the present application;
fig. 6 is a schematic system structure of a second embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
In order that the above-recited objects, features and advantages of the present application will become more readily apparent, a more particular description of the invention briefly described above will be rendered by reference to specific embodiments that are illustrated in the appended drawings.
Example 1
In this embodiment, as shown in fig. 1, an image rain removing method based on a multi-cascade progressive convolution structure includes the following steps:
s1, acquiring a rainy day environment image dataset, and training and optimizing a multi-cascade progressive convolution operator based on the rainy day environment image dataset to obtain an optimized multi-cascade progressive convolution operator.
The method for acquiring the rainy day environment image dataset comprises the following steps: synthesizing the rainy day environment image dataset based on a linear superposition model; dividing the rainy day environment image dataset into a training set and a testing set according to a preset proportion.
In this embodiment, a linear superposition model is used to synthesize the rainy day environment dataset, and the model can be expressed as the following formula:
O＝B+S
wherein O is a rain map, B is a clean rain-free background map, and S is a rain trace map. The modeling method assumes that the rain streak of the image is simply superimposed on a clean rain-free background image, and ignores the rain and fog accumulation effect formed by larger rain drops on the background image. And dividing the rainy day environment data set synthesized by the model into a training set and a testing set in proportion.
The training method of the optimized multi-cascade progressive convolution operator comprises the following steps: inputting the training set into a first-layer dimension-lifting convolution operation in a multi-cascade progressive convolution module to perform dimension augmentation of the feature map, so as to obtain a global feature map; based on a four-layer ladder-type convolution structure in the improved structure, performing convolution calculation from top to bottom on the global feature map to obtain a first output result; splicing the first output results, and integrating the characteristic information to obtain a characteristic extraction result; and comparing the feature extraction result with the test set and completing training to obtain an optimized multi-cascade progressive convolution operator, as shown in fig. 2.
In this embodiment, the training set is defined as X, and first a first-layer upwarp convolution operation f is performed upgrad The channel number is expanded from C to 2C through a layer of 1X 1 convolution, the dimension of the feature map is enlarged, the width and the height of the original feature map are not changed in the process, and only the dimension-increasing operation is carried out on the feature map. General purpose medicineAfter the dimension-up calculation, the global feature map X after dimension-up can be obtained i ；
Then through four-layer ladder-type convolution structure in multi-cascade progressive convolution operator, each layer is convolution kernel with 3X 3, channel number is reduced by half in turn, channel number is reduced in layer by layer, and calculation is recorded as f from top to bottom i (i=1, 2,3, 4), input is X i The output is Yi, and Y can be obtained i ＝f i (X i ). Subsequently, the Y obtained i And performing concat splicing operation, integrating feature information through a layer of 1X 1 convolution, outputting a feature map Y after deep fine granularity feature extraction, and finally obtaining the comprehensive result of the convolution feature extraction. The formula is:
X 1 ＝f upgrad (X)
X i ＝f i (X i-1 )i∈[2，4]
Y i ＝(X i )，Y＝f out (concat(Y i ))i∈[1，4]
and comparing the comprehensive result of the feature extraction with the test set and completing training to obtain the optimized multi-cascade progressive convolution operator.
In this embodiment, the multi-cascade progressive convolution operator may be further subjected to light-weight processing, so as to obtain a light-weight structure. The channel dimension-increasing operation performed in the improved first layer 1 multiplied by 1 convolution kernel is removed, and although the original network effect is slightly reduced, the parameter quantity which can be learned in the original network can be greatly reduced, so that the purpose of network weight reduction is achieved, a light structure is obtained, and the device is conveniently mounted on various simple intelligent platforms. The parameter light weight improvement is schematically shown in fig. 3.
S2, constructing a multi-cascade progressive convolution operator module based on the optimized multi-cascade progressive convolution operator.
The construction method of the multi-cascade progressive convolution operator module comprises the following steps: and connecting a plurality of optimized multi-cascade progressive convolution operators in series in a residual jump connection mode to obtain a multi-cascade progressive convolution operator module.
In this embodiment, the feature analysis capability of the single multi-cascade progressive convolution operator is limited, a multi-cascade progressive convolution operator module is formed after multi-level serial connection is needed, and then the multi-cascade progressive convolution operator module is embedded into a rain removing network, so that the purpose of improving the rain removing effect is achieved, and meanwhile, in order to reduce the network degradation effect caused by network deepening, when the operator is actually applied, residual connection is needed to be introduced to reduce the performance reduction caused by network degradation.
S3, constructing a progressive cyclic rain removal network based on the multi-cascade progressive convolution operator module.
The progressive cycle rain removal network construction method comprises the following steps: constructing a first layer network based on the first convolution layer and the ReLU activation layer; constructing a second-layer network based on the long-period memory network layer; constructing a third layer network based on the multi-cascade progressive convolution module; constructing a fourth layer network based on the second convolution layer; and constructing the progressive cyclic rain removing network based on the first layer network, the second layer network, the third layer network and the fourth layer network.
In this embodiment, as shown in fig. 4, the progressive cyclic rain removing network is composed of four parts, and firstly, the input of the network is tensor splicing of a rain map X and a staged rain removing result Y; the first network is the input layer f in Comprises a convolution layer with convolution kernel size of 3×3 and a ReLU activation layer, and the second network is Long short-term memory (LSTM) network layer f re The layer is used for solving the long-term dependence problem caused by multiple times of network circulation and can effectively memorize the input layer f in The obtained feature map information; the third layer network is a multi-cascade progressive convolution structure module f res The module comprises 5 serially connected optimized multi-cascade progressive convolution structures, and degradation effects caused by network deepening are reduced in the structure through a residual jump connection mode; the fourth layer is the output layer fo ut The method comprises a convolution layer with a convolution kernel size of 3 multiplied by 3, which is used for restoring and outputting the final picture to obtain a rain-removed image.
The detailed structure of the progressive cyclic rain removing network is shown in fig. 5.
S4, acquiring an image to be processed, and processing the image to be processed based on the progressive cyclic rain removing network to obtain a processed image. The processing method comprises the following steps: and inputting the image to be processed into the progressive cyclic rain removing network for repeated cyclic processing to obtain a processed image.
In this embodiment, an image to be processed is first acquired; and inputting the image to be processed into a progressive cyclic rain removing network for repeated cyclic processing to obtain a processed image. Let the t-th cycle phase be S t The t-th cycle can be expressed as:
X t ＝f in (Y t-1 ，X in )
S t ＝f re (S t-1 ，X t )
Y t ＝f out (f res (S t ))
after cyclic processing, a processed image Y is obtained t . In this embodiment, in order to achieve the network new energy optimization, the number of loops is set to 6.
Example two
In a second embodiment, as shown in fig. 6, an image rain removing system based on a multi-cascade progressive convolution structure is characterized by comprising: the device comprises a training unit, an operator module construction unit, a network construction unit and a processing unit;
the training unit is used for acquiring a rainy day environment image dataset, and training and optimizing the multi-cascade progressive convolution operator based on the rainy day environment image dataset to obtain the optimized multi-cascade progressive convolution operator. The workflow of the training unit comprises: synthesizing the rainy day environment image dataset based on a linear superposition model; dividing a rainy day environment image dataset into a training set and a testing set according to a preset proportion; inputting the training set into a first-layer dimension-lifting convolution operation in a multi-cascade progressive convolution module to perform dimension augmentation of the feature map, so as to obtain a global feature map; based on a four-layer ladder-type convolution structure in the improved structure, performing convolution calculation from top to bottom on the global feature map to obtain a first output result; splicing the first output results, and integrating the characteristic information to obtain a characteristic extraction result; and comparing the feature extraction result with the test set and completing training to obtain the optimized multi-cascade progressive convolution operator.
In this embodiment, a linear superposition model is used to synthesize the rainy day environment dataset, and the model can be expressed as the following formula:
O＝B+S
wherein O is a rain map, B is a clean rain-free background map, and S is a rain trace map. The modeling method assumes that the rain streak of the image is simply superimposed on a clean rain-free background image, and ignores the rain and fog accumulation effect formed by larger rain drops on the background image. And dividing the rainy day environment data set synthesized by the model into a training set and a testing set in proportion. Defining training set as X, first performing first-layer upwarp convolution operation f upgrad The channel number is expanded from C to 2C through a layer of 1X 1 convolution, the dimension of the feature map is enlarged, the width and the height of the original feature map are not changed in the process, and only the dimension-increasing operation is carried out on the feature map. The global feature map X after the dimension increase can be obtained after the dimension increase calculation i The method comprises the steps of carrying out a first treatment on the surface of the Then through four-layer ladder-type convolution structure in multi-cascade progressive convolution operator, each layer is convolution kernel with 3X 3, channel number is reduced by half in turn, channel number is reduced in layer by layer, and calculation is recorded as f from top to bottom i (i=1, 2,3, 4), input is X i Output is Y i Y can be obtained i ＝f i (X i ). Subsequently, the Y obtained i And performing concat splicing operation, integrating feature information through a layer of 1X 1 convolution, outputting a feature map Y after deep fine granularity feature extraction, and finally obtaining the comprehensive result of the convolution feature extraction. The formula is:
X 1 ＝f upgrad (X)
X i ＝f i (X i-1 )i∈[2，4]
Y i ＝(X i )，Y＝f out (concat(Y i ))i∈[1，4]
and comparing the comprehensive result of the feature extraction with the test set and completing training to obtain the optimized multi-cascade progressive convolution operator.
Furthermore, the multi-cascade progressive convolution operator can be subjected to light weight processing, so that a light weight structure is obtained. The channel dimension-increasing operation performed in the improved first layer 1 multiplied by 1 convolution kernel is removed, and although the original network effect is slightly reduced, the parameter quantity which can be learned in the original network can be greatly reduced, so that the purpose of network weight reduction is achieved, a light structure is obtained, and the device is conveniently mounted on various simple intelligent platforms.
The operator module construction unit is used for constructing the multi-cascade progressive convolution operator module based on the optimized multi-cascade progressive convolution operator. The workflow of the operator module construction unit includes: and connecting a plurality of optimized multi-cascade progressive convolution operators in series in a residual jump connection mode to obtain a multi-cascade progressive convolution operator module.
In this embodiment, the feature analysis capability of a single multi-cascade progressive convolution operator is limited, and after multi-level serial connection is needed, the single multi-cascade progressive convolution operator is embedded into a rain removing network to achieve the purpose of improving the rain removing effect, and in order to reduce the network degradation effect caused by network deepening, in practical application of the operator, residual connection needs to be introduced to reduce performance degradation caused by network degradation.
The network construction unit is used for constructing a progressive cyclic rain removal network based on the multi-cascade progressive convolution operator module. The workflow of the network construction unit includes: constructing a first layer network based on the first convolution layer and the ReLU activation layer; constructing a second-layer network based on the long-period memory network layer; constructing a third layer network based on the multi-cascade progressive convolution module; constructing a fourth layer network based on the second convolution layer; and constructing the progressive cyclic rain removing network based on the first layer network, the second layer network, the third layer network and the fourth layer network.
In the embodiment, the progressive cycle rain removing network is composed of four parts, and firstly, the input of the network is tensor splicing of a rain map X and a staged rain removing result Y; the first network is the input layer f in Comprises a convolution layer with convolution kernel size of 3×3 and a ReLU activation layer, and the second network is Long short-term memory (LSTM) network layer f re The layer is used for solving the long-term dependence problem caused by multiple times of network circulation and can effectively memorize the input layer f in The obtained feature map information; the third layer network is a multi-cascade progressive convolution structure module f res The module includes optimization of 5 serial connectionsThe back multi-cascade progressive convolution structure reduces degradation effect brought by network deepening in a residual jump connection mode in the structure; the fourth layer is the output layer f out The method comprises a convolution layer with a convolution kernel size of 3 multiplied by 3, which is used for restoring and outputting the final picture to obtain a rain-removed image. Finally, a progressive cyclic rain removing network is obtained.
The processing unit is used for collecting images to be processed, and processing the images to be processed based on the progressive cyclic rain removing network to obtain processed images. The processing unit workflow includes: and inputting the image to be processed into the progressive cyclic rain removing network for repeated cyclic processing to obtain a processed image.
In this embodiment, an image to be processed is first acquired; and inputting the image to be processed into the progressive cyclic rain removing network for repeated cyclic processing to obtain a processed image. Let the t-th cycle phase be S t The t-th cycle can be expressed as:
X t ＝f in (Y t-1 ，X in )
S t ＝f re (S t-1 ，X t )
Y t ＝f out (f res (S t ))
after cyclic processing, a processed image Y is obtained t . In this embodiment, in order to achieve the network new energy optimization, the number of loops is set to 6.
The foregoing embodiments are merely illustrative of the preferred embodiments of the present application and are not intended to limit the scope of the present application, and various modifications and improvements made by those skilled in the art to the technical solutions of the present application should fall within the protection scope defined by the claims of the present application.
Claims (9)
1. An image rain removing method based on a multi-cascade progressive convolution structure is characterized by comprising the following steps of:
acquiring a rainy day environment image dataset, and training and optimizing a multi-cascade progressive convolution operator based on the rainy day environment image dataset to obtain an optimized multi-cascade progressive convolution operator;
constructing a multi-cascade progressive convolution operator module based on the optimized multi-cascade progressive convolution operator;
constructing a progressive cyclic rain removal network based on the multi-cascade progressive convolution operator module;
and acquiring an image to be processed, and processing the image to be processed based on the progressive cyclic rain removing network to obtain a processed image.
2. The method for image rain removal based on a multi-cascade progressive convolution structure according to claim 1, wherein the method for acquiring the rainy day environment image dataset comprises:
synthesizing the rainy day environment image dataset based on a linear superposition model;
and dividing the rainy day environment image data set into a training set and a testing set according to a preset proportion.
3. The image rain removing method based on the multi-cascade progressive convolution structure according to claim 2, wherein the training method of the optimized multi-cascade progressive convolution operator comprises the following steps:
inputting the training set to a first-layer dimension-lifting convolution operation in the multi-cascade progressive convolution module to perform dimension augmentation of the feature map so as to obtain a global feature map;
based on a four-layer ladder-type convolution structure in the multi-cascade progressive convolution module, performing convolution calculation from top to bottom on the global feature map to obtain a first output result;
splicing the first output results, and integrating characteristic information to obtain characteristic extraction results;
and comparing the feature extraction result with the test set and completing training to obtain an optimized multi-cascade progressive convolution operator.
4. The image rain removing method based on the multi-cascade progressive convolution structure according to claim 1, wherein the construction method of the multi-cascade progressive convolution operator module comprises the following steps:
and connecting a plurality of optimized multi-cascade progressive convolution operators in series in a residual jump connection mode to obtain the multi-cascade progressive convolution operator module.
5. The image rain removing method based on the multi-cascade progressive convolution structure according to claim 1, wherein the progressive cyclic rain removing network construction method comprises the following steps:
constructing a first layer network based on the first convolution layer and the ReLU activation layer;
constructing a second-layer network based on the long-period memory network layer;
constructing a third layer network based on the multi-cascade progressive convolution module;
constructing a fourth layer network based on the second convolution layer;
and constructing the progressive cycle rain removing network based on the first layer network, the second layer network, the third layer network and the fourth layer network.
6. The image rain removing method based on the multi-cascade progressive convolution structure according to claim 1, wherein the method for obtaining the processed image comprises the following steps:
and inputting the image to be processed into the progressive cyclic rain removing network for repeated cyclic processing to obtain a processed image.
7. An image rain removal system based on a multi-cascade progressive convolution structure, comprising: the device comprises a training unit, an operator module construction unit, a network construction unit and a processing unit;
the training unit is used for acquiring a rainy day environment image data set, and training and optimizing the multi-cascade progressive convolution operator based on the rainy day environment image data set to obtain an optimized multi-cascade progressive convolution operator;
the operator module construction unit is used for constructing a multi-cascade progressive convolution operator module based on the optimized multi-cascade progressive convolution operator;
the network construction unit is used for constructing a progressive cyclic rain removal network based on the multi-cascade progressive convolution operator module;
the processing unit is used for collecting an image to be processed, and processing the image to be processed based on the progressive cycle rain removing network to obtain a processed image.
8. The image rain removal system based on a multi-cascade progressive convolution structure of claim 7, wherein the method of acquiring the rainy day environment image dataset comprises:
synthesizing the rainy day environment image dataset based on a linear superposition model;
and dividing the rainy day environment image data set into a training set and a testing set according to a preset proportion.
9. The image rain removal system based on the multi-cascade progressive convolution structure according to claim 8, wherein the training method of the optimized multi-cascade progressive convolution operator comprises the following steps:
inputting the training set to a first-layer dimension-lifting convolution operation in the multi-cascade progressive convolution module to perform dimension augmentation of the feature map so as to obtain a global feature map;
based on a four-layer ladder-type convolution structure in the multi-cascade progressive convolution module, performing convolution calculation from top to bottom on the global feature map to obtain a first output result;
splicing the first output results, and integrating characteristic information to obtain characteristic extraction results;
and comparing the feature extraction result with the test set and completing training to obtain an optimized multi-cascade progressive convolution operator.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310053373.4A CN116091350A (en) | 2023-02-03 | 2023-02-03 | Image rain removing method and system based on multi-cascade progressive convolution structure |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310053373.4A CN116091350A (en) | 2023-02-03 | 2023-02-03 | Image rain removing method and system based on multi-cascade progressive convolution structure |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116091350A true CN116091350A (en) | 2023-05-09 |
Family
ID=86204105
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310053373.4A Pending CN116091350A (en) | 2023-02-03 | 2023-02-03 | Image rain removing method and system based on multi-cascade progressive convolution structure |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116091350A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10032256B1 (en) * | 2016-11-18 | 2018-07-24 | The Florida State University Research Foundation, Inc. | System and method for image processing using automatically estimated tuning parameters |
| CN110503613A (en) * | 2019-08-13 | 2019-11-26 | 电子科技大学 | Based on the empty convolutional neural networks of cascade towards removing rain based on single image method |
| CN110796623A (en) * | 2019-10-31 | 2020-02-14 | 上海融军科技有限公司 | Infrared image rain removing method and device based on progressive residual error network |
| AU2020100196A4 (en) * | 2020-02-08 | 2020-03-19 | Juwei Guan | A method of removing rain from single image based on detail supplement |
| CN113344825A (en) * | 2021-07-02 | 2021-09-03 | 南昌航空大学 | Image rain removing method and system |
| CN113450288A (en) * | 2021-08-04 | 2021-09-28 | 广东工业大学 | Single image rain removing method and system based on deep convolutional neural network and storage medium |
-
2023
- 2023-02-03 CN CN202310053373.4A patent/CN116091350A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10032256B1 (en) * | 2016-11-18 | 2018-07-24 | The Florida State University Research Foundation, Inc. | System and method for image processing using automatically estimated tuning parameters |
| CN110503613A (en) * | 2019-08-13 | 2019-11-26 | 电子科技大学 | Based on the empty convolutional neural networks of cascade towards removing rain based on single image method |
| CN110796623A (en) * | 2019-10-31 | 2020-02-14 | 上海融军科技有限公司 | Infrared image rain removing method and device based on progressive residual error network |
| AU2020100196A4 (en) * | 2020-02-08 | 2020-03-19 | Juwei Guan | A method of removing rain from single image based on detail supplement |
| CN113344825A (en) * | 2021-07-02 | 2021-09-03 | 南昌航空大学 | Image rain removing method and system |
| CN113450288A (en) * | 2021-08-04 | 2021-09-28 | 广东工业大学 | Single image rain removing method and system based on deep convolutional neural network and storage medium |
Non-Patent Citations (3)
| Title |
|---|
| KAIGE WANG ET AL.: "An End-to-End Cascaded Image Deraining and Object Detection Neural Network", 《IEEE ROBOTICS AND AUTOMATION LETTERS》, 19 July 2022 (2022-07-19), pages 9541 - 9548 * |
| 柳长源;王琪;毕晓君;: "基于多通道多尺度卷积神经网络的单幅图像去雨方法", 电子与信息学报, no. 09, 15 September 2020 (2020-09-15), pages 224 - 231 * |
| 黄奇: "基于深度学习的单幅图像去雨方法研究", 《中国优秀硕士学位论文全文数据库信息科技辑》, 15 October 2022 (2022-10-15), pages 138 - 314 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110348376B (en) | Pedestrian real-time detection method based on neural network | |
| CN111046821B (en) | Video behavior recognition method and system and electronic equipment | |
| CN111460914B (en) | Pedestrian re-identification method based on global and local fine granularity characteristics | |
| CN112766160A (en) | Face replacement method based on multi-stage attribute encoder and attention mechanism | |
| CN111401293B (en) | Gesture recognition method based on Head lightweight Mask scanning R-CNN | |
| Zhang et al. | Single image dehazing via dual-path recurrent network | |
| CN109886159B (en) | Face detection method under non-limited condition | |
| CN111626960A (en) | Image defogging method, terminal and computer storage medium | |
| CN111539888B (en) | Neural network image defogging method based on pyramid channel feature attention | |
| CN113269224B (en) | Scene image classification method, system and storage medium | |
| CN110852199A (en) | Foreground extraction method based on double-frame coding and decoding model | |
| CN115035567A (en) | Model training, incomplete face image recognition and reconstruction method, equipment and medium | |
| CN113378812A (en) | Digital dial plate identification method based on Mask R-CNN and CRNN | |
| CN116485646A (en) | Micro-attention-based light-weight image super-resolution reconstruction method and device | |
| CN111612803B (en) | Vehicle image semantic segmentation method based on image definition | |
| CN113177956A (en) | Semantic segmentation method for unmanned aerial vehicle remote sensing image | |
| CN117710841A (en) | Small target detection method and device for aerial image of unmanned aerial vehicle | |
| CN116433822B (en) | Neural radiation field training method, device, equipment and medium | |
| CN116091350A (en) | Image rain removing method and system based on multi-cascade progressive convolution structure | |
| CN106934344B (en) | quick pedestrian detection method based on neural network | |
| Aldabbagh et al. | Classification of chili plant growth using deep learning | |
| CN112633089B (en) | Video pedestrian re-identification method, intelligent terminal and storage medium | |
| CN115273009A (en) | Road crack detection method and system based on deep learning | |
| CN117152000B (en) | Rainy day image-clear background paired data set manufacturing method and device and application thereof | |
| CN117392440B (en) | Textile fabric retrieval method and system based on tissue structure and color classification |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |