CN113393385A - Unsupervised rain removal method, system, device and medium based on multi-scale fusion - Google Patents
Unsupervised rain removal method, system, device and medium based on multi-scale fusion Download PDFInfo
- Publication number
- CN113393385A CN113393385A CN202110515593.5A CN202110515593A CN113393385A CN 113393385 A CN113393385 A CN 113393385A CN 202110515593 A CN202110515593 A CN 202110515593A CN 113393385 A CN113393385 A CN 113393385A
- Authority
- CN
- China
- Prior art keywords
- rain
- image
- scale
- micro
- clean
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 230000004927 fusion Effects 0.000 title claims abstract description 25
- 238000005070 sampling Methods 0.000 claims abstract description 20
- 238000007781 pre-processing Methods 0.000 claims abstract description 16
- 238000012549 training Methods 0.000 claims abstract description 16
- 230000006870 function Effects 0.000 claims description 39
- 230000015654 memory Effects 0.000 claims description 13
- 230000003042 antagnostic effect Effects 0.000 claims 1
- 238000012545 processing Methods 0.000 abstract description 10
- 238000001514 detection method Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000013135 deep learning Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000004088 simulation Methods 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000008092 positive effect Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000007723 transport mechanism Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/73—Deblurring; Sharpening
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/088—Non-supervised learning, e.g. competitive learning
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Molecular Biology (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- General Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses an unsupervised rain removal method, a system, a device and a medium based on multi-scale fusion, wherein the method comprises the following steps: preprocessing the rain-carrying image to obtain a plurality of scales of micro rain-removing images; determining clean images corresponding to the micro rain removing images with different scales according to the micro rain removing images and the confrontation model; up-sampling the clean image to determine an up-sampled image; determining a loss function of the countermeasure model according to a reconstruction error between the sampled image on the current scale and the clean image on the previous scale; iterating the loss function to a minimum value, the countermeasure model converges, by which a clean image without rain can be obtained from the image with rain. The embodiment of the application adopts an unsupervised rain removing method, a large number of pairs of images with rain and without rain are not required to be prepared as a training set, the acquisition difficulty of the training set is reduced, the dependence on the training set is reduced, and the generalization of a rain removing system can be effectively provided. The method and the device can be widely applied to the field of image processing.
Description
Technical Field
The present application relates to the field of image processing, and in particular, to an unsupervised rain removal method, system, device, and medium based on multi-scale fusion.
Background
Computer vision systems are commonly used in daily life of people at present, and the vision systems collect and capture images or videos of real environments and realize detection, identification, prejudgment and the like of targets through analysis processing means such as feature extraction and the like. For an outdoor vision system, the images collected by the outdoor vision system are often affected by weather such as rain, fog, snow, etc., and rain is the more common of the three. The rain-carrying image can affect the accuracy of image feature extraction, and further affect subsequent detection and identification, so that the rain-carrying image needs to be subjected to image enhancement processing to achieve the purpose of removing rain from the image.
In general, rain image enhancement algorithms can be divided into three broad categories: the first is to adopt the way processing of detecting + mending, detect the position of the rain at first, and then utilize the correlation of the peripheral pixel to mend, the second is based on the way of layer separation, regard rain and background picture as two kinds of signals, and add different a priori assumptions to these two kinds of signals and separate. The third is a method based on deep learning, which is popular in recent years, and most of the methods adopt a supervised mode to train a network and learn a mapping from rain to no rain. However, in the related art, the method based on deep learning mostly uses supervised learning algorithms, a large number of pairs of images with rain and without rain are required during training, the requirement on a training set is high, and the trained network may have dependence on the training set and low generalization capability.
Disclosure of Invention
The present application is directed to solving, at least to some extent, one of the technical problems in the related art. Therefore, the application provides an unsupervised rain removal method, a system, a device and a medium based on multi-scale fusion, and the generalization of the rain removal method can be effectively improved.
In a first aspect, an embodiment of the present application provides an unsupervised rain removal method based on multi-scale fusion, including: preprocessing the rain-carrying image, and determining micro rain-removing images with multiple scales; determining clean images corresponding to the micro rain removing images with different scales according to the micro rain removing images and the confrontation model; up-sampling the clean image to determine an up-sampled image; determining a loss function of the countermeasure model according to a reconstruction error between the up-sampled image of the current scale and the clean image of the previous scale; and iterating the loss function, and updating the parameters of the countermeasure model according to the loss function until the countermeasure model converges.
Optionally, the preprocessing the rained image to determine a plurality of scales of micro-raining images includes: performing one-dimensional Gaussian convolution along the main direction of the rain image to determine an initial micro rain-removing image; down-sampling the initial micro rain-removing image to determine a sampling result; and performing one-dimensional Gaussian convolution on the sampling result along the main direction to determine a micro rain removal image of the next scale.
Optionally, the determining the main direction comprises: calculating the gradient direction of raindrops in each pixel point in the image with rain; and determining the main direction of the rain image according to the mean value of the gradient directions.
Optionally, the confrontation model specifically comprises a generator and a discriminator; the generator is used for generating a clean image corresponding to the scale micro rain-removing image according to the micro rain-removing image; the discriminator is used for judging the clean image and outputting a judgment result.
Optionally, the determining a loss function of the countermeasure model according to a reconstruction error between the current-scale up-sampled image and the previous-scale clean image includes:
the penalty function of the countermeasure model is:
wherein L is the loss function, i is the scale serial number of the micro rain-removing image,
for the reconstruction error between the up-sampled image at the i +1 th scale and the micro-degrain image at the i-th scale,
and (5) resisting loss for the generation corresponding to the ith scale.
Optionally, the generating counteraction loss
The method specifically comprises the following steps:
wherein E represents the mathematical expectation, DiRepresents said discriminator, GiRepresents said generator, JiRepresenting said micro-raining images at different scales, IcRepresenting the clean graph corresponding to the micro rain removing image, and c representing the serial number of the clean graph.
Optionally, the generator comprises a convolutional layer, an activation layer and a regularization layer.
In a second aspect, an embodiment of the present application provides an unsupervised rain removal system based on multi-scale fusion, including: the preprocessing module is used for preprocessing the rain-carrying image and determining the micro rain-removing images with multiple scales; the rain removing module is used for determining clean images corresponding to the micro rain removing images with different scales according to the micro rain removing images and the confrontation model; the loss function constructing module is used for performing up-sampling on the clean image and determining an up-sampled image; and determining a loss function of the countermeasure model according to a reconstruction error between the current scale up-sampled image and the previous scale clean image; and the model training module is used for iterating the loss function and updating the parameters of the countermeasure model according to the loss function until the countermeasure model converges.
In a third aspect, an embodiment of the present application provides an apparatus, including: at least one processor; at least one memory for storing at least one program; when executed by the at least one processor, cause the at least one processor to implement the multi-scale fusion based unsupervised rain shedding method according to the first aspect.
In a fourth aspect, embodiments of the present application provide a computer storage medium, in which a processor-executable program is stored, and when the processor-executable program is executed by the processor, the processor-executable program is configured to implement the unsupervised rain removal method based on multi-scale fusion according to the first aspect.
The beneficial effects of the embodiment of the application are as follows: preprocessing the rain-carrying image to obtain a plurality of micro rain-removing images with different scales; determining clean images corresponding to the micro rain removing images with different scales according to the micro rain removing images and the confrontation model; performing up-sampling on the obtained clean image to determine an up-sampled image; determining a loss function of the countermeasure model according to a reconstruction error between the up-sampled image of the current scale and the clean image of the previous scale; in the continuous iterative training process, the loss function is iterated to the minimum value, the confrontation model converges, and a clean image without rain can be obtained according to the slightly rain-removed image through the confrontation model. Compared with the supervised region method, the region method provided by the application reduces the acquisition difficulty of the training set, reduces the dependence on the training set and can effectively provide the generalization of the rain removing system. The method and the device can be widely applied to the field of image processing.
Drawings
The accompanying drawings are included to provide a further understanding of the claimed subject matter and are incorporated in and constitute a part of this specification, illustrate embodiments of the subject matter and together with the description serve to explain the principles of the subject matter and not to limit the subject matter.
FIG. 1 is a flowchart illustrating steps of an unsupervised rain removal method based on multi-scale fusion according to an embodiment of the present disclosure;
FIG. 2 is a flowchart of the steps of pre-processing a rain image according to an embodiment of the present disclosure;
FIG. 3 is a schematic diagram of a network architecture of an countermeasure model according to an embodiment of the present application;
fig. 4 is a schematic architecture diagram of an unsupervised rain removal system based on multi-scale fusion according to an embodiment of the present application;
fig. 5 is a device according to an embodiment of the present disclosure.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
It should be noted that although functional block divisions are provided in the system drawings and logical orders are shown in the flowcharts, in some cases, the steps shown and described may be performed in different orders than the block divisions in the systems or in the flowcharts. The terms first, second and the like in the description and in the claims, and the drawings described above, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.
In the whole process of the vision system, the quality of the raw data is very important, and the poor quality of the raw data may cause inaccuracy of extracted features, thereby affecting subsequent detection and identification. For an outdoor vision system, images acquired by the outdoor vision system are often affected by rain, and all acquired images are rain images. In order to improve the accuracy of the post-image detection and analysis, it is very important to perform image enhancement processing on the rain-bearing image. In addition, in the rain image enhancement algorithm, a deep learning-based method is popular in recent years, and a network is often trained in a supervised manner to learn a mapping from rain to no rain. However, this method requires a large number of pairs of images with rain and without rain, the requirement on the training set is high, and the trained network may have the dependency of the training set and is limited greatly.
Based on this, the embodiment of the application provides an unsupervised rain removing method, an unsupervised rain removing system, an unsupervised rain removing device and a medium based on multi-scale fusion, a clean rain removing image is obtained through a confrontation model in an unsupervised mode, a large number of images with rain and without rain are not needed, the dependence on a training set is effectively reduced, and the unsupervised rain removing method, the unsupervised rain removing system, the unsupervised rain removing device and the medium have a positive effect on improving the generalization of a rain removing.
The embodiments of the present application will be further explained with reference to the drawings.
Referring to fig. 1, fig. 1 is a flowchart illustrating steps of an unsupervised rain removal method based on multi-scale fusion provided by an embodiment of the present application, including, but not limited to, steps S100-S140:
s100, preprocessing the rain-carrying image, and determining micro rain-removing images with multiple scales;
specifically, the outdoor vision system acquires a large number of rain images in rainy days, and the rain images are preprocessed to obtain a plurality of micro rain removing images with different scales. And a little, which means that the slightly rain-removed image is not a clean image after completely removing rain, and only is preprocessed on the basis of the rain-carrying image, and the subsequent rain removing step is performed on the basis of the slightly rain-removed image.
Referring to fig. 2, fig. 2 is a flowchart illustrating steps of the rain image preprocessing provided in the embodiment of the present application, where the method includes, but is not limited to, steps S101 to S103:
s101, performing one-dimensional Gaussian convolution along the main direction of the rain image to determine an initial micro rain image;
specifically, the main direction of the rain image is determined, and one-dimensional Gaussian convolution is performed along the main direction of the rain image to obtain an initial micro rain-removing image. The main direction of the rain-carrying image refers to the main direction of raindrops in the rain-carrying image, the mean value of the gradient directions of the raindrops is determined according to the gradient direction of the raindrops in each pixel point in the rain-carrying image, the mean value is used as the main direction of the rain-carrying image, and the main direction can be applied to the subsequent image convolution step.
S102, down-sampling the initial micro-rain-removing image to determine a sampling result;
specifically, the initial micro-raining image is down-sampled, the scale of the image is reduced, and a sampling result is obtained.
S103, performing one-dimensional Gaussian convolution on the sampling result along the main direction, and determining a micro rain removing image of the next scale;
specifically, the sampling result obtained in step S102 is subjected to one-dimensional gaussian convolution along the above-described main direction, and a micro-raining image of the next scale is determined.
Through steps S101 to S103, the embodiment of the present application obtains multiple samples by alternately performing one-dimensional gaussian convolution and down-samplingA micro-raining image of different scales. Taking steps S101-S103 as an example, assume that the initial micro-raining image resulting from convolution of the rain-bearing image is J0The image of the little rain obtained in step S103 can be represented by J1To represent, J1Is J0The image of the next scale. By analogy, for J1The micro rain-removing image J of the next scale can be obtained by carrying out down-sampling and one-dimensional convolution processing2And then, the downsampling and the convolution are carried out continuously, so that the micro rain-removing images with the quantity required by the user can be obtained, and the multi-scale image processing is carried out.
S110, determining clean images corresponding to the micro rain removing images with different scales according to the micro rain removing images and the confrontation model;
specifically, the micro rain removing image obtained in step S100 is input into the countermeasure model, and the countermeasure model outputs a clean image corresponding to the micro rain removing image in different scales, where the clean image is a rain-free image in different scales.
The countermeasure model in the embodiment of the application comprises a generator and a discriminator, wherein the generator is used for generating a clean image corresponding to the scale micro rain removing image according to the micro rain removing image; the discriminator is used for judging the clean image and outputting a judgment result. The generator and the arbiter in the embodiment of the present application may use a network architecture in the related art, and the embodiment of the present application does not specifically limit the specific configurations of the generator and the arbiter. The generator aims at generating data enough for simulation, the discriminator aims at judging the authenticity of the data generated by the generator, the generator and the discriminator continuously adjust parameters in the whole optimization process of the loss function of the countermeasure model, asynchronous iterative updating is carried out, and the simulation degree of the data generated by the generator and the accuracy of judging the authenticity of the data by the discriminator are continuously improved.
S120, performing up-sampling on the clean image to determine an up-sampled image;
specifically, clean images of different scales are up-sampled, and up-sampled images of multiple scales are obtained.
S130, determining a loss function of the countermeasure model according to a reconstruction error between the up-sampled image of the current scale and the clean image of the previous scale;
specifically, as can be seen from step S102 and step S120, the present application obtains pictures of different scales through upsampling and downsampling, and therefore, an upsampled image of a current scale can be compared with a clean image of a previous scale. Assuming that the scale of the current clean image is i +1, the corresponding up-sampled image can be compared with the clean image with the scale i, in the embodiment of the present application, a loss function is added in a multi-scale network architecture, consistency between the up-sampled clean image with each scale and the clean image with the previous scale is constrained, and the loss function of the countermeasure model is determined according to a reconstruction error between the up-sampled image with the current scale and the clean image with the previous scale. The expression for this loss function is as follows:
wherein L is a loss function, i is a scale serial number of the micro rain removing image,
is the reconstruction error between the up-sampled image of the (i + 1) th scale and the clean image of the (i) th scale,
and (5) resisting loss for the generation corresponding to the ith scale. The generation counteracts the loss
The method specifically comprises the following steps:
wherein E represents the mathematical expectation, DiRepresentative discriminator, GiRepresentative Generator, JiRepresenting micro-raining images of different scales, IcRepresenting the clean image corresponding to the micro-raining image, and c representing the serial number of the clean image.
And S140, iterating the loss function, and updating the parameters of the countermeasure model according to the loss function until the countermeasure model converges.
Specifically, the loss function is iterated continuously to minimize the loss function, and parameters of a generator and a discriminator in the countermeasure model are adjusted continuously in the iteration process of the loss function. After the converged confrontation model is obtained, the image with rain is input into the model, and the confrontation model outputs a corresponding clean image without rain.
Referring to fig. 3, fig. 3 is a schematic diagram of a network architecture of an countermeasure model according to an embodiment of the present application, and as shown in fig. 3, J is an input image with rain, and an initial micro rain-removed image J is obtained through one-dimensional gaussian convolution0From J0Obtaining micro rain removing images J with different scales by analogy at first1、J2The micro rain-removing images with different scales pass through generators G with different scales0、G1Waiting to generate clean images I of different scales0、I1Etc., the clean maps generated by these generators are input to discriminators D of different scales0、D1And the like, comparing and judging with the real rainless graph, outputting a judgment result, and when the loss function of the whole network reaches the minimum value, considering that the generator can generate a rainless graph which is simulated enough to resist network convergence.
Referring to fig. 4, fig. 4 is a schematic diagram of an architecture of an unsupervised rain removal system based on multi-scale fusion according to an embodiment of the present application, where the system 400 includes: a preprocessing module 410, a rain removal module 420, a loss function construction module 430, and a model training module 440. The preprocessing module is used for preprocessing the rain-carrying image and determining the micro rain-removing images with multiple scales; the rain removing module is used for determining clean images corresponding to the micro rain removing images with different scales according to the micro rain removing images and the confrontation model; the loss function constructing module is used for performing up-sampling on the clean image and determining an up-sampled image; the method comprises the steps of obtaining a sampled image on a current scale and a clean image on a previous scale, and determining a loss function of a countermeasure model according to a reconstruction error between the sampled image on the current scale and the clean image on the previous scale; the model training module is used for iterating the loss function and updating the parameters of the countermeasure model according to the loss function until the countermeasure model converges.
Referring to fig. 5, fig. 5 illustrates an apparatus 500 according to an embodiment of the present application, where the apparatus 500 includes at least one processor 510 and at least one memory 520 for storing at least one program; in fig. 5, a processor and a memory are taken as an example.
The processor and memory may be connected by a bus or other means, such as by a bus in FIG. 5.
The memory, which is a non-transitory computer readable storage medium, may be used to store non-transitory software programs as well as non-transitory computer executable programs. Further, the memory may include high speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory optionally includes memory located remotely from the processor, and these remote memories may be connected to the device over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
Another embodiment of the present application also provides an apparatus that may be used to perform the control method as in any of the embodiments above, for example, performing the method steps of fig. 1 described above.
The above-described embodiments of the apparatus are merely illustrative, wherein the units illustrated as separate components may or may not be physically separate, i.e. may be located in one place, or may also be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
The embodiment of the application also discloses a computer storage medium, wherein a program executable by a processor is stored, and the program executable by the processor is used for realizing the unsupervised rain removal method based on multi-scale fusion when being executed by the processor.
One of ordinary skill in the art will appreciate that all or some of the steps, systems, and methods disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). The term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, as is well known to those of ordinary skill in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by a computer. In addition, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media as known to those skilled in the art.
While the preferred embodiments of the present invention have been described, the present invention is not limited to the above embodiments, and those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and such equivalent modifications or substitutions are included in the scope of the present invention defined by the claims.
Claims (10)
1. An unsupervised rain removal method based on multi-scale fusion is characterized by comprising the following steps:
preprocessing the rain-carrying image, and determining micro rain-removing images with multiple scales;
determining clean images corresponding to the micro rain removing images with different scales according to the micro rain removing images and the confrontation model;
up-sampling the clean image to determine an up-sampled image;
determining a loss function of the countermeasure model according to a reconstruction error between the up-sampled image of the current scale and the clean image of the previous scale;
and iterating the loss function, and updating the parameters of the countermeasure model according to the loss function until the countermeasure model converges.
2. The unsupervised rain removal method based on multi-scale fusion of claim 1, wherein the pre-processing the rain-bearing image to determine the micro-rain removal image with multiple scales comprises:
performing one-dimensional Gaussian convolution along the main direction of the rain image to determine an initial micro rain-removing image;
down-sampling the initial micro rain-removing image to determine a sampling result;
and performing one-dimensional Gaussian convolution on the sampling result along the main direction to determine a micro rain removal image of the next scale.
3. The unsupervised rain shedding method based on multi-scale fusion according to claim 1, wherein the determining of the principal direction comprises:
calculating the gradient direction of raindrops in each pixel point in the image with rain;
and determining the main direction of the rain image according to the mean value of the gradient directions.
4. The unsupervised rain removal method based on multi-scale fusion according to claim 1, characterized in that the confrontation model comprises in particular a generator and a discriminator;
the generator is used for generating a clean image corresponding to the scale micro rain-removing image according to the micro rain-removing image;
the discriminator is used for judging the clean image and outputting a judgment result.
5. The unsupervised rain removal method based on multi-scale fusion of claim 3, wherein the determining the penalty function of the countermeasure model according to the reconstruction error between the current-scale up-sampled image and the last-scale clean image comprises:
the penalty function of the countermeasure model is:
wherein L is the loss function, i is the scale serial number of the micro rain-removing image,
for the reconstruction error between the up-sampled image at the i +1 th scale and the micro-degrain image at the i-th scale,
and (5) resisting loss for the generation corresponding to the ith scale.
6. The unsupervised rain shedding method based on multi-scale fusion of claim 4, wherein the generating of the antagonistic loss
The method specifically comprises the following steps:
wherein E represents the mathematical expectation, DiRepresents said discriminator, GiRepresents said generator, JiRepresenting said micro-raining images at different scales, IcRepresenting the clean graph corresponding to the micro rain removing image, and c representing the serial number of the clean graph.
7. The unsupervised rain shedding method based on multi-scale fusion of claim 3, wherein the generator comprises a convolutional layer, an active layer and a regularization layer.
8. An unsupervised rain removal system based on multi-scale fusion, comprising:
the preprocessing module is used for preprocessing the rain-carrying image and determining the micro rain-removing images with multiple scales;
the rain removing module is used for determining clean images corresponding to the micro rain removing images with different scales according to the micro rain removing images and the confrontation model;
the loss function constructing module is used for performing up-sampling on the clean image and determining an up-sampled image; and determining a loss function of the countermeasure model according to a reconstruction error between the current scale up-sampled image and the previous scale clean image;
and the model training module is used for iterating the loss function and updating the parameters of the countermeasure model according to the loss function until the countermeasure model converges.
9. An apparatus, comprising:
at least one processor;
at least one memory for storing at least one program;
when executed by the at least one processor, cause the at least one processor to implement the multi-scale fusion based unsupervised rain shedding method of any one of claims 1-7.
10. A computer storage medium having stored therein a processor-executable program, wherein the processor-executable program, when executed by the processor, is configured to implement the multi-scale fusion based unsupervised rain removal method of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110515593.5A CN113393385B (en) | 2021-05-12 | 2021-05-12 | Multi-scale fusion-based unsupervised rain removing method, system, device and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110515593.5A CN113393385B (en) | 2021-05-12 | 2021-05-12 | Multi-scale fusion-based unsupervised rain removing method, system, device and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113393385A true CN113393385A (en) | 2021-09-14 |
| CN113393385B CN113393385B (en) | 2024-01-02 |
Family
ID=77617891
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110515593.5A Active CN113393385B (en) | 2021-05-12 | 2021-05-12 | Multi-scale fusion-based unsupervised rain removing method, system, device and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113393385B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116681625A (en) * | 2023-07-04 | 2023-09-01 | 广州工程技术职业学院 | Multi-scale contrast learning-based unsupervised image rain removing method, device and terminal |
| CN116883275A (en) * | 2023-07-07 | 2023-10-13 | 广州工程技术职业学院 | Image rain removing method, system, device and medium based on boundary guidance |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110163813A (en) * | 2019-04-16 | 2019-08-23 | 中国科学院深圳先进技术研究院 | A kind of image rain removing method, device, readable storage medium storing program for executing and terminal device |
| AU2020100196A4 (en) * | 2020-02-08 | 2020-03-19 | Juwei Guan | A method of removing rain from single image based on detail supplement |
| AU2020100460A4 (en) * | 2020-03-26 | 2020-04-30 | Huang, Shuying DR | Single image deraining algorithm based on multi-scale dictionary |
| CN111340738A (en) * | 2020-03-24 | 2020-06-26 | 武汉大学 | Image rain removing method based on multi-scale progressive fusion |
| CN111860782A (en) * | 2020-07-15 | 2020-10-30 | 西安交通大学 | Triple multi-scale CycleGAN, fundus fluorography generation method, computer device, and storage medium |
| CN112348762A (en) * | 2020-11-30 | 2021-02-09 | 上海海事大学 | Single image rain removing method for generating confrontation network based on multi-scale fusion |
| CN112508083A (en) * | 2020-12-02 | 2021-03-16 | 南京邮电大学 | Image rain and fog removing method based on unsupervised attention mechanism |
-
2021
- 2021-05-12 CN CN202110515593.5A patent/CN113393385B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110163813A (en) * | 2019-04-16 | 2019-08-23 | 中国科学院深圳先进技术研究院 | A kind of image rain removing method, device, readable storage medium storing program for executing and terminal device |
| AU2020100196A4 (en) * | 2020-02-08 | 2020-03-19 | Juwei Guan | A method of removing rain from single image based on detail supplement |
| CN111340738A (en) * | 2020-03-24 | 2020-06-26 | 武汉大学 | Image rain removing method based on multi-scale progressive fusion |
| AU2020100460A4 (en) * | 2020-03-26 | 2020-04-30 | Huang, Shuying DR | Single image deraining algorithm based on multi-scale dictionary |
| CN111860782A (en) * | 2020-07-15 | 2020-10-30 | 西安交通大学 | Triple multi-scale CycleGAN, fundus fluorography generation method, computer device, and storage medium |
| CN112348762A (en) * | 2020-11-30 | 2021-02-09 | 上海海事大学 | Single image rain removing method for generating confrontation network based on multi-scale fusion |
| CN112508083A (en) * | 2020-12-02 | 2021-03-16 | 南京邮电大学 | Image rain and fog removing method based on unsupervised attention mechanism |
Non-Patent Citations (5)
| Title |
|---|
| HUIRU CAO ET AL: "Intelligent Wide-Area Water Quality Monitoring and Analysis System Exploiting Unmanned Surface Vehicles and Ensemble Learning", 《WATER》, vol. 12, no. 3, pages 1 - 15 * |
| 张峥: "基于多尺度特征融合的单帧图像去雨生成算法研究", 《中国优秀博硕士学位论文全文数据库(硕士)信息科技辑》 * |
| 张峥: "基于多尺度特征融合的单帧图像去雨生成算法研究", 《中国优秀博硕士学位论文全文数据库(硕士)信息科技辑》, no. 6, 15 June 2020 (2020-06-15), pages 138 - 1062 * |
| 王应龙: "恶劣天气条件下的图像复原算法研究", 《中国优秀博硕士学位论文全文数据库(博士)信息科技辑》 * |
| 王应龙: "恶劣天气条件下的图像复原算法研究", 《中国优秀博硕士学位论文全文数据库(博士)信息科技辑》, no. 1, 15 January 2021 (2021-01-15), pages 138 - 241 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116681625A (en) * | 2023-07-04 | 2023-09-01 | 广州工程技术职业学院 | Multi-scale contrast learning-based unsupervised image rain removing method, device and terminal |
| CN116681625B (en) * | 2023-07-04 | 2024-02-02 | 广州工程技术职业学院 | Multi-scale contrast learning-based unsupervised image rain removing method, device and terminal |
| CN116883275A (en) * | 2023-07-07 | 2023-10-13 | 广州工程技术职业学院 | Image rain removing method, system, device and medium based on boundary guidance |
| CN116883275B (en) * | 2023-07-07 | 2023-12-29 | 广州工程技术职业学院 | Image rain removing method, system, device and medium based on boundary guidance |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113393385B (en) | 2024-01-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11798132B2 (en) | Image inpainting method and apparatus, computer device, and storage medium | |
| CN113936339B (en) | Fighting identification method and device based on double-channel cross attention mechanism | |
| CN110210513B (en) | Data classification method and device and terminal equipment | |
| CN108921877B (en) | Long-term target tracking method based on width learning | |
| CN111275686B (en) | Method and device for generating medical image data for artificial neural network training | |
| WO2023082453A1 (en) | Image processing method and device | |
| Long et al. | Bishift networks for thick cloud removal with multitemporal remote sensing images | |
| CN113393385B (en) | Multi-scale fusion-based unsupervised rain removing method, system, device and medium | |
| CN116030396B (en) | Accurate segmentation method for video structured extraction | |
| CN115063786A (en) | High-order distant view fuzzy license plate detection method | |
| CN112802076A (en) | Reflection image generation model and training method of reflection removal model | |
| CN110119736B (en) | License plate position identification method and device and electronic equipment | |
| CN115293992A (en) | Polarization image defogging method and device based on unsupervised weight depth model | |
| CN116363535A (en) | Ship detection method in unmanned aerial vehicle aerial image based on convolutional neural network | |
| CN115409789A (en) | Power transmission line engineering defect detection method based on image semantic segmentation | |
| CN114202473A (en) | Image restoration method and device based on multi-scale features and attention mechanism | |
| CN113177956A (en) | Semantic segmentation method for unmanned aerial vehicle remote sensing image | |
| CN112734675A (en) | Image rain removing method based on pyramid model and non-local enhanced dense block | |
| CN115953312A (en) | Joint defogging detection method and device based on single image and storage medium | |
| CN113962332B (en) | Salient target identification method based on self-optimizing fusion feedback | |
| CN115862119A (en) | Human face age estimation method and device based on attention mechanism | |
| CN112446345B (en) | Low-quality three-dimensional face recognition method, system, equipment and storage medium | |
| CN113505650A (en) | Method, device and equipment for extracting topographic feature line | |
| CN108986135B (en) | Target tracking method and device based on LLC and frequency domain residual error significance | |
| CN112288738B (en) | Single image snowflake removing method and device based on deep learning and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |