CN115205136A - Image rain removing method based on Fourier prior - Google Patents
Image rain removing method based on Fourier prior Download PDFInfo
- Publication number
- CN115205136A CN115205136A CN202210630569.0A CN202210630569A CN115205136A CN 115205136 A CN115205136 A CN 115205136A CN 202210630569 A CN202210630569 A CN 202210630569A CN 115205136 A CN115205136 A CN 115205136A
- Authority
- CN
- China
- Prior art keywords
- fourier
- rain
- domain
- feature map
- feature
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000005070 sampling Methods 0.000 claims description 69
- 238000001228 spectrum Methods 0.000 claims description 43
- 238000010586 diagram Methods 0.000 claims description 24
- 230000006870 function Effects 0.000 claims description 24
- 238000012549 training Methods 0.000 claims description 24
- 238000012545 processing Methods 0.000 claims description 13
- 238000000605 extraction Methods 0.000 claims description 10
- 230000009466 transformation Effects 0.000 claims description 6
- 238000001514 detection method Methods 0.000 claims description 3
- 238000005457 optimization Methods 0.000 claims description 2
- 230000003068 static effect Effects 0.000 claims description 2
- NAWXUBYGYWOOIX-SFHVURJKSA-N (2s)-2-[[4-[2-(2,4-diaminoquinazolin-6-yl)ethyl]benzoyl]amino]-4-methylidenepentanedioic acid Chemical compound C1=CC2=NC(N)=NC(N)=C2C=C1CCC1=CC=C(C(=O)N[C@@H](CC(=C)C(O)=O)C(O)=O)C=C1 NAWXUBYGYWOOIX-SFHVURJKSA-N 0.000 claims 1
- 230000000694 effects Effects 0.000 abstract description 7
- 238000007670 refining Methods 0.000 abstract description 2
- 238000012360 testing method Methods 0.000 description 8
- 238000013528 artificial neural network Methods 0.000 description 6
- 238000013461 design Methods 0.000 description 4
- 230000000007 visual effect Effects 0.000 description 4
- 238000002474 experimental method Methods 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 2
- 238000006731 degradation reaction Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 230000000704 physical effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000013210 evaluation model Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/73—Deblurring; Sharpening
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20048—Transform domain processing
- G06T2207/20056—Discrete and fast Fourier transform, [DFT, FFT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20172—Image enhancement details
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses an image rain removing method based on Fourier prior, which comprises the following steps: 1, acquiring rain image data and corresponding clean image data; 2, constructing a Fourier amplitude domain rain removing module, and performing first-stage rain removing on the rain image; 3, reconstructing by utilizing Fourier phase information of the rain image and Fourier amplitude information of a first stage rain removing result to obtain an input image of a second stage; and 4, constructing a Fourier phase domain structure refining module, and performing rain removal in the second stage. The method can be used for removing rain from the image by utilizing the properties of the Fourier amplitude domain and the phase domain of the rain map, thereby effectively improving the rain removing effect and the generalization capability.
Description
Technical Field
The invention relates to the field of image rain removal, in particular to an image rain removal method based on Fourier prior.
Background
80% of human acquired information comes from the visual system, and visual information has been an externally important component of human perception. Various visual tasks have been widely used, such as pedestrian re-identification, object detection, etc. Among various visual tasks, images are often the most basic input data form, but most algorithms are trained on high-quality images to acquire corresponding parameters, so that the generalization of the model on low-quality images is reduced. For example, when a car accident occurs, if the weather of the hit car escaping from the car is a heavy rain condition, the general car identification model cannot obtain real effective information, and the naked eyes cannot effectively discriminate according to the picture. Therefore, the research on the effective image restoration technology can provide technical support for image processing and computer vision research for practical application, and has important significance for realizing high adaptability of the intelligent vision system in real complex environment.
At the beginning of the development of the image rain removing technology, researchers mainly adopt the priori knowledge of manual design to model the degradation process, and then carry out algorithm design according to the model. The method has strong generalization capability, but cannot recover better on images with larger rain because of the complexity of the rain. Meanwhile, most methods based on prior modeling use iterative calculation, are time-consuming and cannot be used in real time. In recent years, due to the excellent fitting ability of deep neural networks, more and more research works have been carried out to learn the mapping process of a rainy image to a corresponding rainless image from data by artificially synthesizing a large number of paired rainy/rainless image pairs and adopting a supervised learning strategy. The data-driven mode can learn the mapping rule for the spatial characteristics of the rain belt under different conditions through a large amount of data. However, the overfitting problem of deep learning can cause that a model trained on a different data set cannot achieve corresponding rain removing capacity on another data set, and the generalization performance is poor. It is therefore necessary to combine physical a priori knowledge with deep neural networks. The rain belt information has huge domain deviation on different data sets, different rain belts share the same physical characteristics, and if the physical characteristics of different rain belts are analyzed and fused into an image recovery process, the image restoration performance can be improved, and the generalization on different data sets can be improved. The rain belt information has the same physical characteristics, is completely different from background natural images, and has concentrated embodiment on specific frequency, so that effective filtering can be performed in a frequency domain. As in the classical denoising task, processing is initially performed in the frequency domain. It is inspired that the rain map has consistent physical characteristics in the fourier frequency domain. The degradation of rain is mainly concentrated in the fourier amplitude domain, while the phase domain of the rain map is substantially identical to the clean picture. This physical property is widely present on the individual data sets and we call this physical property the fourier prior of the rain map.
Disclosure of Invention
The invention aims to overcome the defects of the existing method and provides an image rain removing method based on Fourier prior, so that the properties of a Fourier amplitude domain and a phase domain of a rain image can be utilized to achieve better rain removing performance on rain images in different scenes, and the rain removing effect is effectively improved.
In order to solve the technical problems, the invention adopts the following technical scheme:
the invention relates to an image rain removing method based on Fourier prior, which is characterized by comprising the following steps:
acquiring a rain image dataset, denoted X rain ＝{x rain_1 ，x rain_2 ，...，x rain_i ，...，x rain_N In which x rαin_i Representing the ith rained image, i =1, 2.., N being the number of rained images;
acquiring a rainless image set corresponding to the rainless image one by one and recording the image set as X clean ＝{x clean_1 ，x clean_2 ，...，x clean_i ，...，x clean_N In which x clean_i Representing the ith rainless image;
let I = { X rain ，X clean Represents the training image data set;
step 2 of constructing FourierA leaf prior rain removal network comprising: the system comprises a Fourier amplitude domain rain removing sub-network, a phase domain reconstruction module and a Fourier phase domain structure refining sub-network; and using said rain image data set X rain As a network input;
step 2.1, the Fourier amplitude domain rain removal subnetwork comprises: the device comprises an amplitude domain down-sampling module and an amplitude domain up-sampling module;
the amplitude domain down-sampling module consists of M Fourier amplitude domain residual error modules and M-1 amplitude domain down-sampling layers;
the amplitude domain up-sampling module consists of M Fourier amplitude domain residual error modules and M-1 amplitude domain up-sampling layers;
respectively recording 2M Fourier amplitude domain residual error modules as:
FARBlock 1 ，FARBlock 2 ，...，FARBlock m ，...，FARBlock 2M (ii) a Wherein FARBlock m Represents the mth order fourier magnitude domain residual module, M =1, 2.., 2M;
respectively recording M-1 amplitude domain down-sampling layers as:
AmpDownSampleBlock 1 ，AmpDownSampleBlock 2 ，...，AmpDownSampleBlock m′ ，...，AmpDownSampleBlock M-1 (ii) a Wherein, ampDOwnSampleBlock m′ Represents an M 'th order amplitude domain down-sampling layer, M' =1, 2., M-1;
respectively recording the M-1 amplitude domain up-sampling modules as:
AmpUpSampleBlock M+1 ，AmpUpSampleBlock M+2 ，...，AmpUpSampleBlock m″ ，...，AmpUpSampleBlock 2M-1 ；AmpUpSampleBlock m″ represents the M 'level amplitude domain upsampled slice, M' = M +1, M +2,. 2M-1;
when m =1, the ith rained image x rain_i Input to the mth stage Fourier amplitude domain residual error module FARBlock m And outputs the mth feature map FAfeature m，i (ii) a Said mth feature map FAfeature m，i Inputting the m' th-level amplitude domain downsampling layer AmpDOwnSampleBlock m′ And outputting the mth characteristic map DownAFeature m′，i (ii) a Wherein m' = m;
(M' -1) th feature map DowfAFeature when M =2 m′-1，i Input to the mth stage Fourier amplitude domain residual error module FARBlock m And correspondingly output the mth feature map FAfeature m，i (ii) a Said mth feature map FAfeature m，i Inputting the m' th-level amplitude domain downsampling layer AmpDOwnSampleBlock m′ And outputting the mth characteristic map DownAFeature m′，i (ii) a Wherein m' = m;
when M = M, the M-1 characteristic diagram DowfFature M-1，i Input to the Mth-stage Fourier amplitude domain residual error module FARBlock M And correspondingly outputs the Mth feature map FAfeature M，i Thereby completing the down sampling process of the amplitude domain;
the Mth feature map FAfeature when M = M +1 M，i Inputting an m-th-level Fourier amplitude domain residual error module FARBlock m And outputs the mth feature map FAfeature m，i (ii) a Said mth feature map FAfeature m，i Inputting an m' th level amplitude domain up-sampling layer Amp UpSampleBlock m″ And output the mth' feature map UpFAfeature m″，i (ii) a Wherein m = m;
the M "-1 th feature map UpFAfeature when M = M +2 m″-1，i And 2M-M +1 th feature map FAfeature 2M-m+1，i Are input into an m-th stage Fourier amplitude domain residual error module FARBlock m And outputs the mth feature map FAfeature m，i (ii) a Said mth feature map FAfeature m，i Input to the m' level amplitude domain upsampling layer AmpUpSampleBlock m″ And correspondingly outputs the mth' feature map UpFAfeature m″，i (ii) a Wherein m = m;
when M =2M, the M "-1 feature map UpFAfeature m″-1，i And said stage 2 Fourier amplitude domain residual error module FARBlock 2 2 nd of outputFeature map FAFeature 2，i Are input into an m-th stage Fourier amplitude domain residual error module FARBlock together m Then processed and the final feature map FAResult is output correspondingly i And as an output of the fourier magnitude domain rain subnetwork, wherein m "= m;
2.2, the phase domain reconstruction module consists of a fast Fourier transform layer and an inverse fast Fourier transform layer;
the fast Fourier transform layer respectively carries out image x on the ith raining image rain_i Output characteristic diagram FAResult of rain removing sub-network in sum Fourier amplitude domain i Performing fast Fourier transform to obtain x rain_i Phase spectrum Amp (x) of rain_i ) And FAResult i Amplitude spectrum Phase (FAResult) of i )；
The fast Fourier transform layer pair Amp (x) rain_i ) And Phase (FAResult) i ) Performing inverse fast Fourier transform and outputting the ith rainy image x rain_i Amplitude domain rain removal result y amp_clean_i ；
Step 2.3, the Fourier phase domain structure refinement sub-network comprises: the phase domain down-sampling module and the phase domain up-sampling module;
the phase domain down-sampling module consists of K Fourier phase domain residual error modules and K-1 phase domain down-sampling layers;
the phase domain up-sampling module consists of K Fourier phase domain residual error modules and K-1 phase domain up-sampling layers;
the 2K fourier phase domain residual error modules are respectively recorded as:
FPRBlock 1 ，FPRBlock 2 ，...，FPRBlock k ，...，FPRBlock 2K (ii) a Wherein, FPRBlock k Represents a kth-order fourier phase domain residual module, K =1,2, ·,2K;
let K-1 phase domain down-sampled layers be respectively noted as:
PhaDownSampleBlock 1 ，PhaDownSampleBlock 2 ，...，PhaDownSampleBlock k′ ，...，PhaDownSampleBlock K-1 (ii) a Wherein, phaDownSampleBlock k′ Represents the K 'th phase domain down-sampling layer, K' =1,2, ·, K-1;
the K-1 phase domain up-sampling modules are respectively recorded as:
PhaUpSampleBlock K+1 ，PhaUpSampleBlock K+2 ，...，PhaUpSampleBlock k″ ，...，PhaUpSampleBlock 2K-1 ；PhaUpSampleBlock k″ represents the K "order phase domain upsampled layer, K" = K +1, + 2., 2K-1;
when k =1, the ith rained image x rain_i Amplitude domain rain removal result y amp_clean_i Input to a k-th stage Fourier phase domain residual error module FPRBlock k And outputs the kth feature map FPfeature k，i (ii) a The kth feature map FPfeature k，i Inputting the k' th-order phase domain down-sampling layer PhaDownSampleBlock k′ And outputs the kth' characteristic map Dowffinery k′，i (ii) a Wherein k' = k;
k' -1 characteristic map DowfFature when K =2 k′-1，i Input to a k-th stage Fourier phase domain residual error module FPRBlock m And correspondingly outputs the kth feature map FPfeature k，i (ii) a The kth feature map FPfeature k，i Inputting the k' th-order phase domain down-sampling layer PhaDownSampleBlock k′ And outputting the kth characteristic map DowfFature k′，i (ii) a Wherein k' = k;
the K-1 characteristic diagram DowfFature when K = K M-1，i Input to a K-th stage Fourier phase domain residual error module FPRBlock K And correspondingly outputs the Kth feature map FPfeature K，i Thereby completing the phase domain down-sampling process;
(iv) when K = K +1, the Kth feature map FPfeature K，i Inputting a k-th-order Fourier phase domain residual error module FPRBlock k And outputs the kth feature map FPfeature k，i (ii) a The kth feature map FPfeature k，i Inputting the kth' stage phase domain up-sampling layer PhaUpSampleBlock k″ And outputs the kth' feature map UpFPfeature k″，i (ii) a Wherein k = k;
k "-1 feature map upsfpfeature when K = K +2 k″-1，i And 2K-m +1 th feature map FPfeature 2K-k+1，i Are input into a k-th stage Fourier phase domain residual error module FPRBlock together k And outputs the kth feature map FPfeature k，i (ii) a The kth feature map FPfeature k，i Input to the k' stage phase domain up-sampling layer PhaUpSampleBlock k″ And correspondingly outputs the kth' feature map UpFPfeature k″，i (ii) a Wherein k = k;
feature UpFPFeature of K "-1 when K =2K m″-1，i And said level 2 Fourier phase domain residual block FPRBlock 2 Output 2 nd feature map FPfeature 2，i Are input into a k-th stage Fourier phase domain residual error module FPRBlock together k Is processed and outputs y accordingly clean_i I.e. the ith predicted no-rain picture, wherein k = k;
step 3, constructing a back propagation loss function:
respectively constructing an amplitude domain rain-removing loss function L on a spatial domain by using the structures of formula (2) and formula (3) 1 And loss function L of amplitude domain rain removal in frequency domain amp ：
In the formula (2), | · | represents a fidelity function, FFT -1 For fast inverse Fourier transform, amp is Fourier magnitude spectrum, and Phase is Fourier Phase spectrum;
respectively constructing an image fidelity loss function L by using an equation (4) and an equation (5) 2 And a Fourier loss function L fft ：
In the formula (5), the FFT is fast Fourier transform;
the total loss function L is constructed using equation (6):
L＝L 1 +λ amp L amp +L 2 +λ fft L fft (6)
in formula (6), λ amp And λ fft Respectively, are adjustable parameters;
step 4, training is carried out based on a training image data set I and a Fourier prior rain removal network, a loss function L is calculated, and meanwhile, a self-adaptive moment estimation optimization method is used for learning rate lr s Updating the static detection network weight, and stopping training when the training iteration number reaches a set number or the loss error is smaller than a set threshold value, so as to obtain an optimal rain removal model; and processing the rain image by using the optimal rain removing network, and obtaining a corresponding clear image.
The image rain removing method based on Fourier prior is also characterized in that any mth Fourier amplitude domain residual error module FARBlock in the Fourier amplitude domain rain removing subnetwork m The system is composed of an m-th main branch and an m-th Fourier magnitude spectrum branch;
the mth main branch consists of N convolution layers and N LeakyRelu layers, the convolution kernels are all k, and the step length is all s; let any nth convolutional layer in convolutional layers with k convolutional kernels and s step length be recorded as Conv k×k，s，m，n (ii) a Let any nth LeakyRelu layer in N LeakyRelu layers be recorded as LR m，n (ii) a N ranges from 1,2,. N;
the mth Fourier amplitude spectrum branch consists of convolution layers with L convolution kernels being k and step length being s, and L convolution layerseakyRelu layer, a fast Fourier transform layer FFT and a fast Fourier inverse transform layer FFT -1 Forming; let any L-th convolutional layer of the convolutional layers with k convolutional kernels and s step length be Conv' k×k，s，m，l (ii) a Let any L-th-level LeakyRelu layer of the L LeakyRelu layers be denoted as LR' m，l L is in the range of 1,2,. L;
suppose that the mth Fourier amplitude domain residual error module FARBlock m The input Feature map is Feature input And the signals are respectively input into the mth main branch and the mth Fourier magnitude spectrum branch for processing, and the processing comprises the following steps:
when n =1, the nth convolutional layer Conv of the mth main branch k×k，s，m，n And the nth LeakyRelu layer LR m，n Feature of the Feature map by equation (7) input Performing Feature extraction to obtain the nth Feature map main，m，n ：
Feature main，m，n ＝LR n (Conv k×k，s，m，n (Feature input )) (7)
When N =2,3.. An, N, the nth convolutional layer Conv 'of the mth main branch' k×k，s，m，l And the nth LeakyRelu layer LR m，n Feature of the n-1 th Feature map using equation (8) main，m，n-1 Performing Feature extraction to obtain the nth Feature map main，m，n ：
Feature main，m，n ＝LR m，n (Conv k×k，s，m，n (Feature main，m，n-1 ))，n＝2，3，...，N (8)
The mth Fourier magnitude spectrum branch pair Feature map Feature input Performing fast Fourier transform to obtain a Fourier magnitude spectrum AmpFature amp，m，0 And Fourier phase spectrum PhaseFeature amp，m，0 As shown in formula (9):
AmpFeature amp，m，0 ，PhaseFeature amp，m，0 ＝FFT(Fea t ure input ) (9)
when L =1,2, 3.. Times.l, the L-th convolution of the m-th fourier magnitude spectrum branchLayer Conv' k×k，s，m，l And n 'th LeakyRelu layer LR' m，l Feature of the Feature map by equation (10) input Performing feature extraction to obtain the nth feature map AmpFature main，m，n ：
AmpFeature amp，m，l ＝LR′ m，l (Conv′ k×k，s，m，l (AmpFeature amp，m，l-1 ))，l＝1，2，...，L. (10)
The FFT pair FFTAmpfeature amp，m，L And PhaseFeature amp，m，0 Carrying out fast Fourier inverse transformation to obtain an amplitude domain transformation characteristic diagram Feature amp，m，out ；
The fast Fourier inverse transform layer FFT-1 obtains an output characteristic diagram Feature of the mth Fourier amplitude domain residual error module by using a formula (11) m，out ：
Feature m，out ＝Feature main，m，N +Feature amp，m，out +Feature input (11)。
Compared with the prior art, the invention has the beneficial effects that:
1. the invention firstly provides a Fourier prior of a rain chart, which can be described as that a Fourier magnitude spectrum of the rain chart contains most rain streak information; the Fourier phase spectrum of the rain picture contains structural information basically consistent with the Fourier phase spectrum of the rain-free picture; the statistical characteristics of the Fourier magnitude spectrum of the rain chart are consistent under different scenes. By utilizing the priori knowledge, the rain removing capability and the generalization capability of the image are improved.
2. The invention designs a deep neural network by utilizing Fourier prior, can realize good end-to-end rain removing effect under the conditions of computation complexity and less operation time, and has better robustness on different data sets compared with the existing rain removing method. Experimental results show that the method provided by the invention is superior to the most advanced method in the Rain13K data set and the Rain DS data set.
3. The invention designs a Fourier amplitude residual error module which can promote the learning of the deep neural network in a Fourier amplitude domain, and the module assists the learning training of the whole deep neural network by changing the amplitude spectrum of the characteristic diagram in a residual error branch under the condition of not influencing the spatial convolution operation on a main branch. By promoting the feature learning in the amplitude domain, the rain removing performance and the generalization capability are promoted.
Drawings
FIG. 1 is a flow chart of the inventive method;
FIG. 2 is a block diagram of the image rain removal method based on Fourier prior of the present invention;
fig. 3 is a structural diagram of a fourier magnitude residual error module according to the present invention.
Detailed Description
In this embodiment, a fourier prior based image rain removing method is to implement a two-stage rain removing depth neural network according to a found fourier prior, and meanwhile, to prevent information loss, a phase spectrum of a rain map is directly used for reconstruction in a second stage to achieve a better rain removing effect, and specifically, referring to fig. 1, the method is performed according to the following steps:
acquiring a rain image dataset, denoted X rain ＝{x rain_1 ，x rain_2 ，...，x rain_i ，...，x rain_N In which x rain_i Representing the ith rained image, i =1, 2.., N being the number of rained images;
acquiring a rain-free image set which corresponds to the rain image one by one and is recorded as X clean ＝{x clean_1 ，x clean_2 ，...，x clean_i ，...，x clean_N In which x clean_i Representing the ith rainless image;
let I = { X rain ，X clean Represents the training image data set;
in this example, a raining model of a Rain13K data set was used, and a test set evaluation model of Rain13K and RainDS was used. Wherein the training set of Rain13K comprises 13, 712 Rain pictures and corresponding Rain-free images; the Rain13K test set consists of Rain image data sets of five different scenes; the RainDS test set contains 100 true high-resolution rain maps and the corresponding rain-free images.
Step 2 referring to fig. 2, a fourier prior rain removal network is constructed, including: the system comprises a Fourier amplitude domain rain removing sub-network, a phase domain reconstruction module and a Fourier phase domain structure thinning sub-network; and with a rain image data set X rain As a network input;
step 2.1, the Fourier amplitude domain rain removal subnetwork comprises: the device comprises an amplitude domain down-sampling module and an amplitude domain up-sampling module;
the amplitude domain down-sampling module consists of M Fourier amplitude domain residual error modules and M-1 amplitude domain down-sampling layers;
the amplitude domain up-sampling module consists of M Fourier amplitude domain residual error modules and M-1 amplitude domain up-sampling layers;
respectively recording 2M Fourier amplitude domain residual error modules as:
FARBlock 1 ，FARBlock 2 ，...，FARBlock m ，...，FARBlock 2M (ii) a Wherein FARBlock m Represents the mth order fourier magnitude domain residual module, M =1, 2.., 2M; in this embodiment, M is 4, and the upsampling and downsampling coefficients are 2 and 0.5, respectively.
Respectively recording M-1 amplitude domain down-sampling layers as:
AmpDownSampleBlock 1 ，AmpDownSampleBlock 2 ，...，AmpDownSampleBlock m′ ，...，AmpDownSampleBlock M-1 (ii) a Wherein, ampDOwnSampleBlock m′ Represents an M 'th order amplitude domain down-sampling layer, M' =1, 2., M-1;
respectively recording the M-1 amplitude domain up-sampling modules as:
AmpUpSampleBlock M+1 ，AmpUpSampleBlock M+2 ，...，AmpUpSampleBlock m″ ，...，AmpUpSampleBlock 2M-1 ；AmpUpSampleBlock m″ represents the M 'level amplitude domain upsampling layer, M' = M +1, M + 2., 2M-1;
when m =1, the ith rained image x rain_i Input to the m-th Fourier amplitudeDomain residual error module FARBlock m And outputs the mth feature map FAfeature m，i (ii) a Feature of mth feature map FAfeature m，i Inputting an m' th-level amplitude domain downsampling layer AmpDOwnSampleBlock m′ And outputs the mth characteristic diagram DownFanture m′，i (ii) a Wherein m' = m;
(M' -1) th feature map DowfAFeature when M =2 m′-1，i Input to the mth stage Fourier amplitude domain residual error module FARBlock m And correspondingly output the mth feature map FAfeature m，i (ii) a Feature of mth m，i Inputting an m' level amplitude domain downsampling layer AmpDOwnSampleBlock m′ And outputs the mth characteristic diagram DownFanture m′，i (ii) a Wherein m' = m;
when M = M, the M-1 characteristic diagram DowfFature M-1，i Input to the M-th stage Fourier amplitude domain residual error module FARBlock M And correspondingly output the Mth feature map FAfeature M，i Thereby completing the amplitude domain down-sampling process;
the Mth feature map FAfeature when M = M +1 M，i Inputting an m-th-level Fourier amplitude domain residual error module FARBlock m And output the mth feature map FAfeature m，i (ii) a Feature of mth m，i Inputting an m' th level amplitude domain up-sampling layer AmpUpSampleBlock m″ And output the mth' feature map UpFAfeature m″，i (ii) a Wherein m "= m;
when M = M + 2.., 2M-1, the M "-1 th feature map UpFAFeture m″-1，i And 2M-M +1 th feature map FAfeature 2M-m+1，i Are input into an m-th stage Fourier amplitude domain residual error module FARBlock together m And output the mth feature map FAfeature m，i (ii) a Feature of mth m，i Input to the m' level amplitude domain upsampling layer AmpUpSampleBlock m″ And correspondingly outputs the mth' feature map UpFAfeature m″，i (ii) a Wherein m = m;
when M =2M, the M "-1 th feature map UpFAfeature m″-1，i And a 2 nd stage Fourier amplitude domain residual error module FARBlock 2 Output 2 nd feature map FAfeature 2，i Are input into an m-th stage Fourier amplitude domain residual error module FARBlock m Then processing is carried out, and the final characteristic diagram FAResult is correspondingly output i And as the output of a Fourier amplitude domain rain removal subnetwork, wherein m "= m;
step 2.2, the phase domain reconstruction module consists of a fast Fourier transform layer and a fast Fourier inverse transform layer;
the fast Fourier transform layer respectively processes the ith rain image x raim_i Output characteristic diagram FAResult of rain removing sub-network in sum Fourier amplitude domain i Performing fast Fourier transform to obtain x rain_i Phase spectrum Amp (x) of rain_i ) And FAResult i Amplitude spectrum Phase (FAResult) of i )；
Fast Fourier inverse transform layer pair Amp (x) rain_i ) And Phase (FAResult) i ) Performing inverse fast Fourier transform and outputting the ith raininess image x rain_i Amplitude domain of (d) rain removal result y amp_clean_i ；
Step 2.3, the Fourier phase domain structure refinement sub-network comprises: the phase domain down-sampling module and the phase domain up-sampling module;
the phase domain down-sampling module consists of K Fourier phase domain residual error modules and K-1 phase domain down-sampling layers;
the phase domain up-sampling module consists of a K Fourier phase domain residual error module and K-1 phase domain up-sampling layers;
the 2K fourier phase domain residual error modules are respectively recorded as:
FPRBlock 1 ，FPRBlock 2 ，...，FPRBlock k ，...，FPRBlock 2K (ii) a Wherein, FPRBlock k Represents the kth order fourier phase domain residual module, K =1, 2., 2K;
in this embodiment, K is 4, and the up-sampling and down-sampling coefficients are 2 and 0.5, respectively.
Let K-1 phase domain down-sampled layers be respectively noted as:
PhaDownSampleBlock 1 ，PhaDownSampleBlock 2 ，...，PhaDownSampleBlock k′ ，...，PhaDownSampleBlock K-1 (ii) a Wherein, phaDownSampleBlock k′ Represents the K 'th phase domain down-sampling layer, K' =1,2, ·, K-1;
the K-1 phase domain upsampling modules are respectively recorded as:
PhaUpSampleBlock K+1 ，PhaUpSampleBlock K+2 ，...，PhaUpSampleBlock k″ ，...，PhaUpSampleBlock 2K-1 ；PhaUpSampleBlock k″ denotes the kth "order phase domain upsampling layer, K" = K +1, + 2., 2K-1;
when k =1, the ith rained image x rain_i Amplitude domain rain removal result y amp_clean_i Input to a k-th stage Fourier phase domain residual error module FPRBlock k And outputs the kth feature map FPfeature k，i (ii) a Feature of the kth feature map FPfeature k，i Inputting a k' th-order phase domain down-sampling layer PhaDownSampleBlock k′ And outputting the kth characteristic map DowfFature k′，i (ii) a Wherein k' = k;
k' -1 characteristic map DowfFature when K =2 k′-1，i Input to a k-th stage Fourier phase domain residual error module FPRBlock m And correspondingly outputs the kth feature map FPfeature k，i (ii) a Feature of the kth feature map FPfeature k，i Inputting a k' th-order phase domain down-sampling layer PhaDownSampleBlock k′ And outputs the kth' characteristic map Dowffinery k′，i (ii) a Wherein k' = k;
the K-1 characteristic diagram DowfFature when K = K M-1，i Input to a K-th stage Fourier phase domain residual error module FPRBlock K And correspondingly outputs the Kth feature map FPfeature K，i Thereby completing the phase domain down-sampling process;
(ii) when K = K +1, the Kth feature map FPfeature K，i Input kth order FourierPhase domain residual error module FPRBlock k And outputs the kth feature map FPfeature k，i (ii) a (k) th feature map FPfeature k，i Input the k' th order phase domain upsampling layer PhaUpSampleBlock k″ And outputs the kth' feature map UpFPfeature k″，i (ii) a Wherein k "= k;
k "-1 feature map upsfpfeature when K = K +2 k″-1，i And 2K-m + 1 feature map FPfeature 2K-k+1，i Are input into a k-th stage Fourier phase domain residual error module FPRBlock together k And outputs the kth feature map FPfeature k，i (ii) a Feature of the kth feature map FPfeature k，i Input to the kth' stage phase domain upsampling layer PhaUpSampleBlock k″ And correspondingly outputs the kth' feature map UpFPfeature k″，i (ii) a Wherein k = k;
feature UpFPFeature of K "-1 when K =2K m″-1，i And a 2 nd stage Fourier phase domain residual block FPRBlock 2 Output 2 nd feature map FPfeature 2，i Are input into a k-th stage Fourier phase domain residual error module FPRBlock together k Is processed and outputs y accordingly clean_i I.e. the ith predicted no-rain picture, where k = k;
step 3, constructing a back propagation loss function:
respectively constructing an amplitude domain rain-removing loss function L on a spatial domain by using the structures of formula (2) and formula (3) 1 Loss function L of amplitude domain rain removal in sum frequency domain amp ：
In the formula (2), | · | represents a fidelity function, FFT -1 For inverse fast Fourier transform, amp is the Fourier amplitudeDegree spectrum, phase is fourier Phase spectrum; in this embodiment, the fidelity term minimizes the loss of the predicted result and the true result in a norm space to prevent the problem of over-smoothing of the two norms.
Respectively constructing an image fidelity loss function L by using an equation (4) and an equation (5) 2 And Fourier loss function L fft ：
In the formula (5), the FFT is fast Fourier transform;
the total loss function L is constructed using equation (6):
L＝L 1 +λ amp L amp +L 2 +λ fft L fft (6)
in formula (6), λ amp And λ fft Respectively, are adjustable parameters; in this example λ amp And λ fft Are all 0.2.
Step 4, training is carried out based on the training image data set I and the Fourier prior rain removing network, and a loss function L is calculated, wherein the learning rate lr in the example is s Taking 2e-4, and stopping training when the training iteration number reaches a set number or the loss error is smaller than a set threshold value, so as to obtain an optimal rain removal model; and processing the rain image by using the optimal rain removing network, and obtaining a corresponding clear image.
In a specific embodiment, any mth Fourier amplitude domain residual error module FARBlock in the Fourier amplitude domain rain removing subnetwork m Is composed of the mth main branch and the mth fourier magnitude spectrum branch, see fig. 3;
the mth main branch consists of N convolution layers and N LeakyRelu layers, the convolution kernels are all k, and the step length is all s; let any nth convolutional layer in convolutional layers with k convolutional kernels and s step length be recorded as Conv k×k，s，m，n (ii) a Let any nth LeakyRelu layer in N LeakyRelu layers be recorded as LR m，n (ii) a N ranges from 1,2,. Cndot.n;
the mth Fourier amplitude spectrum branch comprises a convolution layer, L LeakyRelu layers, a fast Fourier transform layer FFT and a fast Fourier inverse transform layer FFT, wherein L convolution kernels are all k, and the step length is all s -1 Composition is carried out; let any L-level convolution layer in the convolution layers with all convolution kernels k and all step lengths s be recorded as Conv' k×k，s，m，l (ii) a Let any L-th LeakyRelu layer of the L LeakyRelu layers be recorded as LR' m，l L is in the range of 1,2,. L;
suppose that the mth Fourier amplitude domain residual error module FARBlock m The input Feature map is Feature input And respectively inputting the m-th main branch and the m-th Fourier magnitude spectrum branch for processing, wherein the processing comprises the following steps:
when n =1, the nth convolutional layer Conv of the mth main branch k×k，s，m，n And the nth LeakyRelu layer LR m，n Feature map Feature using equation (7) input Performing Feature extraction to obtain the nth Feature map Feature main，m，n ：
Feature main，m，n ＝LR n (Conv k×k，s，m，n (Feature input )) (7)
When N =2,3.. N, the nth convolutional layer Conv 'of the mth main branch' k×k，s，m，l And the nth LeakyRelu layer LR m，n Feature of the n-1 th Feature map using equation (8) main，m，n-1 Performing Feature extraction to obtain the nth Feature map main，m，n ：
Feature main，m，n ＝LR m，n (Conv k×k，s，m，n (Feature main，m，n-1 ))，n＝2，3，...，N (8)
Feature of the mth Fourier magnitude spectrum branch pair input Performing fast Fourier transform to obtain a Fourier magnitude spectrum AmpFature amp，m，0 And Fourier phase spectrum PhaseFeature amp，m，0 As shown in formula (9):
AmpFeature amp，m，0 ，PhaseFeature amp，m，0 ＝FFT(Feature input ) (9)
when L =1,2,3.. An, L, the L convolutional layer Conv 'of the m-th Fourier magnitude spectrum branch' k×k，s，m，l And n 'th LeakyRelu layer LR' m，l Feature map Feature using equation (10) input Performing feature extraction to obtain the nth feature map AmpFature main，m，n ：
AmpFeature amp，m，l ＝LR′ m，l (Conv′ k×k，s，m，l (AmpFeature amp，m，l-1 ))，l＝1，2，...，L. (10)
Fast Fourier transform layer FFT to FFTAmpfeature amp，m，L And PhaseFeature amp，m，0 Carrying out fast Fourier inverse transformation to obtain an amplitude domain transformation characteristic diagram Feature amp，m，out (ii) a In this embodiment, N and L are 2, the convolution kernel size is 3 × 3, and the step length is 1; the parameter of leakyRelu is 0.2.
The fast Fourier inverse transform layer FFT-1 obtains an output characteristic diagram Feature of the mth Fourier amplitude domain residual error module by using a formula (11) m，out ：
Feature m，out ＝Feature main，m，N +Feature amp，m，out +Feature input (11)
In this embodiment, any kth fourier phase domain residual error module FPRBlock in the fourier phase domain structure refinement sub-network k The system is composed of a kth main branch and a kth Fourier phase spectrum branch; the specific details are similar to those of a Fourier amplitude domain residual error module, and the only difference is that the kth Fourier phase spectrum branch is used for carrying out feature extraction on a phase spectrum.
Examples
In order to verify the effectiveness of the method of the present invention, a training set of a raining 13K dataset is selected for training, and a test set of the raining 13K dataset and a test set of the RainDS dataset are used for testing.
The method is based on a Rain13K data set for training, wherein the Rain13K training set comprises 13, 712 Rain pictures and corresponding Rain-free images; the Rain13K Test set consists of five synthetic Rain image data sets of different scenes, namely Test100, rain100L, rain100H, test1200 and Test2800; the RainDS test set contains 100 true high-resolution rain maps and the corresponding rain-free images. The two data sets contain various rain strip distributions, such as rain strip size, rain strip shape, and so forth. The wide applicability of the method of the invention can be objectively evaluated.
The invention adopts the structure similarity (PSNR) and the peak signal-to-noise ratio (SSIM) as evaluation indexes.
In the embodiment, five methods and the method disclosed by the invention are selected for effect comparison on Rain13K, and three methods and the method disclosed by the invention are selected for effect comparison on Rain DS. The selected methods are RESCAN, PReNet, MSPFN, MPRNet, and HINet, respectively. FPNet is the invented method.
The results obtained from the experimental results are shown in tables 1 and 2:
TABLE 1 results of Rain removal experiments on Rain13K data set using the method of the present invention and nine selected comparison methods
| Test100 | Rain100L | Rain100H | Test2800 | Test1200 | |
| PSNR|SSIM | PSNR|SSIM | PSNR|SSIM | PSNR|SSIM | PSNR|SSIM | |
| RESCAN | 25.00|0.835 | 29.80|0.881 | 26.36|0.786 | 31.29|0.904 | 30.51|0.882 |
| PReNet | 24.81|0.851 | 32.44|0.950 | 26.77|0.858 | 31.75|0.916 | 31.36|0.911 |
| MSPFN | 27.50|0.876 | 32.40|0.933 | 28.66|0.860 | 32.82|0.930 | 32.39|0.916 |
| MPRNet | 30.27|0.897 | 36.40|0.965 | 30.41|0.890 | 33.64|0.938 | 32.91|0.916 |
| HINet | 30.29|0.906 | 37.28|0.970 | 30.65|0.894 | 33.91|0.941 | 33.05|0.919 |
| FPNet | 30.86|0.915 | 37.96|0.972 | 30.89|0.897 | 34.07|0.943 | 33.08|0.924 |
TABLE 2 results of rain removal experiments on Rainds datasets using the method of the invention and three selected comparison methods
| MSPFN | MPRNet | HINet | FPNet | |
| PSNR | 24.76 | 25.07 | 24.71 | 25.34 |
| SSIM | 0.691 | 0.700 | 0.693 | 0.707 |
The experimental results show that the method of the invention has better effect compared with other most advanced methods on two different data sets, thereby proving the feasibility of the method provided by the invention. Experiments show that the method provided by the invention can effectively utilize the rain belt information of the rain image to be concentrated in the Fourier amplitude domain; and the physical characteristics of the Fourier phase domain structure information of the rain image and the rain-free image are basically consistent, and the image rain removing task is completed.
Claims (2)
1. An image rain removing method based on Fourier prior is characterized by comprising the following steps:
step 1 acquisition of a rained and rainless image pair dataset for training:
acquiring a rain image data set marked as X rain ＝{x rain_1 ，x rain_2 ，...，x rain_i ，...，x rain_N In which x rain_i Representing the ith rained image, i =1, 2.., N being the number of rained images;
acquiring a rain-free image set which corresponds to the rain image one by one and is recorded as X clean ＝{x clean_1 ，x clean_2 ，...，x clean_i ，...，x clean_N In which x clean_i Representing the ith rainless image;
let I = { x = rain ，X clean Represents the training image data set;
step 2, constructing a Fourier prior rain removal network, comprising the following steps: fourier amplitude domain rain removing sub-network, phase domain reconstruction module and Fourier phase domain structure refinerA network; and using said rain image data set X rain As a network input;
step 2.1, the fourier amplitude domain rain removal subnetwork comprises: the device comprises an amplitude domain down-sampling module and an amplitude domain up-sampling module;
the amplitude domain down-sampling module consists of M Fourier amplitude domain residual error modules and M-1 amplitude domain down-sampling layers;
the amplitude domain up-sampling module consists of M Fourier amplitude domain residual error modules and M-1 amplitude domain up-sampling layers;
respectively recording 2M Fourier amplitude domain residual error modules as:
FARBlock 1 ，FARBlock 2 ，...，FARBlock m ，...，FARBlock 2M (ii) a Wherein FARBlock m Represents the mth order fourier magnitude domain residual module, M =1, 2.., 2M;
respectively recording M-1 amplitude domain down-sampling layers as:
AmpDownSampleBlock 1 ，AmpDownSampleBlock 2 ，...，AmpDownSampleBlock m′ ，...，AmpDownSampleBlock M-1 (ii) a Wherein, ampDOwnSampleBlock m′ Represents the M 'th level amplitude domain downsampled layer, M' =1,2, ·, M-1;
respectively recording the M-1 amplitude domain up-sampling modules as:
AmpUpSampleBlock M+1 ，AmpUpSampleBlock M+2 ，...，AmpUpSampleBlock m″ ，...，AmpUpSampleBlock 2M-1 ；AmpUpSampleBlock m″ represents the M 'level amplitude domain upsampling layer, M' = M +1, M + 2., 2M-1;
when m =1, the ith rain image x rain_i Input to the mth stage Fourier amplitude domain residual error module FARBlock m And output the mth feature map FAfeature m，i (ii) a Said mth feature map FAfeature m，i Inputting the m' th level amplitude domain downsampling layer AmpDOwnSampleBlock m′ And outputs the mth characteristic diagram DownFanture m′，i (ii) a Wherein m' = m;
when M =2, 1, M-1, the M' -1 characteristic map is DowfAFeature m′-1，i Input to the mth stage Fourier amplitude domain residual error module FARBlock m And correspondingly outputs the mth feature map FAfeature m，i (ii) a Said mth feature map FAfeature m，i Inputting the m' th level amplitude domain downsampling layer AmpDOwnSampleBlock m′ And outputting the mth characteristic map DownAFeature m′，i (ii) a Wherein m' = m;
m-1 characteristic diagram DowfAFeature when M = M M-1，i Input to the M-th stage Fourier amplitude domain residual error module FARBlock M And correspondingly output the Mth feature map FAfeature M，i Thereby completing the down sampling process of the amplitude domain;
feature of Mth feature map FAfeature when M = M +1 M，i Inputting a m-th-level Fourier amplitude domain residual error module FARBlock m And outputs the mth feature map FAfeature m，i (ii) a Said mth feature map FAfeature m，i Inputting an m' th level amplitude domain up-sampling layer Amp UpSampleBlock m″ And output the mth' feature map UpFAfeature m″，i (ii) a Wherein m = m;
when M = M + 2.., 2M-1, the M "-1 th feature map UpFAFeture m″-1，i And 2M-M +1 th feature map FAfeature 2M-m+1，i Are input into an m-th stage Fourier amplitude domain residual error module FARBlock together m And outputs the mth feature map FAfeature m，i (ii) a Said mth feature map FAfeature m，i Input to the m' th stage amplitude domain up-sampling layer AmpUpSampleBlock m″ And correspondingly outputs the mth' feature map UpFAfeature m″，i (ii) a Wherein m "= m;
when M =2M, the M "-1 feature map UpFAfeature m″-1，i And said stage 2 Fourier amplitude domain residual error module FARBlock 2 Output 2 nd feature map FAfeature 2，i Are input into an m-th stage Fourier amplitude domain residual error module FARBlock m After processing, the final characteristic diagram is output correspondinglyFAResult i And as an output of the fourier magnitude domain rain subnetwork, wherein m "= m;
2.2, the phase domain reconstruction module consists of a fast Fourier transform layer and a fast Fourier inverse transform layer;
the fast Fourier transform layer respectively carries out image x on the ith raining image rain_i Output characteristic diagram FAResult of rain removing sub-network in sum Fourier amplitude domain i Performing fast Fourier transform to obtain x rain_i Phase spectrum Amp (x) of rain_i ) And FAResult i Amplitude spectrum Phase (FAResult) of (1) i )；
The pair of fast Fourier transform layers Amp (x) rain_i ) And Phase (FAResult) i ) Performing inverse fast Fourier transform and outputting the ith raininess image x rain_i Amplitude domain rain removal result y amp_clean_i ；
Step 2.3, the Fourier phase domain structure refinement sub-network comprises: the phase domain down-sampling module and the phase domain up-sampling module;
the phase domain down-sampling module consists of K Fourier phase domain residual error modules and K-1 phase domain down-sampling layers;
the phase domain up-sampling module consists of K Fourier phase domain residual error modules and K-1 phase domain up-sampling layers;
the 2K fourier phase domain residual error modules are respectively recorded as:
FPRBlock 1 ，FPRBlock 2 ，...，FPRBlock k ，...，FPRBlock 2K (ii) a Wherein, FPRBlock k Represents the kth order fourier phase domain residual module, K =1, 2., 2K;
let K-1 phase domain down-sampled layers be respectively noted as:
PhaDownSampleBlock 1 ，PhaDownSampleBlock 2 ，....，PhaDownSampleBlock k′ ，...，PhaDownSampleBlock K-1 (ii) a Wherein, phaDownSampleBlock k′ Represents the K 'th phase domain down-sampling layer, K' =1, 2., K-1;
the K-1 phase domain up-sampling modules are respectively recorded as:
PhaUpSampleBlock K+1 ，PhaUpSampleBlock K+2 ，....，PhaUpSampleBlock k″ ，...，PhaUpSampleBlock 2K-1 ；PhaUpSampleBlock k″ denotes the kth "order phase domain upsampling layer, K" = K +1, + 2., 2K-1;
when k =1, the ith rained image x rain_i Amplitude domain rain removal result y amp_clean_i Input to a k-th stage Fourier phase domain residual error module FPRBlock k And outputs the kth feature map FPfeature k，i (ii) a The kth feature map FPfeature k，i Inputting the k' th-order phase domain down-sampling layer PhaDownSampleBlock k′ And outputs the kth' characteristic map Dowffinery k′，i (ii) a Wherein k' = k;
k' -1 characteristic diagram DowfFature when K =2 k′-1， i is input into a k-th stage Fourier phase domain residual error module FPRBlock m And correspondingly outputs the kth feature map FPfeature k，i (ii) a The kth feature map FPfeature k，i Inputting the k' th phase domain down-sampling layer PhaDownSampleBlock k′ And outputs the kth' characteristic map Dowffinery k′，i (ii) a Wherein k' = k;
the K-1 characteristic diagram DowfFature when K = K M-1，i Input to a K-th stage Fourier phase domain residual error module FPRBlock K And correspondingly outputs the Kth feature map FPfeature K，i Thereby completing the phase domain down-sampling process;
(ii) when K = K +1, the Kth feature map FPfeature K，i Inputting a k-th-order Fourier phase domain residual error module FPRBlock k And outputs the kth feature map FPfeature k，i (ii) a The kth feature map FPfeature k，i Input the k' th order phase domain upsampling layer PhaUpSampleBlock k″ And outputting the kth' feature map UpFPfeature k″，i (ii) a Wherein k "= k;
when K = K +2, 2K-1, the K-1 feature map UpFPfeature k″-1，i And 2K-m +1 th feature map FPfeature 2K-k+1，i Are input into a k-th stage Fourier phase domain residual error module FPRBlock together k And outputs the kth feature map FPfeature k，i (ii) a The kth feature map FPfeature k，i Input to the kth' stage phase domain upsampling layer PhaUpSampleBlock k″ And correspondingly outputs the kth' feature map UpFPfeature k″，i (ii) a Wherein k "= k;
k' -1 feature map UpFPfeature when K =2K m″-1，i And the 2 nd stage Fourier phase domain residual error module FPRBlock 2 Output 2 nd feature map FPfeature 2，i Are input into a k-th stage Fourier phase domain residual error module FPRBlock together k Is processed and outputs y accordingly clean_i I.e. the ith predicted no-rain picture, wherein k = k;
step 3, constructing a loss function of back propagation:
respectively constructing an amplitude domain rain-removing loss function L on a spatial domain by using the structures of formula (2) and formula (3) 1 Loss function L of amplitude domain rain removal in sum frequency domain amp ：
In the formula (2), | · | represents a fidelity function, FFT -1 For fast inverse Fourier transform, amp is Fourier magnitude spectrum, and Phase is Fourier Phase spectrum;
respectively constructing an image fidelity loss function L by using an equation (4) and an equation (5) 2 And Fourier loss function L fft ：
In the formula (5), the FFT is fast Fourier transform;
the total loss function L is constructed using equation (6):
L＝L 1 +λ amp L amp +L 2 +λ fft L fft (6)
in formula (6), λ amp And λ fft Respectively, are adjustable parameters;
step 4, training is carried out based on a training image data set I and a Fourier prior rain removal network, a loss function L is calculated, and meanwhile, a self-adaptive moment estimation optimization method is used for learning rate lr s Updating the static detection network weight, and stopping training when the training iteration number reaches a set number or the loss error is smaller than a set threshold value, so as to obtain an optimal rain removal model; and processing the rain image by using the optimal rain removing network, and obtaining a corresponding clear image.
2. The image rain removing method based on Fourier transform prior as claimed in claim 1, wherein any mth Fourier amplitude domain residual module FARBlock in the Fourier amplitude domain rain removing sub-network m The Fourier amplitude spectrum is composed of an m-th main branch and an m-th Fourier amplitude spectrum branch;
the mth main branch consists of N convolution layers and N LeakyRelu layers, the convolution kernels are all k, and the step length is all s; let any nth convolutional layer in convolutional layers with k convolutional kernels and s step length be recorded as Conv k×k，s，m，n (ii) a Let any nth LeakyRelu layer in N LeakyRelu layers be recorded as LR m，n (ii) a N ranges from 1,2,. Cndot.n;
the mth Fourier magnitude spectrum branch consists of convolution layers with L convolution kernels being k and step length being s, L LeakyRelu layers, a fast Fourier transform layer FFT and a fast Fourier inverse transform layer FFT -1 Composition is carried out; let any L-th convolutional layer of the convolutional layers with k convolutional kernels and s step length be Conv' k×k，s，m，l (ii) a Let any L-th LeakyRelu layer of the L LeakyRelu layers be recorded as LR' m，l L is in the range of 1,2,. L;
suppose that the mth Fourier amplitude domain residual error module FARBlock m The input Feature map is Feature input And respectively inputting the m-th main branch and the m-th Fourier magnitude spectrum branch for processing, wherein the processing comprises the following steps:
when n =1, the nth convolutional layer Conv of the mth main branch k×k，s，m，n And the nth LeakyRelu layer LR m，n Feature of the Feature map by equation (7) input Performing Feature extraction to obtain the nth Feature map main，m，n ：
Feature main，m，n ＝LR n (Conv k×k，s，m，n (Feature input )) (7)
When N =2,3, \8230;, N, the nth convolution layer Conv 'of the mth main branch' k×k，s，m，l And the nth LeakyRelu layer LR m，n Feature of the n-1 th Feature map by using the equation (8) main，m，n-1 Performing Feature extraction to obtain the nth Feature map Feature main，m，n ：
Feature main，m，n ＝LR m，n (Conv k×k，s，m，n (Feature main，m，n-1 ))，n＝2，3，...，N (8)
The mth Fourier magnitude spectrum branch pair the Feature map Feature input Performing fast Fourier transform to obtain a Fourier magnitude spectrum AmpFature amp，m，0 And Fourier phase spectrum PhaseFeature amp，m，0 As shown in formula (9):
AmpFeature amp，m，0 ，PhaseFeature amp，m，0 ＝FFT(Feature input ) (9)
l convolution layer Conv 'of the mth Fourier magnitude spectrum branch when L =1,2,3,. Ann.L' k×k，s，m，l And n 'th LeakyRelu layer LR' m，l Using formula (10) for said featureDrawing Feature input Performing feature extraction to obtain the nth feature map AmpFiture main，m，n ：
AmpFeature amp，m，l ＝LR′ m，l (Conv′ k×k，s，m，l (AmpFeature amp，m，l-1 ))，l＝1，2，...，L. (10)
The FFT pair FFTAmpfeature amp，m，L And PhaseFeature amp，m，0 Carrying out fast Fourier inverse transformation to obtain an amplitude domain transformation characteristic diagram Feature amp，m，out ；
The FFT of the FFT layer -1 Obtaining an output characteristic diagram Feature of the mth Fourier amplitude domain residual error module by using the formula (11) m，out ：
Feature m，out ＝Feature main，m，N +Feature amp，m，out +Feature input (11)。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210630569.0A CN115205136A (en) | 2022-06-06 | 2022-06-06 | Image rain removing method based on Fourier prior |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210630569.0A CN115205136A (en) | 2022-06-06 | 2022-06-06 | Image rain removing method based on Fourier prior |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115205136A true CN115205136A (en) | 2022-10-18 |
Family
ID=83576520
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210630569.0A Pending CN115205136A (en) | 2022-06-06 | 2022-06-06 | Image rain removing method based on Fourier prior |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115205136A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116912489A (en) * | 2023-06-26 | 2023-10-20 | 天津师范大学 | Medical image segmentation method and system based on Fourier priori knowledge |
| CN118014892A (en) * | 2024-04-08 | 2024-05-10 | 北京航空航天大学 | Single image rain removal model construction method and rain removal method based on frequency domain comparison regularization |
| CN116912489B (en) * | 2023-06-26 | 2024-06-21 | 天津师范大学 | Medical image segmentation method and system based on Fourier priori knowledge |
-
2022
- 2022-06-06 CN CN202210630569.0A patent/CN115205136A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116912489A (en) * | 2023-06-26 | 2023-10-20 | 天津师范大学 | Medical image segmentation method and system based on Fourier priori knowledge |
| CN116912489B (en) * | 2023-06-26 | 2024-06-21 | 天津师范大学 | Medical image segmentation method and system based on Fourier priori knowledge |
| CN118014892A (en) * | 2024-04-08 | 2024-05-10 | 北京航空航天大学 | Single image rain removal model construction method and rain removal method based on frequency domain comparison regularization |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110599409B (en) | Convolutional neural network image denoising method based on multi-scale convolutional groups and parallel | |
| CN109389556B (en) | Multi-scale cavity convolutional neural network super-resolution reconstruction method and device | |
| CN107633486B (en) | Structural magnetic resonance image denoising method based on three-dimensional full-convolution neural network | |
| CN106952228B (en) | Super-resolution reconstruction method of single image based on image non-local self-similarity | |
| CN110276726B (en) | Image deblurring method based on multichannel network prior information guidance | |
| CN110766632A (en) | Image denoising method based on channel attention mechanism and characteristic pyramid | |
| CN110443761B (en) | Single image rain removing method based on multi-scale aggregation characteristics | |
| CN108765330B (en) | Image denoising method and device based on global and local prior joint constraint | |
| CN113269818B (en) | Deep learning-based seismic data texture feature reconstruction method | |
| CN111738954B (en) | Single-frame turbulence degradation image distortion removal method based on double-layer cavity U-Net model | |
| CN113362338B (en) | Rail segmentation method, device, computer equipment and rail segmentation processing system | |
| CN114723630A (en) | Image deblurring method and system based on cavity double-residual multi-scale depth network | |
| CN115953303B (en) | Multi-scale image compressed sensing reconstruction method and system combining channel attention | |
| CN111652804A (en) | Super-resolution reconstruction method based on expansion convolution pyramid and bottleneck network | |
| CN113538246A (en) | Remote sensing image super-resolution reconstruction method based on unsupervised multi-stage fusion network | |
| He et al. | Remote sensing image super-resolution using deep–shallow cascaded convolutional neural networks | |
| CN114240797A (en) | OCT image denoising method, device, equipment and medium | |
| CN107451961B (en) | Method for recovering sharp image under multiple fuzzy noise images | |
| CN112991199A (en) | Image high-low frequency decomposition noise removing method based on residual error dense network | |
| CN114202459A (en) | Blind image super-resolution method based on depth prior | |
| CN110503608B (en) | Image denoising method based on multi-view convolutional neural network | |
| CN115526779A (en) | Infrared image super-resolution reconstruction method based on dynamic attention mechanism | |
| CN115293966A (en) | Face image reconstruction method and device and storage medium | |
| CN113327205B (en) | Phase denoising method based on convolutional neural network | |
| CN113962905A (en) | Single image rain removing method based on multi-stage feature complementary network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |