CN112132756A - Attention mechanism-based single raindrop image enhancement method - Google Patents
Attention mechanism-based single raindrop image enhancement method Download PDFInfo
- Publication number
- CN112132756A CN112132756A CN201910587306.4A CN201910587306A CN112132756A CN 112132756 A CN112132756 A CN 112132756A CN 201910587306 A CN201910587306 A CN 201910587306A CN 112132756 A CN112132756 A CN 112132756A
- Authority
- CN
- China
- Prior art keywords
- image
- convolution
- raindrop
- attention
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000007246 mechanism Effects 0.000 title claims abstract description 31
- 238000000034 method Methods 0.000 title claims abstract description 29
- 238000012549 training Methods 0.000 claims description 13
- 230000010339 dilation Effects 0.000 claims description 8
- 230000008569 process Effects 0.000 claims description 8
- 238000013528 artificial neural network Methods 0.000 claims description 7
- 230000006870 function Effects 0.000 claims description 7
- 238000010586 diagram Methods 0.000 claims description 6
- 238000012545 processing Methods 0.000 claims description 6
- 238000010606 normalization Methods 0.000 claims description 4
- 230000008447 perception Effects 0.000 claims description 3
- 230000004927 fusion Effects 0.000 claims description 2
- 238000012544 monitoring process Methods 0.000 claims description 2
- 238000005096 rolling process Methods 0.000 claims description 2
- 238000011084 recovery Methods 0.000 claims 1
- 238000011282 treatment Methods 0.000 claims 1
- 238000000605 extraction Methods 0.000 abstract description 2
- 238000007781 pre-processing Methods 0.000 abstract 1
- 238000012360 testing method Methods 0.000 description 7
- 230000000007 visual effect Effects 0.000 description 6
- 230000000694 effects Effects 0.000 description 4
- 238000002474 experimental method Methods 0.000 description 3
- 241000282414 Homo sapiens Species 0.000 description 2
- 230000002708 enhancing effect Effects 0.000 description 2
- 238000013441 quality evaluation Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 210000004556 brain Anatomy 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 230000004438 eyesight Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000016776 visual perception Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/73—Deblurring; Sharpening
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computational Linguistics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Evolutionary Biology (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses and provides a single raindrop image enhancement method based on an attention mechanism, which comprises the following steps: preprocessing a data set; performing feature extraction by adopting multi-scale expansion convolution with different expansion factors; an attention mechanism model was introduced to focus on the details of the raindrop regions in the image. The method provided by the invention can well remove the raindrop part in the raindrop image, so that the raindrop blurred image has better quality.
Description
Technical Field
The invention relates to the field of image enhancement, in particular to a single raindrop image enhancement method.
Background
The image acquired in a rainy environment is often doped with raindrops or rainstripes to cause image blurring, more noise is easily introduced or larger raindrops cannot be removed by the conventional raindrop removing image enhancement algorithm, and in order to better remove raindrops on the image and restore more details of the image, the inventor provides a single raindrop image enhancement method based on attention-driven system.
The expanded convolution (also called dilation convolution or hole convolution) is different from other convolution methods in that it introduces a new parameter, namely a dilation rate parameter, which is used to indicate the size of dilation.
The structure of the dilation convolution is shown in fig. 1. Such as Yu et alLet us mention that F0，F1，....，Fn-1：Z1→ R is a discrete function, let K0，K1，....，Kn-2：Z2→ R is a discrete 3 x 3 filter, considering filtering with exponential increasing dilation:
Fi+1＝Fi×2′ki，i＝0，1，2，....，n-2 (1)
f is to bei+1The pixel receptive field area of the middle element p is defined as being in F0Modifying on the basis of Fi+1Elemental values of (p) value, assume Fi+1The receptive field size of the pixel p in (a) is the number of these elements, and the receptive field size under the dilation convolution in the pixel can be expressed as:
Fi+1＝(2i+2-1)×(2i+2-1) (2)
the expansion convolution can weight the pixels and aggregate information, so that the perception visual field of the pixels can be enhanced under the condition of not losing the resolution, more characteristic information can be obtained, and the size of the output characteristic mapping can be ensured not to change.
The visual attention mechanism is a specific visual signal processing mechanism in the human brain, the essence of the attention mechanism is similar to the visual attention of human beings, and the main significance is to screen a large amount of information and select key high-value information relevant to the current task. The attention mechanism thought was originally applied to computer image vision research, and more researchers in recent years use the attention mechanism thought to combine with a neural network to perform related problem research.
Therefore, by means of the attention mechanism idea, a single raindrop image enhancement method based on the attention mechanism is provided, so that raindrop fuzzy image enhancement can be effectively improved, image details are richer, and a better image visual effect can be achieved.
Disclosure of Invention
In order to solve the above problems, the present invention provides a method for enhancing blurred raindrops and making details of an image richer.
The invention adopts the following technical scheme: firstly, extracting characteristic values of an input raindrop-containing picture by adopting multi-scale expansion convolution with different expansion factors; secondly, an attention mechanism is introduced into the model to pay attention to the details of the raindrop region in the image so as to help better remove raindrops and restore the details of the image: finally, raindrop-like blurred image enhancement is achieved end-to-end.
The method comprises the following steps:
(1) and carrying out uniform resolution processing on the data set pictures to divide a training set and a testing set.
(2) And establishing a raindrop fuzzy image enhancement model based on an attention mechanism, and initializing parameters of the network.
(3) And training the model by using the divided training set, and continuously updating the model parameters by performing back propagation through a loss function so as to minimize the calculated loss and optimize the performance effect of the model.
(4) Training every 50 pairs of pictures, setting the maximum iteration number to be 3000, storing model parameters, and testing the model by using a test set.
Preferably, the specific implementation includes: the attention mechanism model helps to extract features of the raindrop image, and the attention mechanism can enable a network to pay attention to a raindrop area, so that raindrop image enhancement can be better achieved. The attention mechanism model comprises three layers of residual error networks (ResBlock), a long and short memory neural network (LSTM) and a common convolution layer (conv) with a convolution kernel size of 3 x 3 and a step size of 1.
The preferred loss function consists of two parts, attention model loss LattAnd a perceptual loss Lp。
Lp＝LMSE(VGG(0)，VGG(T)) (4)
Pre-training the network by using VGG16, extracting high-level features and monitoring to ensure the quality of generated images, wherein AtIs attention toAnd (3) drawing an attention machine generated by the mechanism model at the time T, wherein M is a binary mask, N is 4, theta is 0.8, O is a raindrop removing image output after model processing, and T is an original clear raindrop-free image corresponding to the raindrop image. An adaptive moment estimation (ADAM) algorithm is selected to optimize the loss function. The model continuously updates the weight of the neural network through a training data set, and after multiple times of training and tuning, the learning rate alpha is determined to be 0.0001, beta 1 is 0, beta 2 is 0.9, and in the training process, the value of the attention map is initialized to be 0.5.
The method has the advantages that by means of the technical scheme, the perception visual field can be effectively enhanced during feature extraction, more feature information is obtained, and raindrop areas in the image are paid more attention.
Drawings
FIG. 1 is a diagram of a dilated convolution;
FIG. 2 is a model for enhancing blurred images in rainy days based on attention mechanism
FIG. 3 attention mechanism model
Residual error network adopted by FIG. 4
FIG. 5 comparative graph of experimental results
Wherein:
in fig. 5: (a) raindrop pattern (b) clear pattern (c) model (d) tanaka (e) Eigen proposed by the invention
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the accompanying drawings. It should be understood that the detailed description and specific examples, while indicating the invention, are intended for purposes of illustration only and are not intended to limit the scope of the invention.
As shown in fig. 1, the overall structure of the raininess blurred image enhancement model based on the attention mechanism proposed by the present invention, wherein conv, dia _ c and convtranspose represent a convolutional layer, an extended convolutional layer and a deconvolution layer, respectively, and relu is an activation function. The invention provides an attention mechanism-based rainy day blurred image enhancement model, which comprises the following components:
(1) and (5) preliminarily extracting features. The input image was subjected to two convolution processes with convolution kernel sizes 7 x 64, 5 x 128 and a step size of 1,
and obtaining the preliminarily extracted features.
(2) And extracting features by multi-scale expansion convolution. Dilation convolution with convolution kernel 3 x 256, namely dia _ c1, dia _ c2, dia _ c3
The values of the factors (differentiation rates) are 2, 4 and 6, respectively, and three features are obtained after the expansion convolution processing.
(3) An attention map is generated. Inputting the three features into the attention mechanism model introduced in 3.2, outputting the result as three attention diagrams,
and performing feature fusion on the three attention diagrams and the original image.
(4) And outputting the image. The output was sequentially subjected to 3 convolution processes with a convolution kernel size of 3 × 256 and a step size of 1, 2 deconvolution operations with convolution kernel sizes of 4 × 128, 4 × 64 and a step size of 2, and convolution processes with a convolution kernel size of 3 × 64 and a step size of 1, and finally the image after rain division was output.
As shown in fig. 3, the attention mechanism model designed in the present invention is composed of three layers of residual error network (ResBlock), long and short memory neural network (LSTM), and convolutional layer (conv):
(1) residual error network: since the Normalization layer ignores absolute differences between image features, we use a residual network that removes the Normalization layer from the conventional residual network structure.
(2) Long and short memory neural networks (LSTM).
(3) And (3) rolling layers: a normal convolution with a convolution kernel size of 3 x 3 and a step size of 1 is used.
The experimental procedure was as follows:
(1) and carrying out uniform resolution processing on the data set pictures to divide a training set and a testing set.
(2) And establishing a raindrop fuzzy image enhancement model based on an attention mechanism, and initializing parameters of the network.
(3) And training the model by using the divided training set, and continuously updating the model parameters by performing back propagation through a loss function so as to minimize the calculated loss and optimize the performance effect of the model.
(4) Training every 50 pairs of pictures, setting the maximum iteration number to be 3000, storing model parameters, and testing the model by using a test set.
To illustrate the superiority of the model proposed by the present invention, we performed a comparison experiment on the algorithms proposed by Tanaka et al and Eigen et al, and using the raindrop image data set self-constructed herein, the image quality evaluation results of the experiment are shown in table 1, and fig. 5 is an experimental effect diagram of 4 images in the selected test set.
Table 1 picture quality evaluation results
As can be seen from Table 1, the peak SNR of the model for rain removal proposed by Tanaka et al is 26.84, the peak SNR of the model proposed by Eigen et al is 27.41, and the peak SNR of the model proposed in this chapter reaches 28.37, while the structural similarity of the model proposed by the present invention reaches 0.9091, which is higher than that of the other two models. As can be seen from FIG. 5, the Tanaka et al method cannot remove dense raindrops and introduce much noise to the image, the Eigen et al method cannot process the image to obtain an image with abundant details, and the image has distortion, but the image processed by the model of the present invention has good effect in subjective visual perception. Therefore, the algorithm model of the invention is superior to other algorithms in terms of peak signal-to-noise ratio and structural similarity.
Claims (5)
1. A single raindrop image enhancement method based on an attention mechanism is characterized in that a perception field of the single raindrop image enhancement method is enhanced by adopting multi-scale expansion convolution with different expansion factors to obtain more characteristic information; the model introduces an attention mechanism to construct an attention mechanism model to pay attention to a raindrop area in an image, so that raindrops are removed better, and the recovery of raindrop fuzzy image details is realized.
2. The method of claim 1, wherein the method comprises the steps of:
(1) and (5) preliminarily extracting features. The input image was subjected to two convolution processes with convolution kernel sizes of 7 x 64, 5 x 128 and step size of 1 to obtain the preliminary extracted features.
(2) And extracting features by multi-scale expansion convolution. Three features obtained after the three expanding convolution treatments.
(3) An attention map is generated. And respectively inputting the three features into the attention mechanism model, outputting the result into three attention diagrams, and performing feature fusion on the three attention diagrams and the original image.
(4) And outputting the image. The output was sequentially subjected to 3 convolution processes with a convolution kernel size of 3 × 256 and a step size of 1, 2 deconvolution operations with convolution kernel sizes of 4 × 128, 4 × 64 and a step size of 2, and convolution processes with a convolution kernel size of 3 × 64 and a step size of 1, and finally the image after rain division was output.
3. The multi-scale dilation convolution of claim 2, the method comprising: the expansion convolution of the convolution kernel 3 × 256, namely dia _ c1, dia _ c2, dia _ c3, has expansion factor (expansion rate) values of 2, 4, and 6, respectively.
4. The attention mechanism model of claim 2, the method comprising the steps of:
the attention mechanism model is composed of three layers of residual error networks (ResBlock), a long and short memory neural network (LSTM) and a convolutional layer (conv):
(1) residual error network: since the Normalization layer ignores absolute differences between image features, we use a residual network that removes the Normalization layer from the conventional residual network structure.
(2) Long and short memory neural networks (LSTM);
(3) and (3) rolling layers: a normal convolution with a convolution kernel size of 3 x 3 and a step size of 1 is used.
5. The attention-based single raindrop image enhancement method according to claim 1, wherein: the network loss resistance LadvAnd a perceptual loss LcComprises the following steps:
attention mechanism model loss LattAnd a perceptual loss Lp。
Lp＝LMSE(VGG(O)，VGG(T)) (2)
Pre-training the network by using VGG16, extracting high-level features and monitoring to ensure the quality of generated images, wherein AtThe method is characterized in that an attention machine drawing is generated by an attention machine model at time T, M is a binary mask, N is 4, theta is 0.8, O is a raindrop removing image output after model processing, and T is an original clear raindrop-free image corresponding to the raindrop image.
The overall loss function of the model herein is:
L＝Latt+Lp (3)。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910587306.4A CN112132756A (en) | 2019-06-24 | 2019-06-24 | Attention mechanism-based single raindrop image enhancement method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910587306.4A CN112132756A (en) | 2019-06-24 | 2019-06-24 | Attention mechanism-based single raindrop image enhancement method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112132756A true CN112132756A (en) | 2020-12-25 |
Family
ID=73849748
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910587306.4A Pending CN112132756A (en) | 2019-06-24 | 2019-06-24 | Attention mechanism-based single raindrop image enhancement method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112132756A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112767280A (en) * | 2021-02-01 | 2021-05-07 | 福州大学 | Single image raindrop removing method based on loop iteration mechanism |
| CN113160078A (en) * | 2021-04-09 | 2021-07-23 | 长安大学 | Method, device and equipment for removing rain from traffic vehicle image in rainy day and readable storage medium |
-
2019
- 2019-06-24 CN CN201910587306.4A patent/CN112132756A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112767280A (en) * | 2021-02-01 | 2021-05-07 | 福州大学 | Single image raindrop removing method based on loop iteration mechanism |
| CN112767280B (en) * | 2021-02-01 | 2022-06-14 | 福州大学 | Single image raindrop removing method based on loop iteration mechanism |
| CN113160078A (en) * | 2021-04-09 | 2021-07-23 | 长安大学 | Method, device and equipment for removing rain from traffic vehicle image in rainy day and readable storage medium |
| CN113160078B (en) * | 2021-04-09 | 2023-01-24 | 长安大学 | Method, device and equipment for removing rain from traffic vehicle image in rainy day and readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110782399B (en) | Image deblurring method based on multitasking CNN | |
| CN112419184B (en) | Spatial attention map image denoising method integrating local information and global information | |
| CN109087273B (en) | Image restoration method, storage medium and system based on enhanced neural network | |
| CN108549892B (en) | License plate image sharpening method based on convolutional neural network | |
| CN111028177B (en) | Edge-based deep learning image motion blur removing method | |
| CN111275643B (en) | Real noise blind denoising network system and method based on channel and space attention | |
| CN103761710B (en) | The blind deblurring method of efficient image based on edge self-adaption | |
| CN113808032B (en) | Multi-stage progressive image denoising algorithm | |
| CN112164011B (en) | Motion image deblurring method based on self-adaptive residual error and recursive cross attention | |
| CN108830812B (en) | Video high frame rate reproduction method based on grid structure deep learning | |
| CN113450290B (en) | Low-illumination image enhancement method and system based on image inpainting technology | |
| CN108765330B (en) | Image denoising method and device based on global and local prior joint constraint | |
| CN110942436B (en) | Image deblurring method based on image quality evaluation | |
| CN107133923A (en) | A kind of blurred picture non-blind deblurring method based on self-adaption gradient sparse model | |
| CN114331886B (en) | Image deblurring method based on depth features | |
| CN109949234B (en) | Video restoration model training method and video restoration method based on deep network | |
| CN112819705B (en) | Real image denoising method based on mesh structure and long-distance correlation | |
| CN112132756A (en) | Attention mechanism-based single raindrop image enhancement method | |
| CN106960417A (en) | A kind of noise based on the notable structure of image obscures Image Blind mesh Deconvolution Method | |
| CN117333398A (en) | Multi-scale image denoising method and device based on self-supervision | |
| CN113344804A (en) | Training method of low-light image enhancement model and low-light image enhancement method | |
| CN113888405B (en) | Denoising and demosaicing method based on clustering self-adaptive expansion convolutional neural network | |
| CN115345791A (en) | Infrared image deblurring algorithm based on attention mechanism residual error network model | |
| CN113096032B (en) | Non-uniform blurring removal method based on image region division | |
| CN118097159A (en) | Self-supervision image denoising method based on three-stage feature extraction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| DD01 | Delivery of document by public notice | ||
| DD01 | Delivery of document by public notice |
Addressee: Li Jinhua Document name: Notice of publication of application for patent for invention |
|
| DD01 | Delivery of document by public notice | ||
| DD01 | Delivery of document by public notice |
Addressee: Li Jinhua Document name: deemed notice of withdrawal |
|
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20201225 |