CN115311155A - Improved KPN-based network picture rain removing method, system and storage medium - Google Patents
Improved KPN-based network picture rain removing method, system and storage medium Download PDFInfo
- Publication number
- CN115311155A CN115311155A CN202210843808.0A CN202210843808A CN115311155A CN 115311155 A CN115311155 A CN 115311155A CN 202210843808 A CN202210843808 A CN 202210843808A CN 115311155 A CN115311155 A CN 115311155A
- Authority
- CN
- China
- Prior art keywords
- network
- layer
- convolution
- kpn
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 62
- 238000012545 processing Methods 0.000 claims abstract description 79
- 238000000605 extraction Methods 0.000 claims abstract description 75
- 230000007246 mechanism Effects 0.000 claims abstract description 16
- 230000004913 activation Effects 0.000 claims description 28
- 238000011176 pooling Methods 0.000 claims description 18
- 238000012937 correction Methods 0.000 claims description 13
- 239000011159 matrix material Substances 0.000 claims description 11
- 238000004891 communication Methods 0.000 claims description 9
- 230000008569 process Effects 0.000 claims description 8
- 238000012986 modification Methods 0.000 claims description 4
- 230000004048 modification Effects 0.000 claims description 4
- 238000010606 normalization Methods 0.000 claims description 4
- 230000000694 effects Effects 0.000 abstract description 11
- 238000004364 calculation method Methods 0.000 abstract description 7
- 238000013528 artificial neural network Methods 0.000 abstract description 4
- 238000004422 calculation algorithm Methods 0.000 description 13
- 238000010586 diagram Methods 0.000 description 9
- 230000003044 adaptive effect Effects 0.000 description 3
- 230000015556 catabolic process Effects 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 3
- 238000006731 degradation reaction Methods 0.000 description 3
- 238000001914 filtration Methods 0.000 description 3
- 238000005070 sampling Methods 0.000 description 3
- 230000004927 fusion Effects 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000006378 damage Effects 0.000 description 1
- 238000013434 data augmentation Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000002427 irreversible effect Effects 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000013178 mathematical model Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000000149 penetrating effect Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20216—Image averaging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Multimedia (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Processing (AREA)
Abstract
The invention provides a method, a system and a storage medium for removing rain based on an improved KPN network picture, wherein the method comprises the following steps: inputting the preprocessed picture into a feature extraction network layer to obtain an output picture after feature extraction; inputting the output graph after the characteristic extraction into a common convolution network layer introducing a space attention mechanism to obtain a prediction kernel; inputting the obtained prediction kernels into convolution network layers with different sizes to obtain prediction graphs of different receptive field characteristics; and weighting and summing the characteristics of the obtained prediction images, inputting the characteristics into the depth separable convolution layer, and obtaining the rain removal enhancement image. According to the method, the image rain removing processing is realized through the improved KPN based image rain removing method and the improved KPN based image rain removing processing, the deep separable convolution is introduced, and the parameter quantity and the calculation complexity of the convolution neural network are greatly reduced; a spatial attention mechanism is introduced, so that the feature expression of a key region of the feature map is improved, and the image enhancement effect is effectively improved.
Description
Technical Field
The invention relates to the field of data processing, in particular to a method, a system and a storage medium for removing rain based on an improved KPN network picture.
Background
Digital images are ubiquitous in our productive life. Image degradation is easily caused by various factors such as noise destruction, camera shake, object motion, resolution limitation, haze, rain stripes, or a combination thereof during imaging. Image degradation is generally irreversible and can only be handled by means of image enhancement algorithms. Among them, there is an increasing interest in image degradation due to raindrops falling on the camera.
The main ideas of raindrop-free image enhancement include removing raindrop noise from an original input image through filtering or other operations, and generating a raindrop-free image through a countermeasure generation network. The fundamental idea of image rain removal enhancement is to obtain the mapping relation from a noise image to a noise-free image, and the method comprises a traditional noise removal enhancement method and a deep learning-based raindrop removal algorithm. The traditional denoising and enhancing method mainly comprises two types, namely spatial domain filtering and frequency domain filtering, can remove partial noise, is difficult to remove complex noise, has complex operation, can not carry out blind denoising, has long time consumption, and does not have real-time property when being deployed and applied at a terminal part. With the rise of deep learning, more and more neural network models are applied to image enhancement. The CNN, resnet, u-net, hinet and GAN series are mainly popular, and the network models have innovation breakthroughs in network architecture. With the network evolution becoming more and more complex, the network parameter quantity is also larger and larger, the real-time performance of the algorithm is restricted, and the requirement on terminal hardware is higher. Many of the rain removing methods based in part on deep learning are based on rain removing pattern assumptions or a priori knowledge, and the mathematical model of raindrops is expressed as:
i = (1-M) <' > B + R formula (1)
Wherein, I represents a degraded image affected by raindrops, M represents a binary mask (mask is 1 indicates that the pixels in the region are raindrop affected regions, otherwise, indicates that the pixels in the region are not affected by raindrops), B represents an image background, and R represents an effect caused by raindrops, and represents a complex mixture of background information and light rays reflected by the environment and penetrating through the raindrops. At this time, a large amount of fine adjustment and optimization processes are needed for a rain removing network, and various situations of a real rainfall scene cannot be covered while time is consumed. The problem can be well solved by an image enhancement algorithm irrelevant to the rain model. The KPN algorithm absorbs the thought of an excellent de-noising algorithm, the algorithm does not make a modeling hypothesis on how rainwater in the image is generated, a network structure integrates u-net and kernel prediction, a pixel-by-pixel filter kernel is predicted through the u-net network, the filter kernel is convoluted with an input noise image to reduce noise, the method has great advantages in the aspect of image rain removing efficiency, the effect similar to that of a rain removing SOTA algorithm is obtained, rain removing of a single image can be kept within 10ms, and the method can be conveniently deployed on terminal hardware.
The evaluation indexes of the image enhancement accuracy include PSNR (Peak Signal to Noise Ratio) and SSIM (structural similarity index). PSNR is an objective standard for evaluating images, and the calculation formula is as follows:
SSIM is an index for measuring the similarity between two images. Of the two images used, one is an uncompressed undistorted image and the other is a distorted image, where the range of values is between 0 and 1, 1 is completely consistent, and 0 is completely inconsistent. Assuming that the two given images of M X N are X, Y, where the mean, standard deviation of X, and the covariance of X and Y are denoted ux, σ X, σ xy, respectively, the comparison functions defining brightness, contrast, and texture are:
SSIM(X,Y)＝[l(X,Y)] α [c(X,Y)] β [s(X,Y)] γ formula (6)
Qing Guo et al propose to adopt RainMix data augmentation method expand real scene data, and then train KPN network to go rain, in the image go rain task (involving to go raindrops, mainly go rain line) speed and effectiveness show good. Bin Zhang et al propose AME-KPN (attribute metric enhanced kernel prediction networks) image denoising method, adopt almost no cost attention module to refine feature map, further utilize interframe and intraframe redundancy of noise image. Wherein the prediction kernel recovers roughly clean pixels at their corresponding locations by adaptive convolution operations, weighting and subtracting the residual to compensate the prediction kernel. Talmaj et al propose image denoising methods for multi-KPNs that predict kernels of more than one size, but with kernels of different sizes. Kernels of different sizes help to extract different information from the image, enabling better reconstruction, and kernel fusion ensures that the extracted information is preserved while computational efficiency is maintained.
Disclosure of Invention
The invention mainly aims to provide an image rain removing method, device and readable storage medium based on an improved KPN (Kernel Key network), aiming at solving the technical problems of poor image enhancement effect, excessive network parameters, high hardware loss and low calculation efficiency of the conventional image rain removing processing method based on the KPN.
In a first aspect, an improved KPN network-based image rain removing method comprises a feature extraction network layer, a common convolution network layer, different-scale convolution network layers and a depth separable convolution network layer;
the method comprises the following steps:
inputting the preprocessed pictures into a feature extraction network layer, and acquiring output pictures after feature extraction;
inputting the output graph after the characteristic extraction into a common convolution network layer introducing a space attention mechanism to obtain a prediction kernel;
inputting the obtained prediction kernels into convolution network layers with different sizes to obtain prediction graphs of different receptive field characteristics;
and (4) performing weighted summation on the characteristics of the obtained prediction images, inputting the weighted summation into the depth separable convolution layer, and obtaining a rain removal enhancement image.
In some embodiments, the step of inputting the preprocessed picture into a feature extraction network layer of the improved KPN network model, and acquiring an output graph of the feature extraction network layer includes:
the feature extraction network layer comprises a first network sublayer, a second network sublayer, a third network sublayer and a fourth network sublayer, the first network sublayer comprises a depth separable convolutional layer, the second network sublayer comprises a mean value pooling layer and a depth separable convolutional layer, the third network sublayer comprises a bicubic interpolation upsampling layer, a depth separable convolutional layer and a modified linear unit activation layer, and the fourth network sublayer comprises a bicubic interpolation upsampling layer;
carrying out convolution processing on the input image through the depth separable convolution layer of the first network sublayer to obtain a feature image after the convolution processing;
pooling the feature map average value after convolution processing through the average pooling layer of the second network sublayer, and performing convolution processing on the depth separable convolution layer to obtain a pooled feature map and a feature map after convolution processing;
and performing interpolation upsampling processing on the pooled and convolved feature maps through a bicubic interpolation upsampling layer of the third network sublayer, performing convolution processing on the depth separable convolution layer, performing correction activation processing on a correction linear unit activation layer, performing residual connection with the pooled and convolved feature maps acquired by the second network sublayer, and acquiring a feature extraction layer output map.
In some embodiments, the convolving the input map by the depth separable convolutional layer, and the obtaining the convolved feature map includes:
performing convolution processing on an input graph through depth-separable convolutional layers, wherein the depth-separable convolutional layers comprise depth convolutional layers and point-by-point convolutional layers;
performing single-channel convolution operation on the input image through the depth convolution layer to obtain a feature image after the convolution operation;
and integrating all channel information on the position of a single feature point on the feature map after the convolution operation through point-by-point convolution to complete feature extraction, and acquiring the feature map after the convolution processing.
In some embodiments, the number of the second network sublayers and the third network sublayers is multiple, the obtaining the output map of the feature extraction layer includes performing interpolation upsampling processing on the feature map subjected to pooling and convolution processing by using a bicubic interpolation upsampling layer of the third network sublayer, performing convolution processing on the depth separable convolutional layer, and performing modification activation processing on the linear unit activation layer, and performing residual connection with the feature map obtained by pooling and convolution processing by using the second network sublayer, where the obtaining the output map of the feature extraction layer includes:
performing interpolation upsampling processing on the pooled feature map and the feature map after convolution processing through a bicubic interpolation upsampling layer of the third network sublayer, performing convolution processing on the depth separable convolution layer, and performing correction activation processing on a correction linear unit activation layer to obtain a feature map after correction activation;
and residual error connection is carried out on the corrected and activated feature map output by each third network sublayer and the feature map output by each second network sublayer after convolution processing, and the feature map obtained after residual error connection is the feature map after pooling and convolution processing.
In some embodiments, before the step of inputting the preprocessed picture into the feature extraction network layer of the improved KPN network model and acquiring the feature-extracted output map, the method further includes the following steps:
adjusting the size of the input picture to obtain the picture with the adjusted size;
and carrying out data enhancement processing on the picture with the adjusted size to obtain the preprocessed picture.
In some embodiments, the data enhancement process includes, but is not limited to, the following operations:
random rotation, horizontal inversion and normalization processing.
In some embodiments, the step of inputting the output graph after feature extraction into a common convolutional network layer introducing a spatial attention mechanism to obtain a prediction kernel specifically includes the following steps:
inputting the output graph after feature extraction into a common convolution network layer introducing a space attention mechanism, and learning the output graph after feature extraction on a two-dimensional plane for all channels to obtain a weight matrix;
and adding the weight matrix to the output image after the characteristic extraction to obtain a prediction kernel.
In a second aspect, the application provides an improved KPN network image rain removal system, where the improved KPN network comprises a feature extraction network layer, a common convolution network layer, different scale convolution network layers, and a depth separable convolution network layer;
the rain removing system based on the improved KPN network image comprises:
the feature extraction module is used for inputting the preprocessed pictures into a feature extraction layer of the improved KPN network model and acquiring an output image after feature extraction;
the common convolution module is in communication connection with the feature extraction module and is used for inputting the output image after feature extraction into a common convolution network layer introducing a space attention mechanism to obtain a prediction kernel; the different-scale convolution module is in communication connection with the common convolution module and is used for inputting the acquired prediction kernels into convolution network layers with different sizes to acquire prediction images of different receptive field characteristics;
and the depth separable convolution module is in communication connection with the convolution modules with different scales and is used for weighting and summing the characteristics of the obtained prediction images, inputting the characteristics into the depth separable convolution layer and obtaining the rain removal enhancement image.
In a third aspect, the present application provides an improved KPN-based network image rain removal device, which includes a processor, a memory, and an improved KPN-based network image rain removal program stored on the memory and executable by the processor, wherein the improved KPN-based network image rain removal program, when executed by the processor, implements the steps of the improved KPN-based network image rain removal method as described above.
In a fourth aspect, the present application provides a readable storage medium, on which a rain removal program based on an improved KPN network image is stored, wherein when the rain removal program based on an improved KPN network image is executed by a processor, the steps of the rain removal method based on an improved KPN network image as described above are implemented.
According to the method, the image rain removing processing is realized through the improved KPN based image rain removing method and the improved KPN based image rain removing processing, the deep separable convolution is introduced, the conventional convolution is replaced by the deep separable convolution, and the parameter quantity and the calculation complexity of the convolution neural network are greatly reduced; and a spatial attention matrix is introduced, so that the feature expression of a key region of a feature map is improved, and the image enhancement effect is effectively improved.
Drawings
FIG. 1 is a schematic diagram of a network structure of an image rain removing KPN according to one prior art scheme;
FIG. 2 is a schematic diagram of another image rain removal KPN network structure in the prior art solution I;
FIG. 3 is a schematic diagram of an image rain removing KPN network structure in a second prior art;
FIG. 4 is a schematic diagram of an image rain removing KPN network structure in prior art solution III;
FIG. 5 is a schematic diagram of an image rain removal KPN network structure in prior art solution four;
FIG. 6 is a schematic diagram of an image rain removal KPN network structure in prior art solution five;
fig. 7 is a KPN network structure diagram based on an improved KPN network image rain removal method according to an embodiment of the present application;
FIG. 8 is a flowchart of a method for removing rain based on an improved KPN network image according to an embodiment of the present application;
fig. 9 is a functional block diagram of a rain removal system based on an improved KPN network image according to an embodiment of the present application.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In the prior art, as shown in fig. 1, the first scheme is implemented as follows: the KPN network inputs successive N +1 frames of images, where N frames are successive multi-noise images, the estimated noise of one frame of image. The N frames of noise images are filtered and denoised by a pixel-by-pixel filter output by a network to generate N frames of clean denoised images, and a final noiseless image Y can be generated after alignment and averaging. And 2-5 layers of the KPN are 3 layers of convolution and a mean pooling layer, 6-8 layers are respectively subjected to Bilinear interpolation upsampling and then convolution, and meanwhile skip connection is required to be performed, and a pixel kernel is output. And (4) performing convolution on the pixel kernels and the original input corresponding frames respectively, and finally performing weighted average to obtain an output image. The network structure is simple and clear, the model parameters are few, some rain lines can be removed, but the PSNR and SSIM are slightly worse than the SOTA, and large raindrops can not be removed almost.
Another scheme adopts the same network structure as the first scheme, and the structure is shown in fig. 2.
As shown in fig. 3, the second prior art solution is implemented as:
some improvements were made to KPN networks, where the predicted kernels for multi-KPN are not just one size, but multiple sizes, with kernels fused into one kernel on a per pixel basis. Kernels of different sizes help to extract different information from the image, thereby enabling better reconstruction, and kernel fusion ensures retention of the extracted information while maintaining computational efficiency.
As shown in fig. 4, the third prior art solution is implemented as follows:
the deformable kernel prediction network improves the defect that the KPN algorithm cannot be used for denoising color images, and simultaneously introduces a deformable convolution kernel to improve the network denoising performance. The method replaces the ordinary convolution with the deformable convolution, namely, the position of each sampling point in the convolution kernel is added with a variable of offset. With these variables, the convolution kernel can be sampled arbitrarily around the current position. The method increases the complexity of the algorithm, increases the hardware consumption and does not effectively improve the use effect.
As shown in fig. 5, the prior art scheme four-way through the two-channel deformable KPN is implemented as:
the method provides a deformable kernel prediction network structure of two channels, wherein an N-frame Bayer image and an RGB image corresponding to the N-frame Bayer image are respectively sent to the two channels of a network, the network is provided with two identical similar u-net channels, and the tail end of each channel is divided into two channels which are respectively a prediction kernel feature map and an offset feature map. And multiplying the feature maps pairwise to obtain a final prediction kernel and offset feature map, and exchanging and fusing the information of the two channels so that the robustness of the output feature map is higher. At the end of a network, a multi-frame Bayer image is subjected to offset processing by using an offset characteristic diagram, then the offset Bayer image is subjected to final noise reduction operation by using a prediction core, so that a multi-frame Bayer image subjected to noise reduction can be obtained, and finally a single-frame Bayer image can be recovered by mean processing
As shown in fig. 6, the fifth prior art solution is implemented as:
the method provides a KPN network structure with attention mechanism enhancement for image enhancement, and an attention module with almost no cost is used for extracting a feature map, so that the interframe and intraframe redundancy of a noise image is effectively utilized. The AME-KPN network comprises a feature extraction module, an output branch module and a reconstruction module, wherein Channel Attention and spatial Attention structures are introduced into the KPN network during feature extraction
The AME-KPNs output a pixel-by-pixel spatially adaptive kernel, a residual map, and a corresponding weight map, where the prediction kernel approximately restores the clean pixels in their corresponding locations by an adaptive convolution operation, and then the residuals are weighted and summed to compensate for the limited receptive field of the prediction kernel. Simulation and practical experiments verify the robustness of the proposed AME-KPN.
The image rain removing schemes based on the KPN absorb the thought of an excellent noise removing algorithm, a pixel-by-pixel filter kernel is predicted by the algorithm through the u-net network, and is convoluted with an input noise image to reduce noise, so that the method has great advantages in the aspect of image rain removing efficiency, but the PSNR, the SSIM and the rain removing SOTA algorithm have some differences.
In view of this, the present application provides an image rain removal method based on an improved KPN network, which effectively solves the technical problems of poor image enhancement effect, excessive network parameters, large hardware loss, and low computational efficiency of the existing image rain removal processing method based on the KPN network.
In a first aspect, please refer to fig. 7 and 8, an embodiment of the present invention provides a method for removing rain based on an improved KPN network image, where the improved KPN network includes a feature extraction network layer, a common convolution network layer, convolution network layers with different scales, and a depth separable convolution network layer;
the method comprises the following steps:
s1, inputting the preprocessed picture into a feature extraction network layer, and acquiring an output picture after feature extraction;
s2, inputting the output image after the characteristic extraction into a common convolution network layer introducing a space attention mechanism to obtain a prediction kernel;
s3, inputting the obtained prediction kernels into convolution network layers with different sizes to obtain prediction graphs of different receptive field characteristics;
and S4, weighting and summing the acquired features of the prediction images, inputting the feature into the depth separable convolution layer, and acquiring a rain removal enhancement image.
According to the method, the image rain removing processing is realized through the improved KPN based image rain removing method and the improved KPN based image rain removing processing, the deep separable convolution is introduced, the conventional convolution is replaced by the deep separable convolution, and the parameter quantity and the calculation complexity of the convolution neural network are greatly reduced; and a space attention matrix is introduced, so that the feature expression of a key region of the feature map is improved, and the image enhancement effect is effectively improved.
In an embodiment, the step of inputting the preprocessed picture into a feature extraction network layer of the improved KPN network model, and acquiring an output graph of the feature extraction network layer includes:
the feature extraction network layer comprises a first network sublayer, a second network sublayer, a third network sublayer and a fourth network sublayer, the first network sublayer comprises a depth separable convolutional layer, the second network sublayer comprises a mean value pooling layer and a depth separable convolutional layer, the third network sublayer comprises a bicubic interpolation upsampling layer, a depth separable convolutional layer and a modified linear unit activation layer, and the fourth network sublayer comprises a bicubic interpolation upsampling layer;
performing convolution processing on the input image through the depth separable convolution layer of the first network sublayer to obtain a feature image after the convolution processing;
pooling the feature map average value after convolution processing through the average pooling layer of the second network sublayer, and performing convolution processing on the depth separable convolution layer to obtain a pooled feature map and a feature map after convolution processing;
and performing interpolation upsampling processing on the pooled and convolved feature maps through a bicubic interpolation upsampling layer of the third network sublayer, performing convolution processing on the depth separable convolutional layer, and performing correction activation processing on the linear unit activation layer, and performing residual connection with the pooled and convolved feature maps obtained by the second network sublayer to obtain a feature extraction layer output map.
In a more specific embodiment, the first network sublayer is three depth separable convolutional layers at the leftmost side of the frame feature extraction network layer in fig. 7, the second network sublayer is 2 to 5 layers in the frame of fig. 7, each layer includes a mean pooling layer from left to right and three depth separable convolutional layers, the second network sublayer is 6 to 8 layers in the frame of fig. 7, each layer includes one Bicubic interpolated upsampling layer (Bicubic upsampling layer), two depth separable convolutional layers, and a ReLU activation layer (modified linear element activation layer), and the fourth network sublayer is one Bicubic interpolated upsampling layer at the rightmost side in the frame of fig. 7 (Bicubic upsampling layer).
In an embodiment, the convolving the input map by the depth separable convolutional layer, and the obtaining the feature map after the convolving includes:
performing convolution processing on an input graph through depth-separable convolutional layers, wherein the depth-separable convolutional layers comprise depth convolutional layers and point-by-point convolutional layers;
performing single-channel convolution operation on the input image through the depth convolution layer to obtain a feature image after the convolution operation;
and integrating all channel information on the position of a single feature point on the feature map subjected to the convolution operation through point-by-point convolution to complete feature extraction, and acquiring the feature map subjected to convolution processing.
The present application introduces a depth separable Convolution layer, i.e. replacing the conventional Convolution with a depth separable Convolution, which consists of a depth Convolution (Depthwise Convolution) and a point-wise Convolution (Pointwise Convolution). One convolution kernel of the deep convolution is responsible for one channel, one channel is only convoluted by one convolution kernel, the number of the finished Feature maps is the same as that of the channels of the input layer, and the Feature maps cannot be expanded. Therefore, poitwise conversion is needed to combine these Feature maps to generate a new Feature map, and the number of parameters and the operation cost are low. The number of parameters required by convolution calculation is reduced by splitting the correlation between the space dimensionality and the channel dimensionality or called depth dimensionality, the depth convolution is firstly used for carrying out convolution operation on a single channel, then point convolution in a convolution mode of 1 x 1 is adopted, and feature extraction is completed by integrating all channel information on the position of a single feature point, because the initial design of a 1 x 1 convolution kernel reduces the parameter number and the calculated amount, the parameter amount of the depth separable convolution is reduced by 8-9 times compared with standard convolution, the parameter amount and the calculation complexity of a Convolution Neural Network (CNN) are greatly reduced, and meanwhile, although the parameter number is greatly reduced, the accuracy rate is not influenced.
In the interpolation upsampling process, the bilinear is not used, bicubic interpolation upsampling is used, the calculated amount of Bicubic is larger than that of the bilinear, but the interpolation restoring characteristic of the Bicubic has better effect.
In an embodiment, the number of the second network sublayers and the third network sublayers is multiple, the obtaining of the output map of the feature extraction layer includes performing interpolation upsampling processing on the feature map subjected to pooling and convolution processing by using a bicubic interpolation upsampling layer of the third network sublayer, performing convolution processing on the depth separable convolutional layer, and performing modification activation processing on a modified linear unit activation layer, and performing residual connection with the feature map subjected to pooling and convolution processing obtained by the second network sublayer, where the step of obtaining the output map of the feature extraction layer includes:
performing interpolation upsampling processing on the pooled feature map and the feature map after convolution processing through a bicubic interpolation upsampling layer of the third network sublayer, performing convolution processing on the depth separable convolution layer, and performing correction activation processing on a correction linear unit activation layer to obtain a feature map after correction activation;
and residual error connection is carried out on the corrected and activated feature map output by each third network sublayer and the feature map output by each second network sublayer after convolution processing, and the feature map obtained after residual error connection is the feature map after pooling and convolution processing.
In a more specific embodiment, the layer 6 of the feature extraction network layer performs Bicubic interpolation upsampling on the feature map output by the layer 5, and performs skip connection with the fourth layer while performing depth separable convolution and ReLU activation. And the 7 th layer performs Bicubic interpolation up-sampling on the feature map output by the 6 th layer, performs depth separable convolution and ReLU activation layer, and performs skip connection (residual connection) with the 2 nd layer. And the 8 th layer performs Bicubic interpolation up-sampling on the feature map output by the 7 th layer, and performs depth separable convolution and ReLU activation layer and skip connection with the 2 nd layer.
In an embodiment, before the step of inputting the preprocessed picture into the feature extraction network layer of the improved KPN network model and acquiring the output graph after feature extraction, the method further includes the following steps:
adjusting the size of the input picture to obtain the picture with the adjusted size;
and acquiring the preprocessed picture by performing data enhancement processing on the picture with the adjusted size.
In one embodiment, the data enhancement processing includes, but is not limited to, the following operations:
random rotation, horizontal inversion and normalization processing.
In one embodiment, the input picture is first resize, 480 × 480 in size, and then data enhancement processing, including random rotation and horizontal flipping, and normalization processing.
In an embodiment, the step of inputting the output graph after feature extraction into a common convolutional network layer introducing a spatial attention mechanism to obtain a prediction kernel specifically includes the following steps:
inputting the output graph after feature extraction into a common convolution network layer introducing a space attention mechanism, and learning the output graph after feature extraction on a two-dimensional plane for all channels to obtain a weight matrix;
and adding the weight matrix to the output image after the characteristic extraction to obtain a prediction kernel.
Not all regions in the image contribute equally to the task, and only the regions relevant to the task need to be of interest. For the convolutional neural network, each layer of the CNN outputs a feature map, and the spatial attention mechanism is that for all channels, a weight matrix is learned for the feature map on a two-dimensional plane, and a weight is learned for each pixel. These weights represent the importance of certain spatial position information, and a spatial attention matrix is formed, and the spatial attention matrix is added to the original feature map to increase useful features and weaken useless features, thereby achieving the effects of feature selection and enhancement.
In one embodiment, the prediction kernel and the input image are convolved in different sizes to obtain prediction images with different receptive field characteristics, the prediction image characteristics are weighted and summed, the convolution layers can be separated through a depth of 5 x 1, and finally the rain-removed enhanced image is output.
The improved KPN network model provided by the present application was tested using image pairs under different conditions, each image pair consisting of a clean, noise-free true image and a noisy degraded image. The test data set covers various scenes such as indoor and outdoor underground, parking lots and non-parking lots with multiple dimensions of different weather, different time, different illumination conditions and the like. PSNR and SSIM between the image after the model is degrained and the true value image are calculated by formula (2) and formulas (3) to (6). The result shows that the image enhancement effect is improved, namely the SSIM and PSNR first contrast schemes are improved by 0.12 and 3.54, and the fifth contrast scheme is improved by 0.06 and 3.58 respectively.
In a second aspect, please refer to fig. 9, the present application provides an image rain removing system based on an improved KPN network, wherein the improved KPN network comprises a feature extraction network layer, a common convolution network layer, a convolution network layer with different scales and a depth separable convolution network layer;
the rain removing system based on the improved KPN network image comprises a feature extraction module 100, a common convolution module 200, a convolution module 300 with different scales and a depth separable convolution module 400, wherein the feature extraction module 100 is used for inputting a preprocessed picture into a feature extraction layer of an improved KPN network model and acquiring an output image after feature extraction; a common convolution module 200, communicatively connected to the feature extraction module 100, configured to input the output graph after feature extraction into a common convolution network layer that introduces a spatial attention mechanism, and obtain a prediction kernel; the convolution module 300 with different scales is in communication connection with the ordinary convolution module 200 and is used for inputting the obtained prediction kernels into convolution network layers with different sizes to obtain prediction graphs of different receptive field characteristics; the depth separable convolution module 400 is in communication connection with the different scale convolution module 300, and is configured to perform weighted summation on the features of the obtained prediction maps, and input the weighted summation into the depth separable convolution layer to obtain a rain-removed enhanced image.
In a third aspect, the present application provides an improved KPN-based network image rain removal device, which includes a processor, a memory, and an improved KPN-based network image rain removal program stored on the memory and executable by the processor, wherein the improved KPN-based network image rain removal program, when executed by the processor, implements the steps of the improved KPN-based network image rain removal method as described above.
In a fourth aspect, the present application provides a readable storage medium, on which a rain removal program based on an improved KPN network image is stored, wherein when the rain removal program based on an improved KPN network image is executed by a processor, the steps of the rain removal method based on an improved KPN network image as described above are implemented.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising a … …" does not exclude the presence of another identical element in a process, method, article, or system that comprises the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) as described above and includes instructions for causing a terminal device to execute the method according to the embodiments of the present invention.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A rain removing method based on improved KPN network image is characterized in that,
the improved KPN network comprises a feature extraction network layer, a common convolution network layer, convolution network layers with different scales and a depth separable convolution network layer;
the rain removing method based on the improved KPN network image comprises the following steps:
inputting the preprocessed picture into a feature extraction network layer to obtain an output picture after feature extraction;
inputting the output graph after the characteristic extraction into a common convolution network layer introducing a space attention mechanism to obtain a prediction kernel;
inputting the obtained prediction kernels into convolution network layers with different sizes to obtain prediction graphs of different receptive field characteristics;
and weighting and summing the characteristics of the obtained prediction images, inputting the characteristics into the depth separable convolution layer, and obtaining the rain removal enhancement image.
2. An improved KPN network image rain removal method as claimed in claim 1, wherein said step of inputting the preprocessed picture into the feature extraction network layer of the improved KPN network model, and obtaining the feature extraction network layer output map comprises:
the feature extraction network layer comprises a first network sublayer, a second network sublayer, a third network sublayer and a fourth network sublayer, the first network sublayer comprises a depth separable convolution layer, the second network sublayer comprises a mean value pooling layer and a depth separable convolution layer, the third network sublayer comprises a bicubic interpolation upsampling layer, a depth separable convolution layer and a modified linear unit activation layer, and the fourth network sublayer comprises a bicubic interpolation upsampling layer;
performing convolution processing on the input image through the depth separable convolution layer of the first network sublayer to obtain a feature image after the convolution processing;
pooling the feature map average value after convolution processing through the average pooling layer of the second network sublayer, and performing convolution processing on the depth separable convolution layer to obtain a pooled feature map and a feature map after convolution processing;
and performing interpolation upsampling processing on the pooled and convolved feature maps through a bicubic interpolation upsampling layer of the third network sublayer, performing convolution processing on the depth separable convolutional layer, and performing correction activation processing on the linear unit activation layer, and performing communication connection with the pooled and convolved feature maps acquired by the second network sublayer to acquire a feature extraction layer output map.
3. An improved KPN network image rain removal method as claimed in claim 2, wherein said step of convolving the input map by depth separable convolutional layers, obtaining the convolved feature map comprises:
performing convolution processing on an input graph through depth-separable convolutional layers, wherein the depth-separable convolutional layers comprise depth convolutional layers and point-by-point convolutional layers;
performing single-channel convolution operation on the input image through the depth convolution layer to obtain a feature image after the convolution operation;
and integrating all channel information on the position of a single feature point on the feature map after the convolution operation through point-by-point convolution to complete feature extraction, and acquiring the feature map after the convolution processing.
4. The method for removing rain based on an improved KPN network image according to claim 2, wherein the number of the second network sub-layer and the third network sub-layer is multiple, the step of performing interpolation upsampling processing on the pooled and convolved feature map, performing convolution processing on the depth separable convolutional layer, and performing modification activation processing on the modified linear unit activation layer through the bicubic interpolation upsampling layer of the third network sub-layer, and performing residual connection with the pooled and convolved feature map obtained by the second network sub-layer to obtain the feature extraction layer output map comprises:
performing interpolation upsampling processing on the pooled feature map and the feature map subjected to convolution processing through a bicubic interpolation upsampling layer of the third network sublayer, performing convolution processing on the depth separable convolution layer, and performing correction activation processing on a correction linear unit activation layer to obtain a correction activated feature map;
and residual error connection is carried out on the corrected and activated feature map output by each third network sublayer and the feature map output by each second network sublayer after convolution processing, and the feature map obtained after residual error connection is the feature map after pooling and convolution processing.
5. An improved KPN network image rain removal method as claimed in claim 1, wherein said step of inputting the preprocessed picture into the feature extraction network layer of the improved KPN network model, and obtaining the feature extracted output map further comprises the following steps before:
adjusting the size of the input picture to obtain the picture with the adjusted size;
and acquiring the preprocessed picture by performing data enhancement processing on the picture with the adjusted size.
6. An improved KPN network image rain removal method as claimed in claim 5, wherein said data enhancement process includes but is not limited to the following operations:
random rotation, horizontal inversion and normalization processing.
7. The improved KPN network image rain-removing method according to claim 1, wherein the step of inputting the output graph after feature extraction into a common convolutional network layer introducing a spatial attention mechanism to obtain a prediction kernel specifically comprises the following steps:
inputting the output graph after feature extraction into a common convolution network layer introducing a space attention mechanism, and learning the output graph after feature extraction on a two-dimensional plane for all channels to obtain a weight matrix;
and adding the weight matrix to the output image after the characteristic extraction to obtain a prediction kernel.
8. A rain removing system based on improved KPN network images is characterized in that,
the improved KPN network comprises a feature extraction network layer, a common convolution network layer, convolution network layers with different scales and a depth separable convolution network layer;
the rain removing system based on the improved KPN network image comprises:
the feature extraction module is used for inputting the preprocessed pictures into a feature extraction layer of the improved KPN network model and acquiring an output image after feature extraction;
the ordinary convolution module is in communication connection with the feature extraction module and is used for inputting the output image after feature extraction into an ordinary convolution network layer introducing a space attention mechanism to obtain a prediction kernel; the different-scale convolution module is in communication connection with the common convolution module and is used for inputting the acquired prediction kernels into convolution network layers with different sizes to acquire prediction images of different receptive field characteristics;
and the depth separable convolution module is in communication connection with the different scale convolution modules and is used for weighting and summing the characteristics of the obtained prediction images and inputting the sum into the depth separable convolution layer to obtain the rain-removing enhanced image.
9. An improved KPN network image based rain removal apparatus, comprising a processor, a memory, and an improved KPN network image based rain removal program stored on the memory and executable by the processor, wherein the improved KPN network image based rain removal program, when executed by the processor, implements the steps of the improved KPN network image based rain removal method according to any one of claims 1 to 7.
10. A readable storage medium, having stored thereon an improved KPN network image based rain removal program, wherein the improved KPN network image based rain removal program, when executed by a processor, implements the steps of the improved KPN network image based rain removal method according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210843808.0A CN115311155A (en) | 2022-07-18 | 2022-07-18 | Improved KPN-based network picture rain removing method, system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210843808.0A CN115311155A (en) | 2022-07-18 | 2022-07-18 | Improved KPN-based network picture rain removing method, system and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115311155A true CN115311155A (en) | 2022-11-08 |
Family
ID=83857246
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210843808.0A Pending CN115311155A (en) | 2022-07-18 | 2022-07-18 | Improved KPN-based network picture rain removing method, system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115311155A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024152809A1 (en) * | 2023-01-17 | 2024-07-25 | 岚图汽车科技有限公司 | Image deraining network, and image deraining network training method and device |
-
2022
- 2022-07-18 CN CN202210843808.0A patent/CN115311155A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024152809A1 (en) * | 2023-01-17 | 2024-07-25 | 岚图汽车科技有限公司 | Image deraining network, and image deraining network training method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111709895A (en) | Image blind deblurring method and system based on attention mechanism | |
| CN110675347B (en) | Image blind restoration method based on group sparse representation | |
| CN108510451B (en) | Method for reconstructing license plate based on double-layer convolutional neural network | |
| CN111462019A (en) | Image deblurring method and system based on deep neural network parameter estimation | |
| KR101633397B1 (en) | Image restoration device, image restoration method and image restoration system | |
| CN111986084A (en) | Multi-camera low-illumination image quality enhancement method based on multi-task fusion | |
| CN114746895A (en) | Noise reconstruction for image denoising | |
| CN112529776B (en) | Training method of image processing model, image processing method and device | |
| US20240020796A1 (en) | Noise reconstruction for image denoising | |
| CN115345791A (en) | Infrared image deblurring algorithm based on attention mechanism residual error network model | |
| CN118097159A (en) | Self-supervision image denoising method based on three-stage feature extraction | |
| CN117114984A (en) | Remote sensing image super-resolution reconstruction method based on generation countermeasure network | |
| CN115311155A (en) | Improved KPN-based network picture rain removing method, system and storage medium | |
| CN112200752B (en) | Multi-frame image deblurring system and method based on ER network | |
| CN115311149A (en) | Image denoising method, model, computer-readable storage medium and terminal device | |
| CN112132757B (en) | General image restoration method based on neural network | |
| CN111402173A (en) | Hybrid noise removing method and device, electronic equipment and storage medium | |
| CN116433509A (en) | Progressive image defogging method and system based on CNN and convolution LSTM network | |
| CN116523790A (en) | SAR image denoising optimization method, system and storage medium | |
| CN115760638A (en) | End-to-end deblurring super-resolution method based on deep learning | |
| CN113191959B (en) | Digital imaging system limit image quality improving method based on degradation calibration | |
| Shrivastava et al. | Noise removal using first order neighborhood mean filter | |
| CN110648291B (en) | Unmanned aerial vehicle motion blurred image restoration method based on deep learning | |
| CN113066023A (en) | SAR image speckle removing method based on self-calibration convolutional neural network | |
| Murthy et al. | Design and implementation of interpolation algorithms for image super resolution |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |