CN116309110A - Low-light image defogging method based on lightweight deep neural network - Google Patents
Low-light image defogging method based on lightweight deep neural network Download PDFInfo
- Publication number
- CN116309110A CN116309110A CN202310017854.XA CN202310017854A CN116309110A CN 116309110 A CN116309110 A CN 116309110A CN 202310017854 A CN202310017854 A CN 202310017854A CN 116309110 A CN116309110 A CN 116309110A
- Authority
- CN
- China
- Prior art keywords
- layer
- image
- low
- lightweight
- defogging
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Landscapes
- Image Analysis (AREA)
Abstract
The invention discloses a low-light image defogging method based on a lightweight deep neural network, which mainly solves the problems that the extraction of fog image features is difficult, the effective correction of image color cast is lacking in a low-light scene, the defogging network structure is complex, and more resources are occupied. The method comprises the following steps: constructing a training and testing data set containing a synthesized low-light fog chart and a real low-light fog chart; an end-to-end lightweight deep neural network for defogging a low-light image is constructed, and the end-to-end lightweight deep neural network comprises a lightweight multi-level feature fusion sub-module and a lightweight channel attention sub-module; constructing a network target loss function; training the network by using the constructed data set; and inputting the foggy image in the low-light scene into a trained network to obtain a defogged image. On the premise of keeping the contrast of the restored image, the invention corrects the color cast of the image better and restores the details of the image, and has the advantages of less resource occupation, small parameter quantity and low operation quantity.
Description
Technical Field
The invention relates to an image defogging method, in particular to a low-light image defogging method based on a lightweight deep neural network.
Background
Image defogging is an important technology in the field of monitoring security and image enhancement. Under the weather conditions of fog, haze and the like, the visibility is seriously reduced, so that the image acquired by a vision system is seriously degraded. Particularly in the low-illumination haze scenes such as the morning, the evening and the night, the influence of the haze on the imaging quality is further amplified, the image contrast is lower, the details are more fuzzy, and the performance of advanced visual tasks such as target detection, tracking and recognition is affected. Therefore, the defogging of the low-light haze scene image has very important significance and application value.
Existing low-light defogging algorithms are largely divided into two types, namely a traditional method and a deep learning method. At present, the traditional algorithm is mostly adopted, for example, jiang et al propose a low-illumination image defogging method based on a guided image filtering theory, so that the interference of various artificial light sources in a night foggy scene is effectively reduced, and the interference is shown in Jiang B, meng H, maX, et al, nighttime image Dehazing with modified models of color transfer and guided image filter [ J ]. Multimedia tools and applications,2018,77 (3): 3125-3141; yang et al designed a new illumination model, introduced a structural preservation optimization stream based on retinal theory to obtain ambient illumination, and proposed a robust night image defogging stream based on Variation to better solve the non-global atmospheric light problem in night scenes, see c. -h.yang, y. -h.lin and y. -c.lu, "a Variation-Based Nighttime Image Dehazing Flow With aPhysically Valid Illumination Estimator and a Luminance-Guided Coloring Model," in IEEE Access, vol.10, pp.50153-50166,2022, doi:10.1109/access.2022.3173130. However, with the development of deep learning, a large number of experiments show that in the field of image restoration, the method of deep learning has stronger generalization capability and enhancement effect than the traditional method, and is easier to interface with advanced visual tasks (target detection, recognition and segmentation). Typical defogging algorithms under normal illumination have excellent processing results, for example, dong et al propose a multi-scale enhanced defogging network, and combine enhancement strategies with back projection techniques for image defogging, see DONG H, PAN J, XIANG L, et al Multi-scale boosted dehazing network with dense feature fusion [ C ]// Proceedings of the IEEE/CVF conference on computer vision and pattern recognment.2020:2157-2167; chen et al propose gating-based information aggregation networks to remove fog in images and use smooth dilation techniques to repair the grid effect present in the restoration map, see Chen D, HE M, FAN Q, et al, gated context aggregation network for image dehazing and deraining [ C ]//2019IEEE winter conference on applications of computer vision (WACV) & IEEE,2019:1375-1383; qin et al propose a mixed attention based feature fusion network, see QIN X, WANG Z, BAIY, et al FFA-Net Feature fusion attention network for single image dehazing [ C ]// Proceedings of the AAAI Conference on Artificial Intelligent.2020, 34 (07): 11908-11915. These algorithms perform well in normal light defogging tasks, however, there are still shortcomings, mainly manifested in the following two aspects: (1) Under a non-single light source scene at night, such as a yellow street lamp common at night, the existing method lacks effective correction of image color cast, and the image is easy to have serious color cast phenomenon; (2) The model often occupies more memory and has low processing speed, and is often unfavorable for application due to limited computing resources in actual deployment.
Disclosure of Invention
The invention aims to: the invention aims to solve the technical problem of providing a low-light image defogging method based on a lightweight deep neural network, aiming at the defects of the prior art.
In order to solve the technical problems, the invention discloses a low-light image defogging method based on a lightweight deep neural network, which comprises the following steps:
step 2, constructing an end-to-end lightweight depth neural network for defogging a low-light image;
step 3, constructing the target loss function of the end-to-end lightweight depth neural network for defogging the low-light image;
step 4, training the end-to-end lightweight depth neural network for defogging the low-illumination image constructed in the step 2 by adopting the training data set constructed in the step 1;
and 5, inputting the foggy image in the low-light scene into the end-to-end lightweight depth neural network trained in the step 4 and used for defogging the low-light image to obtain a defogged image, and finishing defogging of the low-light image based on the lightweight depth neural network.
The beneficial effects are that:
firstly, the network of the invention has the advantages of less resource occupation, small parameter amount and low operation amount while ensuring good defogging effect due to the introduction of the light-weight multi-stage feature fusion module and the light-weight channel attention module.
Secondly, the network is subjected to joint optimization due to the introduced content loss and color loss, so that the network can better correct the color cast of the image, improve the visual effect, recover the image details and compensate the influence of low light on defogging on the premise of keeping the contrast of the restored image.
Drawings
The foregoing and/or other advantages of the invention will become more apparent from the following detailed description of the invention when taken in conjunction with the accompanying drawings and detailed description.
FIG. 1 is a schematic diagram of the overall process of the present invention.
FIG. 2 is a schematic diagram of a lightweight multi-level feature fusion submodule.
FIG. 3 is a schematic diagram of a lightweight channel attention module.
Fig. 4 is a schematic diagram of a lightweight deep neural network architecture for low-light image defogging.
Fig. 5 is a comparative schematic of the fogging dataset of the present invention.
FIG. 6 is a schematic view showing the effect of the present invention on the treatment of a synthetic mist pattern.
FIG. 7 is a schematic view of the effect of the present invention on the treatment of a real fog pattern.
Detailed Description
The following describes the embodiments and effects of the present invention with reference to the drawings.
As shown in fig. 1, the specific implementation steps of the present invention are as follows:
step 1: a training and test dataset is constructed containing a composite low-light foggy and a true low-light foggy. Two different data sets were selected from the disclosed low-light defogging data set to train and test the network, namely the HDP-Net data set and the 3R data set.
1.1 In order to alleviate or even avoid the artifacts, the algorithm fogging is reused on the basis of the clear image of the HDP data set to obtain a new HDP fogging data set. The invention adopts a method for synthesizing fog images based on image depth images to carry out fog adding, firstly, depth estimation networks proposed by Li and the like are used for estimating depth images D (x) of clear images, see LI B, REN W, FU D, et al, standard imaging single-image dehazing and beyond [ J ]. IEEE Transactions on Image Processing,2018,28 (1): 492-505. Then, with the image depth map known, a new fog-containing data set is constructed in combination with the atmospheric scattering model I (x) =l (x) η (x) R (x) t (x) +l (x) η (x) (1-t (x)).
Wherein x is a pixel index, I (x) is a low-light foggy image, L (x) and η (x) are the illumination intensity and color shift of the light source, R (x) is the reflectivity, t (x) is the haze transmittance, and the global transmittance value t (x) =e is obtained from the depth map d (x) of the image -βd(x) And further obtaining an image after mist adding, wherein beta is 1.3, and the generated mist image is more similar to the real mist image. As shown in fig. 5, wherein the original low-light foggy image is fig. 5 (a) and the new low-light foggy image is fig. 5 (b), the contrast of the image in the HDP dataset and the image in the original dataset after foggy according to the present invention is shown.
1.2 Using the new HDP foggy dataset and the 3R dataset to construct a training dataset containing a composite low-light foggy map and a true low-light foggy map.
Step 2: the method comprises the steps of establishing a lightweight multistage feature fusion module, and using the lightweight multistage feature fusion module in the construction process: convolution layers, see Fukushima, k.,1980.Neocognitron:A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position, biological Cybernetics,36 (4), 193-202; reLu activates the function layer, see R Hahnloser, H.S. Seung (2001), permitted and Forbidden Sets in Symmetric Threshold-Linear networks.NIPS 2001; BN (BatchNorm) normalization layer, see Ioffe S, szegedy C.Batch normalization: accelerating deep network training by reducing internal covariate shift [ J ]. ArXiv preprint arXiv:1502.03167,2015; ghost convolution layers, see HAN K, WANG Y, TIANQ, et al Ghostnet More features from cheap operations [ C ]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recgntion.2020:1580-1589; the shuffle operation, see ZHANG X, ZHOU X, LIN M, et al Shuffenet An extremely efficient convolutional neural network for mobile devices [ C ]// Proceedings of the IEEE conference on computer vision and pattern recognment.2018:6848-6856; sigmoid functions are found in Han, jun; morag, claudio. The influence of the sigmoid function parameters on the speed of backpropagation learning: from Natural to Artificial Neural calculation, 1995:195-201.
as shown in fig. 2, this step is implemented as follows:
2.1 A separate convolution sub-module is established, a convolution layer with the convolution kernel size of 3*1, a ReLu activation function layer and a convolution kernel size of 1*3 with the convolution layer with the step size of 1, a BN normalization layer and the ReLu activation function layer are sequentially overlapped, the characteristics of the output of the overlapped layers and the original input characteristics are spliced in the channel dimension, and the spliced result is sent to the next module in the network.
2.2 A GSC (Ghost Shuffle Convolution Block) submodule is established: and sequentially superposing a convolution layer with the convolution kernel size of 1*1 and the step length of 1, a BN normalization layer and a ReLu activation function layer to obtain a feature a. And inputting the feature a into a Ghost convolution layer with the convolution kernel size of 3*3, sequentially superposing a BN normalization layer and a ReLu activation function layer to obtain a feature b, splicing the feature a and the feature b in a channel dimension, and finally adding a shuffle operation and sending the obtained result to the next module.
2.3 Inputting the lightweight multi-stage feature fusion module into a separation convolution sub-module, connecting an output result with the initial input dimension of the module, and then inputting the output result into a GSC sub-module, wherein the output result is connected with the upper-stage input and the initial input dimension together; the method comprises the steps that an original input feature a is spliced with an input feature a in a channel dimension by separating an output b of a convolution sub-module to obtain a feature c, the feature c is input into a GSC sub-module to obtain a feature d, and the original input feature a, the feature c and the feature d are spliced in the channel dimension to obtain a result e, namely the lightweight multi-stage feature fusion module outputs.
Step 3: establishing a light channel attention module:
as shown in fig. 3, the module includes a packet convolution layer, an SE submodule, a convolution layer, a shuffle layer, wherein:
the grouping convolution layer consists of grouping convolutions with the convolution kernel size of 3*3 and BN normalization hierarchical linkage;
the SE submodule: and sequentially superposing an average pooling layer, a convolution layer with the convolution kernel size of 1*1, a BN normalization layer, a ReLu activation function layer, a convolution layer with the convolution kernel size of 1*1 and a Sigmoid activation function layer, performing pixel-by-pixel multiplication operation on the characteristics output by the superposed layers and the original input characteristics, and sending the operated result to the next module in the network.
The convolution layer consists of convolution with a convolution kernel of 1*1 and BN normalized hierarchical linkage;
the structural relationship among the parts is as follows: and the grouping convolution layer, the SE submodule, the convolution layer and the shuffle layer are sequentially connected in series and then are subjected to residual addition with the original input, and a residual addition result is a module output result.
Step 4: under the open source Pytorch deep learning framework programmed by Facebook corporation using Python language, a lightweight low-light defogging network is built using the modules:
as shown in fig. 4, the light-weight low-light defogging network can be divided into an encoding end and a decoding end, each of which comprises four layers, and the layers are connected through pixel-by-pixel addition;
the first layer of the coding end inputs the original characteristic into a layer of convolution layer to obtain a characteristic a1, inputs the characteristic a1 into two sequentially overlapped light-weight channel attention submodules to output a characteristic b1, and performs pixel-by-pixel addition operation on the characteristic a1 and the characteristic b1 to obtain a characteristic c1, namely the first layer of coding end output. Wherein the convolution layer is filled with 5, the convolution kernel size is 11 x 11, and the step size is 1.
The second layer of the coding end takes the output of the first layer (namely the feature c 1) as input, the feature c1 is input into a convolution layer and a lightweight multi-stage feature fusion submodule to obtain a feature a2, the feature a2 is input into two sequentially overlapped lightweight channel attention submodules to output a feature b2, and the feature a2 and the feature b2 are added pixel by pixel to obtain a feature c2, namely the output of the second layer. Wherein the convolution layer is filled with 1, the convolution kernel size is 3*3, and the step size is 2.
The third and fourth layers of the coding end are the same as the second layer, and outputs c3 and c4 are obtained. Wherein the convolution layer is filled with 1, the convolution kernel size is 3*3, and the step size is 2.
The fourth layer of the decoding end takes the output of the fourth layer of the encoding end (namely the feature c 4) as an input d4, the feature e4 is obtained by sequentially overlapping 11 lightweight channel attention submodules, the feature f4 is obtained by pixel-by-pixel addition operation of the feature d4, and the output g4 of the decoding end is obtained by inputting a lightweight multistage feature fusion submodule.
The input of the third layer of the decoding end is the result of pixel-by-pixel addition of the output c3 of the third layer of the encoding end and the output g4 of the fourth layer of the decoding end, d3 is obtained through a deconvolution layer, the characteristic e4 is obtained through sequential superposition of two light-weight channel attention submodules, the characteristic f3 is obtained through pixel-by-pixel addition operation with the characteristic d3, and the output g3 of the decoding end is obtained through the input light-weight multistage characteristic fusion submodule. Wherein the deconvolution layer is filled with 1, the convolution kernel size is 3*3, and the convolution step size is 2.
The second layer and the first layer of the decoding end also take the output c of the layer and the output g of the upper layer as inputs, and the outputs g2 and g1 of the decoding end are obtained through the same operation as the third layer of the decoding end. Wherein the deconvolution layer is filled with 1, the convolution kernel size is 3*3, and the convolution step size is 2.
The first layer output g1 of the decoding end is followed by a convolution layer with a filling of 1, a convolution kernel size of 3*3 and a convolution step length of 1 to obtain the final output of the whole light low-illumination defogging network.
Step 5: and constructing a network target loss function.
Using L cont Content loss and L color The color Loss performs joint optimization on the network, and the total Loss function Loss is:
Loss＝L cont +L color
the content loss calculation formula is as follows:
wherein n is the number of data, X is the network prediction haze-free image of the method, and Y is the corresponding clear haze-free image.
Color loss uses the object color standard color evaluation formula provided by the international commission on illumination to the color industry: CIEDE2000 color difference formula, see SHARMA G, WU W, DALAL E N.the CIEDE2000 color-difference formula: implementation notes, supplementary test data, and mathematical observations [ J ]].Color Research&Application:Endorsed by Inter-Society Color Council,The Colour Group(Great Britain),Canadian Society for Color,Color Science Association of Japan,Dutch Society for the Study of Color,The Swedish Colour Centre Foundation,Colour Society of Australia,Centre 
de la Coulteur, 2005,30 (1): 21-30. The calculation formula is as follows:
wherein ΔE is 00 Representing the total color difference;
the network predicts the color values of a haze-free image in CIELAB (english: CIELAB color space, also written as l.a.b.l represents perceived brightness, a and b represent four distinct colors of human vision: red, green, blue, yellow) color space (l.a.b.color space), for the present method>
Color values in CIELAB space for the corresponding clear haze-free image; ΔL ′ Representing the brightness difference, ΔC, between the predicted haze-free image and the corresponding clear haze-free image ′ Representing the difference in chroma, deltaH ′ Representing a color difference; k (k) L ,k C ,k H The value of the parameter weight factor is 1; s is S L ,S C ,S H A weight function for correcting color space uniformity; r is R T To correct the rotation function of the blue region of the color space to accommodate the elliptical principal axis direction deflection.
Step 6: and training the network constructed in the step 2 by adopting the data set in the step 1.
6.1 Dividing the training image set into a plurality of batches, wherein each batch is eight paired image groups, and simultaneously inputting the first image group for training to obtain the initial weight W of each convolution operation of the network n And offset value B n Weight D of deconvolution operation n EstimationCorresponding low light haze free image sets.
6.2 Substituting the clear haze-free image set and the estimated corresponding low-light haze-free image into a total loss formula, and calculating a total loss value corresponding to the training image;
6.3 Using an optimizer in the PyTorch architecture for managing and updating the learnable parameter values in the model, an Adam optimizer, updating network parameters, setting the minimum total loss value as a target, setting the initial learning rate to 0.0002, changing the initial learning rate to 0.1 times of the previous learning rate every 100 rounds, and updating the parameters in the low-light defogging network;
6.4 Inputting the second image group into the network after the first parameter updating, and repeating the steps (4 a) to (4 c) to obtain the network after the second parameter updating; and so on until the last group of image groups is input to the network updated last time, obtaining a light-weight low-illumination defogging network after one training;
6.5 All the image groups are sequentially input into a network for completing primary training, and a network after secondary training is obtained; and the like, until all image groups are input 400 times, finishing training of the lightweight low-light defogging network;
step 7: defogging is carried out through a lightweight low-light defogging network:
and (3) inputting the foggy image in the low-light scene into the trained network in the step (6) to obtain a defogged image. And comparing the images before and after defogging, and comparing the indexes such as the running speed, the operand, the parameter quantity and the like of the network.
The effects of the present invention are further described below in conjunction with simulation experiments:
1. test conditions and methods
1. Simulation experiment conditions:
CPU: intel Kuri 9 10900X with a dominant frequency of 3.7Ghz and a memory of 64G;
display card: injevia GeForce RTX3090;
operating system: ubuntu18.04, cuda version 11.3, pytorch version 1.10, python version 3.8.
2. Test pictures: the test dataset constructed in step (1);
3. simulation content and result analysis:
simulation test 1: the effect of testing on the composite dataset using the algorithm of the present invention is shown in fig. 6, where:
FIG. 6a is a diagram of four synthetic fogs;
FIG. 6b is a graph of the synthetic fog of FIG. 6a defogging using the network of the present invention;
FIG. 6c is an unopened sharp image;
from fig. 6, it can be seen that the network processing result of the present invention is close to a true clear image in terms of subjective effect.
Simulation test 2: the effect of testing on a real dataset using the algorithm of the present invention is shown in fig. 7, where:
FIG. 7a is four true low haze plots;
FIG. 7b is a graph of the defogging of the fog chart of FIG. 7a using the network of the present invention;
from fig. 7, it can be seen that the subjective impression of the defogging result through the network of the present invention is clearly seen in terms of subjective effect.
Simulation test 3: the data set is cut into pictures with the uniform size of 512 multiplied by 512 pixels, and indexes such as running speed, operation amount, parameter number and the like of a comparison network are tested by using three existing MSBDNet (multi-scale enhanced defogging network with dense feature fusion), GCANet (gating context aggregation network for image defogging and rain removal) and FFANet (feature fusion attention network for single image defogging).
| Algorithm name | Run time/s | Calculation amount/M | Parameter number/M |
| MSBDNet | 0.037 | 24536 | 28.7 |
| GCANet | 0.021 | 18397 | 0.7 |
| FFANet | 0.015 | 126706 | 2.0 |
| The algorithm of the invention | 0.02 | 3407 | 1.1 |
From the standpoint of resource consumption and operating speed, the network computation is minimal and the parameter is only inferior to GCANet, operating speed is only 5 milliseconds different from FFANet. In combination with the two factors, the algorithm is superior to other comparison algorithms in whole.
According to the method, the image low-light defogging network based on the lightweight network achieves the aims of defogging the low-light scene image, reducing resource occupation and improving operation speed.
In a specific implementation, the application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and the computer program can run the invention content of the low-light image defogging method based on the lightweight deep neural network and part or all of the steps in each embodiment when being executed by the data processing unit. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a random-access memory (random access memory, RAM), or the like.
It will be apparent to those skilled in the art that the technical solutions in the embodiments of the present invention may be implemented by means of a computer program and its corresponding general hardware platform. Based on such understanding, the technical solutions in the embodiments of the present invention may be embodied essentially or in the form of a computer program, i.e. a software product, which may be stored in a storage medium, and include several instructions to cause a device (which may be a personal computer, a server, a single-chip microcomputer, MUU or a network device, etc.) including a data processing unit to perform the methods described in the embodiments or some parts of the embodiments of the present invention.
The invention provides a thought and a method for defogging a low-light image based on a lightweight deep neural network, and particularly the method and the way for realizing the technical scheme are numerous, the above is only a preferred embodiment of the invention, and it should be pointed out that a plurality of improvements and modifications can be made by those skilled in the art without departing from the principle of the invention, and the improvements and the modifications are also regarded as the protection scope of the invention. The components not explicitly described in this embodiment can be implemented by using the prior art.
Claims (10)
1. The low-light image defogging method based on the lightweight deep neural network is characterized by comprising the following steps of:
step 1, constructing a training data set and a test data set which contain a synthesized low-light fog chart and a real low-light fog chart;
step 2, constructing an end-to-end lightweight depth neural network for defogging a low-light image;
step 3, constructing the target loss function of the end-to-end lightweight depth neural network for defogging the low-light image;
step 4, training the end-to-end lightweight depth neural network for defogging the low-illumination image constructed in the step 2 by adopting the training data set constructed in the step 1;
and 5, inputting the foggy image in the low-light scene into the end-to-end lightweight depth neural network trained in the step 4 and used for defogging the low-light image to obtain a defogged image, and finishing defogging of the low-light image based on the lightweight depth neural network.
2. The low-light image defogging method based on the lightweight deep neural network according to claim 1, wherein the synthetic low-light fog chart obtaining method in the step 1 is as follows:
on the basis of the existing clear images in the data set, the fog adding is carried out by adopting a method for synthesizing a fog image based on the depth image of the image, and under the condition of the depth image of the known image, the processing procedure is as follows:
I(x)＝L(x)η(x)R(x)t(x)+L(x)η(x)(1-t(x))
wherein x is a pixel index, I (x) is a low-illumination foggy image, L (x) and eta (x) are illumination intensity and color shift of a light source, R (x) is reflectivity, t (x) is haze transmissivity, and a global transmissivity value t (x) is obtained according to a depth map to obtain a fogged image I (x).
3. The low-light image defogging method based on a lightweight deep neural network according to claim 2, wherein the end-to-end lightweight deep neural network for low-light image defogging described in the step 2 uses an encoder-decoder architecture; the encoder part converts the input into features, and the decoder part converts the features into targets which are expected to be obtained, and the targets obtain defogged images; the encoder-decoder architecture is divided into four layers, and each layer is composed of a convolution layer, a lightweight multi-stage feature fusion module, a lightweight channel attention module and a deconvolution layer; the specific construction method is as follows:
step 2-1, a lightweight multi-level feature fusion module is established;
step 2-2, a lightweight channel attention module is established;
and 2-3, forming the end-to-end lightweight depth neural network for defogging the low-light image.
4. The low-light image defogging method based on the lightweight deep neural network according to claim 3, wherein the specific method for establishing the lightweight multi-level feature fusion module in the step 2-1 is as follows:
step 2-1-1, respectively establishing a separation convolution sub-module and a GSC sub-module;
and 2-1-2, sending the input of the lightweight multi-stage feature fusion module into a separation convolution sub-module, sending an output result into a GSC sub-module after performing dimension connection with the initial input of the lightweight multi-stage feature fusion module, and performing dimension connection with the input of the GSC sub-module and the initial input to form the lightweight multi-stage feature fusion module.
5. The low-light image defogging method based on a lightweight deep neural network according to claim 4, wherein the separating convolution sub-module in the step 2-1-1 comprises the following steps:
the convolution kernel size is 3*1, the convolution layer with the step length of 1, the first ReLu activation function layer, the convolution kernel size is 1*3, the convolution layer with the step length of 1, the BN normalization layer and the second ReLu activation function layer are sequentially overlapped to be used as an overlapped layer, the characteristics of the output of the overlapped layer and the original input characteristics of the separation convolution sub-module are spliced in the channel dimension, and the spliced result is the output of the separation convolution sub-module.
6. The low-light image defogging method based on the lightweight deep neural network according to claim 5, wherein the GSC sub-module in the step 2-1-1 has the following structure:
the convolution kernel is 1*1, and a convolution layer with the step length of 1, a BN normalization layer and a ReLu activation function layer are sequentially overlapped to obtain a feature a; inputting the feature a into a Ghost convolution layer with the convolution kernel size of 3*3, sequentially superposing a BN normalization layer and a ReLu activation function layer to obtain a feature b; and splicing the feature a and the feature b in the channel dimension, and finally adding a shuffle operation to obtain a result which is output by the GSC submodule.
7. The method for defogging a low-light image based on a lightweight deep neural network according to claim 6, wherein the method for establishing a lightweight channel attention module in step 2-2 comprises:
the lightweight channel attention module includes: grouping convolutional layer, SE submodule, convolutional layer, and shuffle layer:
the grouping convolution layer is formed by grouping convolution with a convolution kernel of 3*3 and BN normalization layering;
the SE submodule comprises: sequentially superposing an average pooling layer, a convolution layer with the convolution kernel size of 1*1, a BN normalization layer, a ReLu activation function layer, a convolution layer with the convolution kernel size of 1*1 and a Sigmoid activation function layer, and performing pixel-by-pixel multiplication operation on the characteristics output after superposition and the original input characteristics of the SE submodule to obtain an output of the SE submodule;
the convolution layer is formed by the convolution of a convolution kernel size 1*1 and the hierarchical concatenation of BN normalization;
the structural relation of the light channel attention module is as follows: and after the grouping convolution layer, the SE submodule, the convolution layer and the shuffle layer are sequentially connected in series, carrying out residual addition on the original input of the light-weight channel attention module, and obtaining an output result of the light-weight channel attention module after residual addition.
8. The method for defogging a low-light image based on a lightweight depth neural network according to claim 7, wherein the method for defogging a low-light image comprising the end-to-end lightweight depth neural network according to the step 2-3 comprises the steps of: the method comprises the steps of using a convolution unit, a residual error module, a constructed lightweight multi-stage feature fusion module and a lightweight channel attention module to respectively form an encoder-decoder structure under a Pytorch framework, further forming an end-to-end lightweight deep neural network for defogging a low-light image, wherein the network comprises a four-layer coding and decoding structure, and the layers are connected through pixel-by-pixel addition, and the specific method is as follows:
the first layer of the coding end inputs the original characteristic into a layer of convolution layer to obtain a characteristic a1, inputs the characteristic a1 into two sequentially overlapped light-weight channel attention submodules to output a characteristic b1, and performs pixel-by-pixel addition operation on the characteristic a1 and the characteristic b1 to obtain a characteristic c1, namely the first layer of coding end output; wherein the convolution layer is filled with 5, the convolution kernel size is 11 x 11, and the step length is 1;
the second layer of the coding end takes the output of the first layer, namely the feature c1 as input, the feature c1 is input into a convolution layer and a lightweight multi-stage feature fusion submodule to obtain a feature a2, the feature a2 is input into two sequentially overlapped lightweight channel attention submodules to output a feature b2, and the feature a2 and the feature b2 are added pixel by pixel to obtain a feature c2, namely the output of the second layer; wherein the convolution layer is filled with 1, the convolution kernel size is 3*3, and the step length is 2;
the third layer and the fourth layer of the coding end have the same structure as the second layer, and output is respectively c3 and c4, wherein the convolution layer is filled with 1, the convolution kernel size is 3*3, and the step length is 2;
the fourth layer of the decoding end takes the fourth layer output of the encoding end, namely the characteristic c4 as an input d4, sequentially overlaps the characteristic d4 through 11 light-weight channel attention submodules, performs pixel-by-pixel addition operation on the characteristic d4 to obtain a characteristic f4, and inputs the characteristic f4 into a light-weight multistage characteristic fusion submodule to obtain a decoding end output g4;
the input of the third layer of the decoding end is the result of pixel-by-pixel addition of the output c3 of the third layer of the encoding end and the output g4 of the fourth layer of the decoding end, d3 is obtained through a deconvolution layer, the characteristic e4 is obtained through sequential superposition of two light-weight channel attention submodules, the characteristic f3 is obtained through pixel-by-pixel addition operation of the characteristic d3, the output g3 of the decoding end is obtained through an input light-weight multistage characteristic fusion submodule, the deconvolution layer is filled with 1, the convolution kernel size is 3*3, and the convolution step length is 2;
the second layer and the first layer of the decoding end also take the output c of the layer and the output g of the upper layer as inputs, and the output g2 and the output g1 of the decoding end are respectively obtained through the same operation as the third layer of the decoding end, wherein the deconvolution layer is filled with 1, the convolution kernel size is 3*3, and the convolution step length is 2;
the first layer output g1 of the decoding end is followed by a convolution layer with a filling of 1, a convolution kernel size of 3*3 and a convolution step length of 1 to obtain the final output of the end-to-end lightweight depth neural network for defogging the low-light image.
9. The method for defogging a low-light image based on a lightweight depth neural network according to claim 8, wherein the constructing the objective loss function of the end-to-end lightweight depth neural network for defogging a low-light image in step 3 comprises the following steps:
using content loss L cont And color loss L color And performing joint optimization on the end-to-end lightweight depth neural network for defogging the low-light image, wherein the total Loss function Loss is as follows:
Loss＝L cont +L color
the content loss calculation method is as follows:
wherein n is the number of data, X i Defogging images obtained by predicting foggy images in training data sets for the ith end-to-end lightweight depth neural network defogging by using low-light images, Y i The method comprises the steps that (1) a clear and fogless image is corresponding to an i-th clear and fogless image, wherein the clear and fogless image is the clear image in the step (1);
the color loss is calculated by the following method:
wherein ΔE is 00 For the low-light graphThe total chromatic aberration between the defogging image predicted by the defogging end-to-end lightweight depth neural network and the clear defogging image is calculated;
color values in CIELAB space of the foggy-free image predicted by the end-to-end lightweight depth neural network for defogging the low-light image, +.>
Color values in CIELAB space for the corresponding clear haze-free image; Δl ' represents the brightness difference between the predicted haze-free image and the corresponding clear haze-free image, Δc ' represents the chroma difference, and Δh ' represents the color difference; k (k) L ，k C ，k H The value of the parameter weight factor is 1; s is S L ，S C ，S H A weight function for correcting color space uniformity; r is R T To correct the rotation function of the blue region of the color space to accommodate the elliptical principal axis direction deflection.
10. The method for defogging a low-light image based on a lightweight deep neural network according to claim 9, wherein the training method in step 4 comprises the following steps:
step 4-1, dividing the training data set in step 1 into eight paired image groups in batches, and simultaneously inputting the first image group for training to obtain the initial weight W of each convolution operation in the end-to-end lightweight depth neural network for defogging the low-light image n And offset value B n Weight D of deconvolution operation n And an estimated corresponding low-light haze-free image group;
step 4-1, substituting the clear image in the existing data set and the estimated corresponding low-light haze-free image used in the step 1 when the low-light haze-free image is synthesized into the calculated total loss value corresponding to the image used for training;
step 4-2, updating network parameters by using an Adam optimizer, setting a minimum total loss value as a target, setting an initial learning rate to be 0.0002, and updating parameters in the end-to-end lightweight deep neural network for defogging the low-light image every 100 rounds which become 0.1 times of the previous rounds;
step 4-3, inputting the second image group into the end-to-end lightweight deep neural network for defogging the low-light image after the first parameter update, and repeating the steps 4-1 to 4-3 to obtain a network after the second parameter update; and the like, until the last group of image groups in the batch are input into the network after the previous update, obtaining an end-to-end lightweight depth neural network for defogging the low-illumination images after one training;
step 4-4, sequentially inputting all image groups of the next batch into a network for completing primary training to obtain a network after secondary training; and the like, until all image groups of all batches are input 400 times, the training of the end-to-end lightweight deep neural network for defogging the low-light images is completed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310017854.XA CN116309110A (en) | 2023-01-06 | 2023-01-06 | Low-light image defogging method based on lightweight deep neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310017854.XA CN116309110A (en) | 2023-01-06 | 2023-01-06 | Low-light image defogging method based on lightweight deep neural network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116309110A true CN116309110A (en) | 2023-06-23 |
Family
ID=86796707
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310017854.XA Pending CN116309110A (en) | 2023-01-06 | 2023-01-06 | Low-light image defogging method based on lightweight deep neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116309110A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117237859A (en) * | 2023-11-14 | 2023-12-15 | 南京信息工程大学 | Night expressway foggy day visibility detection method based on low illumination enhancement |
| CN117975173A (en) * | 2024-04-02 | 2024-05-03 | 华侨大学 | Child evil dictionary picture identification method and device based on light-weight visual converter |
-
2023
- 2023-01-06 CN CN202310017854.XA patent/CN116309110A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117237859A (en) * | 2023-11-14 | 2023-12-15 | 南京信息工程大学 | Night expressway foggy day visibility detection method based on low illumination enhancement |
| CN117237859B (en) * | 2023-11-14 | 2024-02-13 | 南京信息工程大学 | Night expressway foggy day visibility detection method based on low illumination enhancement |
| CN117975173A (en) * | 2024-04-02 | 2024-05-03 | 华侨大学 | Child evil dictionary picture identification method and device based on light-weight visual converter |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Golts et al. | Unsupervised single image dehazing using dark channel prior loss | |
| CN110458844B (en) | Semantic segmentation method for low-illumination scene | |
| CN116309110A (en) | Low-light image defogging method based on lightweight deep neural network | |
| CN111784602B (en) | Method for generating countermeasure network for image restoration | |
| CN110288555B (en) | Low-illumination enhancement method based on improved capsule network | |
| CN111292264A (en) | Image high dynamic range reconstruction method based on deep learning | |
| CN109558806A (en) | The detection method and system of high score Remote Sensing Imagery Change | |
| CN112614077A (en) | Unsupervised low-illumination image enhancement method based on generation countermeasure network | |
| CN109255758A (en) | Image enchancing method based on full 1*1 convolutional neural networks | |
| CN112116537B (en) | Image reflected light elimination method and image reflected light elimination network construction method | |
| CN110163884A (en) | A kind of single image dividing method based on full connection deep learning neural network | |
| CN113870124B (en) | Weak supervision-based double-network mutual excitation learning shadow removing method | |
| CN116563693A (en) | Underwater image color restoration method based on lightweight attention mechanism | |
| CN112102186A (en) | Real-time enhancement method for underwater video image | |
| CN116757986A (en) | Infrared and visible light image fusion method and device | |
| CN117115033A (en) | Electric power operation site weak light image enhancement method based on strong light inhibition | |
| Singh et al. | Low-light image enhancement for UAVs with multi-feature fusion deep neural networks | |
| CN113096023A (en) | Neural network training method, image processing method and device, and storage medium | |
| CN117252778A (en) | Color constancy method and system based on semantic preservation | |
| CN111275751A (en) | Unsupervised absolute scale calculation method and system | |
| CN116452472A (en) | Low-illumination image enhancement method based on semantic knowledge guidance | |
| CN113506230B (en) | Photovoltaic power station aerial image dodging processing method based on machine vision | |
| CN115100076A (en) | Low-light image defogging method based on context-aware attention | |
| CN115937048A (en) | Illumination controllable defogging method based on non-supervision layer embedding and vision conversion model | |
| CN116823973B (en) | Black-white video coloring method, black-white video coloring device and computer readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |