CN117011181A - Classification-guided unmanned aerial vehicle imaging dense fog removal method - Google Patents
Classification-guided unmanned aerial vehicle imaging dense fog removal method Download PDFInfo
- Publication number
- CN117011181A CN117011181A CN202311004759.2A CN202311004759A CN117011181A CN 117011181 A CN117011181 A CN 117011181A CN 202311004759 A CN202311004759 A CN 202311004759A CN 117011181 A CN117011181 A CN 117011181A
- Authority
- CN
- China
- Prior art keywords
- image
- fog
- aerial vehicle
- unmanned aerial
- foggy
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 128
- 238000003384 imaging method Methods 0.000 title claims abstract description 23
- 238000000605 extraction Methods 0.000 claims abstract description 20
- 238000013508 migration Methods 0.000 claims abstract description 17
- 230000005012 migration Effects 0.000 claims abstract description 17
- 238000009792 diffusion process Methods 0.000 claims abstract description 10
- 238000012545 processing Methods 0.000 claims abstract description 8
- 230000008569 process Effects 0.000 claims description 18
- 238000012549 training Methods 0.000 claims description 13
- 238000009826 distribution Methods 0.000 claims description 12
- 238000011156 evaluation Methods 0.000 claims description 12
- 238000013441 quality evaluation Methods 0.000 claims description 12
- 238000012360 testing method Methods 0.000 claims description 11
- 230000006870 function Effects 0.000 claims description 9
- 238000005070 sampling Methods 0.000 claims description 9
- 238000011084 recovery Methods 0.000 claims description 6
- 238000002834 transmittance Methods 0.000 claims description 6
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 230000004913 activation Effects 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 3
- 238000010606 normalization Methods 0.000 claims description 3
- 230000008447 perception Effects 0.000 claims description 2
- 230000015572 biosynthetic process Effects 0.000 claims 1
- 238000003786 synthesis reaction Methods 0.000 claims 1
- 230000000694 effects Effects 0.000 abstract description 18
- 239000003595 mist Substances 0.000 description 22
- 238000013135 deep learning Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 6
- 238000003909 pattern recognition Methods 0.000 description 6
- 238000011160 research Methods 0.000 description 4
- 230000000007 visual effect Effects 0.000 description 4
- 239000000443 aerosol Substances 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 239000013589 supplement Substances 0.000 description 3
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Chemical compound O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 3
- 230000002776 aggregation Effects 0.000 description 2
- 238000004220 aggregation Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000015556 catabolic process Effects 0.000 description 2
- 238000006731 degradation reaction Methods 0.000 description 2
- 239000000428 dust Substances 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 239000000203 mixture Substances 0.000 description 2
- 230000000750 progressive effect Effects 0.000 description 2
- 229920002430 Fibre-reinforced plastic Polymers 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000008034 disappearance Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000004880 explosion Methods 0.000 description 1
- 239000011151 fibre-reinforced plastic Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 229920006395 saturated elastomer Polymers 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/40—Analysis of texture
- G06T7/41—Analysis of texture based on statistical description of texture
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/90—Determination of colour characteristics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/54—Extraction of image or video features relating to texture
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/56—Extraction of image or video features relating to colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Software Systems (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Molecular Biology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Image Processing (AREA)
Abstract
The invention provides a classified guided unmanned aerial vehicle imaging dense fog removal method, belongs to the technical field of image processing, and is used for solving the technical problems that an existing defogging method generates overfitting and is poor in defogging effect when being applied to a real scene. The method comprises the following steps: designing a fog diffusion method to generate a unmanned aerial vehicle dense fog data set; designing an image category feature extraction and texture information jump migration structure to realize feature extraction of the unmanned aerial vehicle dense fog image, and obtaining a defogging network; taking the generated unmanned aerial vehicle dense fog image in the unmanned aerial vehicle dense fog data set as input, designing a multi-task loss function to train a defogging network, and obtaining a trained defogging model; and processing the foggy image by using the trained foggy model to obtain a foggy image. The invention can realize the removal of the dense fog image of the unmanned aerial vehicle under the condition of small-scale image input, has better removal effects of the dense fog and the dense fog, and is difficult to generate phenomena such as overexposure, distortion and the like because texture information and tone are restored naturally.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to a classified guided unmanned aerial vehicle imaging dense fog removal method which is used for carrying out defogging treatment on imaging in an unmanned aerial vehicle dense fog environment.
Background
Mist is a natural phenomenon that water vapor saturated in air condenses to form small water drops (mist) and dust (haze), and as the small water drops and dust in the air are optically absorbed and scattered, the imaging quality of an unmanned aerial vehicle is seriously reduced, the visibility is reduced, the subjective visual effect of human eyes is greatly influenced, and the performances of various visual systems are limited. Computer vision tasks such as object detection, object tracking, and region segmentation rely on high quality images with minimal degradation. While unmanned aerial vehicle images are typically captured at higher altitudes, fog is thicker and image degradation is more severe, impeding the development of the computer vision task described above.
Early image prior (manual prior) based methods obtained defogging results by calculating the fog concentration reversal, but due to the instability of the algorithm, inconsistent results are easy to occur when the method is applied to a real scene, so that the generated image is not natural enough. And the method is limited by prior assumptions, if the image is inconsistent with the assumptions, the tone and texture information of the image can be changed, and the high-quality computer vision task can not be met.
Convolutional neural networks have better feature representation capabilities than image prior methods, which benefit from their massive data sets and powerful computational power. Most of the current defogging methods based on deep learning achieve defogging effects by directly acquiring image information, the influence caused by inaccuracy of parameters is reduced, and compared with the image priori method, the method based on deep learning is faster in speed and better in defogging effect. However, the deep learning-based method relies on a large number of paired foggy and non-foggy images, which are difficult to capture in real life at the same time, and which can create significant over-fitting problems when applied to real scenes. The existing defogging method research is mainly focused on a mist defogging method, but the high-altitude mist is unevenly distributed, and the mist imaging captured by an unmanned aerial vehicle is uneven, so that the thickness of the mist in the image is obviously different, and therefore, the conventional defogging method for treating even mist has poor performance in a thick mist imaging environment. Moreover, the lack of the unmanned aerial vehicle dense fog data set currently causes a certain limit on the research of the unmanned aerial vehicle dense fog removal method. By combining the advantages of deep learning and the defects of the existing defogging method, the research on the fog removal method which can generate the foggy diagram with the most authenticity and no distortion and simultaneously retain rich texture information and color tone and the foggy data set of the unmanned aerial vehicle close to the reality is very important to the research.
Disclosure of Invention
Aiming at the problem that the existing defogging method based on deep learning depends on a large number of paired foggy graphs and foggy graphs, obvious overfitting problem can be generated when the defogging method is applied to a real scene, and the defogging performance of the defogging method for processing uniform mist is poor in a thick mist imaging environment, the invention provides a classified guided unmanned aerial vehicle imaging thick mist removing method, which can realize the removal of thick mist images of an unmanned aerial vehicle under the condition of small-scale image input and has better mist and thick mist removing effects.
In order to achieve the above purpose, the technical scheme of the invention is realized as follows: a classification-guided unmanned aerial vehicle imaging dense fog removal method comprises the following steps:
s1, generating a thick fog data set of the unmanned aerial vehicle by designing a fog diffusion method;
s2, designing a defogging network for the foggy image of the unmanned aerial vehicle: designing an image category feature extraction and texture information jump migration structure to realize feature extraction of the unmanned aerial vehicle dense fog image;
s3, training a defogging network: taking the unmanned aerial vehicle dense fog image in the unmanned aerial vehicle dense fog data set generated in the step S1 as input of a defogging network, designing a multi-task loss function to train the defogging network, and obtaining a final trained defogging model;
and S4, defogging the foggy image by using the defogging model in the step S3 to obtain a defogged image.
Preferably, the three channels of RGB of the image in the VisDRone2019 unmanned aerial vehicle data set are subjected to fog adding processing, so that a multi-category unmanned aerial vehicle thick fog data set with different thick fog thicknesses is generated; the generation method comprises the following steps: designating brightness and setting fixed fog concentration value or randomly initializing brightness and fog concentration value, adjusting selected position and determining fog size, and adding fog to RGB three channels of image.
Preferably, the fog diffusion method starts to diffuse and synthesize from the center point of fog, and the farther from the center of fog, the weaker the fog concentration; and:
zn＝e (-beta*n)
H fog ＝img f[u][v][:] zn+λ(1-zn)
where u and v represent the length and width of the image, respectively, n represents the distance from the current pixel to the center pixel, Q is the set fogging size, ce []Represents the position of the center point, ce [ 0]]And ce [1]]Representing the row and the ordinate, zn represents the transmittance, beta is a parameter controlling the rate of decay of the transmittance, lambda represents the luminance, img f[u][v][:] Representing the size and the sum of dimensions of the normalized imageNumber of channels, H fog The pixel data representing the acquired image, i.e., RGB values.
Preferably, the unmanned aerial vehicle dense fog dataset comprises 1559 paired fog patterns and no fog patterns, 76 unpaired fog patterns; by setting random parameters of the density, the image is classified into three kinds of haze, thick haze and lump haze according to the haze density and the haze size in the haze image generation process.
Preferably, the defogging network is trained using the foggy map of the unmanned aerial vehicle foggy data set and the corresponding real foggy-free map as training data sets, the unpaired synthetic foggy map and the captured real foggy map as test data sets.
Preferably, the defogging network comprises an image category feature extraction structure and a texture information jump migration structure, wherein the image category feature extraction structure uses a residual error linear module as a basis, and a feature map A1 containing relevant category feature information is obtained through normalization operation and linear operation;
the feature extraction method of the texture information jump migration structure comprises the following steps: carrying out reflection filling on the feature map A1 to obtain more comprehensive image features to obtain a feature map B1; performing convolution operation with step length of 1 and convolution kernel of 3×3 on the feature map B1 to obtain a feature map B2; performing downsampling operation on the feature map B2 to obtain a feature map B3 with the original half image resolution and 2 times channel number; performing residual migration operation on the feature map B3 for a plurality of times to obtain a feature map B4; performing up-sampling operation on the feature map B4 to obtain a feature map B5 of the original input image with image resolution and channel number recovery; performing convolution operation with the convolution kernel of 3×3 on the feature map B5 to obtain a feature map B6; performing jump connection operation on the feature map B6 and the feature map B2, and then performing reflection filling and convolution operation with a convolution kernel of 3 multiplied by 3 to obtain a feature map C1; the feature map C2 is obtained by performing a convolution operation with a convolution kernel of 3×3 and a Tanh activation function on the feature map C1.
Preferably, the reflection filling operation adds reflection symmetric filling to the input data along all axes, and the input data is filled with the same size up and down and left and right, and for a given convolution kernel size K and a filling number P, the R value corresponding to the mth element in the input data is:
r=input data [ m-k+2pl ]
Wherein L is 0 or 1, r represents a filled value;
the jump connection operation is long jump connection, and the output of the first layer and the output of the subsequent layer are added at the pixel level and then output;
the residual migration operation uses a convolution layer to carry out dimension matching and adjustment, and the space size of input and output is kept consistent; the output in the process is subjected to pixel-level addition operation through multiple convolution and jump connection operations, so that the generator can better learn the mapping relation between the target domain and the source domain.
Preferably, the multitasking loss function is:
wherein K is the total number of categories, N is the total number of samples, g i,j The true class for the ith sample is j, f i,j For the probability of the ith sample being misclassified into the j category, A represents the foggy image domain, B represents the foggy image domain, x represents the foggy image, y represents the foggy image, P data(x) Representing the distribution of image data from domain A, P data(y) Representing the distribution of image data from domain B,representing the slave image data distribution P data(x) Sample x, obtained by sampling>Representing the slave image data distribution P data(y) Sample y, I obtained by middle sampling 1 Representing the L1 norm for calculating the difference between the original image and the converted image, generator G AB Converting fog patterns into haze-free patterns, G AB (x) Representing fog x through generator G AB The resulting haze-free image, D B Indicating whether the inputted image is an defogging image y from the defogging image domain B or a generated defogging image G AB (x) A kind of electronic deviceDiscriminator, generator G BA Representing conversion of haze-free image into haze image, G BA (y) represents the haze-free image y passing through the generator G BA The generated hazy image D A Indicating whether the inputted image is a foggy image x from the foggy image domain a or a generated foggy image G BA A discriminator of (y).
Preferably, the non-paired synthetic thick fog image and the captured real thick fog image are selected from the generated unmanned aerial vehicle thick fog data set and the HSTS data set as test data sets, and the generated defogging image is evaluated by using a no-reference image quality evaluation method and a full-reference image quality evaluation method.
Preferably, the evaluation index of the reference-free image quality evaluation method is time; the evaluation indexes of the full-reference image quality evaluation method comprise peak signal-to-noise ratio, structural similarity and learning perception image block similarity.
Compared with the prior art, the invention has the beneficial effects that: performing mist adding processing on the clear image of the unmanned aerial vehicle by using a mist diffusion method to generate a concentrated mist data set of the unmanned aerial vehicle, wherein the concentrated mist data set of the unmanned aerial vehicle comprises a generated synthesized concentrated mist image and a photographed real concentrated mist image, and is used as a training data set and a test data set; feature extraction is achieved through design of image category feature extraction and texture information jump migration structure, so that defogging of the unmanned aerial vehicle dense fog image is achieved, phenomena of overexposure, distortion, chromatic aberration, defogging, uncleanness and the like of the defogged image are not easy to occur, and the defogged image contains rich texture information and good visual effect. Compared with the prior image priori or deep learning defogging method, the defogging method can realize the removal of the unmanned aerial vehicle foggy image under the condition of small-scale image input, has better foggy and foggy removal effects, and is difficult to generate phenomena of overexposure, distortion and the like because texture information and tone are restored naturally.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of the overall process of the present invention.
Fig. 2 is a schematic structural diagram of the defogging network according to the present invention.
FIG. 3 is a graph of a random example of an aerosol dataset of the present invention, wherein (a) is a partial non-aerosol plot and (b) is a corresponding aerosol plot.
FIG. 4 is a graph comparing the pair-wise data in the fogging dataset according to the present invention based on the embodiment shown in FIG. 3 with the defogging results of the conventional method, wherein (a) is a fog chart, (b) is a defogging chart obtained by the method of the document [1], (c) is a defogging chart obtained by the method of the document [2], (d) is a defogging chart obtained by the method of the document [3], (e) is a defogging chart obtained by the method of the document [4], (f) is a defogging chart obtained by the method of the document [5], (g) is a defogging chart obtained by the method of the document [6], (h) is a defogging chart obtained by the method of the present invention, and (i) is a true chart corresponding to (a).
FIG. 5 is a graph comparing the defogging results of the method according to the present invention based on the published data set HSTS with those of the prior art, wherein, (a) is a defogging graph, (b) is a defogging graph obtained by the method according to the document [1], (c) is a defogging graph obtained by the method according to the document [2], (d) is a defogging graph obtained by the method according to the document [3], (e) is a defogging graph obtained by the method according to the document [4], (f) is a defogging graph obtained by the method according to the document [5], (g) is a defogging graph obtained by the method according to the document [6], (h) is a defogging graph according to the method according to the present invention, and (i) is a true graph corresponding to (a).
FIG. 6 is a graph comparing the defogging results of the prior art with the defogging results of the data set corresponding to FIG. 3, wherein, (a) is a fog chart, (b) is a defogging chart obtained by the method of the document [1], (c) is a defogging chart obtained by the method of the document [2], (d) is a defogging chart obtained by the method of the document [3], (e) is a defogging chart obtained by the method of the document [4], (f) is a defogging chart obtained by the method of the document [5], (g) is a defogging chart obtained by the method of the document [6], (h) is a defogging chart obtained by the method of the invention;
FIG. 7 is a graph showing the defogging effect of the method of the present invention on different concentration mist patterns.
Wherein, document [1] is [ Zhang, h., and v.m. patel, densely Connected Pyramid Dehazing network.in IEEE Conference on Computer Vision and Pattern Recognition,2018 ]; document [2] is [ Yang, y., C.Wang, R.Liu, L.Zhang, X.Guo, and d.tao, self-augmented Unpaired Image Dehazing via Density and Depth composition. In IEEE Conference on Computer Vision and Pattern Recognition,2022]; document [3] is [ Song, y., z.he, h.qian, and x.du, vision Transformers for Single Image dehazing.ieee trans.image Process,2023]; document [4] is [ Das, s.d., and s.dutta, fast Deep Multi-Patch Hierarchical Network for Nonhomogeneous Image dehazing.in IEEE Conference on Computer Vision and Pattern Recognition Workshops,2020]; document [5] is [ Chen, d., M.He, Q.Fan, J.Liao, L.Zhang, and d.hou, gated Context Aggregation Network for Image Dehazing and derailing, in IEEE Winter Conference on Applications of Computer Vision,2019]; document [6] is [ Mei, k., a.jiang, and m.wang, progressive Feature Fusion Network for Realistic Image dehazing.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without any inventive effort, are intended to be within the scope of the invention.
As shown in fig. 1, the embodiment of the invention provides a classification-guided unmanned aerial vehicle imaging dense fog removal method, which comprises the following specific steps:
s1, generating a thick fog data set of the unmanned aerial vehicle by designing a fog diffusion method, and using a pair of thick fog images and corresponding real non-fog images in the thick fog data set as training data sets, and using a non-pair synthesized thick fog image and a captured real thick fog image as test data sets.
Because the prior art lacks unmanned aerial vehicle imaging dense fog data sets, on the basis of the Visdrone2019 unmanned aerial vehicle data sets, the multi-category unmanned aerial vehicle dense fog data sets with different dense fog thicknesses are obtained by carrying out fog adding processing on RGB three channels of images. The fog is added to the RGB three channels of the image by specifying the brightness and setting a fixed fog concentration value or randomly initializing the above values, then adjusting the selected position and determining the size of the fog. The fog diffusion method of the invention is synthesized by starting diffusion from the center point of fog, and the farther the fog is away from the center of fog, the weaker the fog concentration is. The process of the mist diffusing method is described by the following formula:
zn＝e (-beta*n)
H fog ＝img f[u][v][:] zn+λ(1-zn)
where u and v represent the length and width of the image, respectively, n represents the distance from the current pixel to the center pixel, Q is the set fogging size, ce []Represents the position of the center point, ce [ 0]]And ce [1]]Representing the row and the ordinate, zn represents the transmittance, beta is a parameter controlling the rate of decay of the transmittance, lambda represents the luminance, img f[u][v][:] Representing the size and the channel number of the normalized image, H fog The pixel data representing the acquired image, i.e., RGB values.
The unmanned aerial vehicle dense fog dataset contains 1559 pairs of fog patterns and no fog patterns, 76 unpaired fog patterns. By setting random parameters of the density, the images are classified into three types of fog, thick fog and cluster fog according to fog density and fog size in the fog image generation process, as shown in fig. 3, the 1 st, 3 rd and 5 th images in (b) are thick fog, and the 2 nd and 4 th images are thin fog according to fig. 3. The resulting dense fog dataset, captured real dense fog image and HSTS dataset at the perspective of the drone are then used as training dataset and test dataset, respectively.
S2, designing a defogging network structure for a foggy image of the unmanned aerial vehicle: and designing an image category feature extraction and texture information jump migration structure to realize feature extraction.
In the image category characteristic extraction structure, a residual error linear module is used as a basis, so that the process of category characteristic extraction can be better converged, the risk of overfitting is reduced, more useful characteristic information is obtained, and the image category characteristic extraction structure obtains the related category characteristic information of the characteristic diagram A through normalization operation and linear operation.
Because the deep neural network has a deep network architecture, as the depth of the network structure is continuously increased, deep characteristic information of a clear image is acquired, but shallow characteristic information is lost, so that when the fog image restores the clear image, information such as detail textures and the like is restored to a certain extent. Therefore, the image category feature extraction and texture information jump migration structure designed by the invention can more perfectly supplement various texture information and color information of the generated image, mutually supplement shallow layer feature information and deep layer feature information, and perfect supplement of the texture information and the color information. The texture information jump migration structure improves the image resolution by carrying out reflection filling on the feature map A, and obtains more comprehensive image features to obtain a feature map B1; the feature map B2 is obtained by carrying out convolution operation of step length 1 and convolution kernel 3 multiplied by 3 on the feature map B1; the method is also used for carrying out downsampling operation on the feature map B2 to obtain a feature map B3 with the original half image resolution and 2 times channel number; performing residual migration operation on the feature map B3 for a plurality of times to obtain a feature map B4; the method is also used for carrying out up-sampling operation on the characteristic diagram B4 to obtain a characteristic diagram B5 of an original input image with image resolution and channel number recovery; performing convolution operation with the convolution kernel of 3×3 on the feature map B5 to obtain a feature map B6; performing jump connection operation on the feature map B6 and the feature map B2, and then performing convolution operation with reflection filling and convolution kernel of 3×3 to obtain a feature map C1, so as to solve the problems of gradient explosion and gradient disappearance in the training process; and performing convolution operation with the convolution kernel of 3 multiplied by 3 on the feature map C1 and a Tanh activation function to obtain a feature map C2, so that the image feature information is better reserved.
The reflection filling operation is to add a reflection symmetric filling around the input data along all axes, i.e. for a given convolution kernel size K and filling number P, the R value corresponding to the mth element in the input data is expressed as follows:
r=input data [ m-k+2pl ]
Wherein L is 0 or 1, R represents a filled value, and the operation can effectively enlarge the boundary of input data, reduce the occurrence of block artifacts in a generated graph, keep information complete and acquire more comprehensive image features.
The jump connection operation is long jump connection, and the output of the first layer and the output of the subsequent layer are added and output at the pixel level so as to transfer more complete information to the next layer.
The residual migration operation uses a convolution layer to perform dimension matching and adjustment, and the space size of input and output is kept consistent. The mapping relation between the target domain and the source domain can be better learned by the generator through the addition operation of the pixel level of the output in the process through the convolution and jump connection operation for a plurality of times. The operation can capture more image details and features in the process of learning the defogging network on the defogging image, and can add the features of the input image and the converted image features to ensure that the converted image has consistency in content with the original image.
S3, training a defogging network: taking the unmanned aerial vehicle dense fog image in the unmanned aerial vehicle dense fog data set generated in the step S1 as input of a defogging network, designing a multi-task loss function to train the defogging network, and obtaining a final trained defogging model.
And (2) training a defogging network by using the foggy image and the corresponding clear image as training data sets, wherein in the training process, the foggy image and the corresponding clear image of the unmanned aerial vehicle generated in the step (S1) and a real data set HSTS from the RESIDE data set are respectively input into the defogging network.
Designing a multitasking loss function L to optimize the whole network, wherein the following formula is as follows:
wherein K is a categoryTotal number, N is the total number of samples, g i,j The true class for the ith sample is j, f i,j For the probability of the ith sample being misclassified into the j category, A represents the foggy image domain, B represents the foggy image domain, x represents the foggy image, y represents the foggy image, P data(x) Representing the distribution of image data from domain A, P data(y) Representing the distribution of image data from domain B,representing the slave image data distribution P data(x) Sample x, obtained by sampling>Representing the slave image data distribution P data(y) Sample y, I obtained by middle sampling 1 Representing the L1 norm for calculating the difference between the original image and the converted image, generator G AB Can convert fog patterns into non-fog patterns, and x passes through a generator G AB Can be expressed as G after conversion AB (x) Representing the generated haze-free image; distinguishing device D B For determining whether the image input to the discriminator is from y of the haze-free image field B or the generated haze-free image G AB (x) A. The invention relates to a method for producing a fibre-reinforced plastic composite Generator G BA The reverse process is indicated, i.e. converting the haze free image into a hazy image, the haze free image y passing through the generator G BA Expressed as G after conversion BA (y) the resulting hazy image; distinguishing device D A For determining whether the image inputted to the discriminator is a foggy image x from the domain foggy image domain a or a generated foggy image G BA (y). Generator G BA During the training process, the generated G is also generated AB (x) Image reconversion to F AB (x) I.e. the reconstructed graph of x, generator G AB Will G BA (y) image reconversion to F BA (y), i.e., a reconstructed graph of y, the process does not require the use of additional new generators, reducing consumption.
And S4, defogging the foggy image by using the defogging model in the step S3 to obtain a defogging image, and evaluating the defogging image.
To further test the feasibility and effectiveness of the method of the invention, experiments were performed. The invention utilizes the non-reference image quality evaluation method and the full-reference image quality evaluation method to evaluate the experimental results of the method and the existing image defogging method on the generated unmanned aerial vehicle dense fog data set and the public data set HSTS.
The evaluation index of the reference-free image quality evaluation method is time. The evaluation indexes of the full-reference image quality evaluation method comprise peak signal-to-noise ratio (PSNR), structural Similarity (SSIM) and learning perceived image block similarity (LPIPS). The PSNR is used for comparing the required signal intensity with the intensity of background noise, and the larger the value is, the smaller the image noise is, and the higher the image quality is; the SSIM reflects the similarity between two images, and the higher the SSIM value (the range is 0-1), the more similar the two images are; the LPIPS reflects the difference between the two images, the lower the LPIPS value, indicating that the two images are more similar.
The existing image defogging method comprises the following steps: document [1] - [ Zhang, h., and v.m. Patel, densely Connected Pyramid Dehazing network.in IEEE Conference on Computer Vision and Pattern Recognition,2018 ] ], document [2] - [ Yang, y., C.Wang, R.Liu, L.Zhang, X.Guo, and d.tao, self-augmented Unpaired Image Dehazing via Density and Depth De-composition.in IEEE Conference on Computer Vision and Pattern Recognition,2022], document [3] - [ Song, y., z.he, h.qian, and x.du, vision Transformers for Single Image dehazing.ieee trans.image Process,2023], document [4] - [ Das, s.d., and s.dutta, fast Deep Multi-Patch Hierarchical Network for Nonhomogeneous Image dehazin.in IEEE Conference on Computer Vision and Pattern Recognition Workshops,2020], document [5] - [ Chen, d., M.He, Q.Fan, J.Liao, L.Zhang, and d.hou, gated Context Aggregation Network for Image Dehazing and Decingin IEEE Winter Conference on Applications of Computer Vision,2019], document [6] - [ Mei, k., a.jing, m.waia, progressive Feature Fusion Network for Realistic Image Dehang.8, computer.
Tables 1 and 2 show PSNR, SSIM, LPIPS of defogging-post-defogging no-fog graph and time evaluation index values obtained by using the experimental results of the method of the present invention and the existing image defogging method on the unmanned aerial vehicle thick fog dataset and the published HSTS dataset, respectively.
Table 1 the data set provided by the invention is different in defogging method defogging result evaluation index value
Table 2 HSTS data sets different defogging methods defogging result evaluation index values
From the data presented in table 1, it can be seen that the time evaluation index of the haze-free image obtained by the method of the present invention is slow compared to other algorithms, but is sufficient for fluency, and the present invention is superior to other methods in other image evaluation indexes PSMR, SSIM, and LPIPS. As can be seen from the data listed in table 2, the SSIM evaluation index value of the document [3] is higher than that of the method of the present invention, but as can be seen from the fourth line image of fig. 5, the document [3] has a distortion problem for the texture information recovery of the image. This shows that the defogging picture obtained by the method has higher image quality, and is closest to the real picture whether the image texture information or the overall color tone is restored.
As shown in fig. 4, the method of the present invention uses the image defogging method to test the thick fog image of the unmanned aerial vehicle under the visual angle on the data set provided by the method of the present invention, (a) is the fog image, (b) is the defogging image obtained by the method of the document [1], (c) is the defogging image obtained by the method of the document [2], (d) is the defogging image obtained by the method of the document [3], (e) is the defogging image obtained by the method of the document [4], (f) is the defogging image obtained by the method of the document [5], (g) is the defogging image obtained by the method of the present invention, (h) is the true image corresponding to (a), and the blue area at the right lower corner in the image is the enlarged display effect of the red area in the image. From the above experimental results, it can be seen that document [6] cannot effectively defogging and change the overall color of the image, not only the brightness of the image is greatly reduced, but also the image information cannot be clearly distinguished, and whether the image is daytime or night cannot be judged; document [2] does not change the overall color tone, but the defogging effect is not good, causing overexposure in bright areas in the graph; document [1] has a limited defogging effect only on a region where fog is thin, i.e., an image edge region; document [4] turns the color tone of an image to be white as a whole, and does not maintain tone information well; the defogging effect of the document [3] and the document [5] is relatively good, but is significantly different from the color of the real image, and in the seventh line of fig. 4, the document [3] darkens the entire image with the appearance of a block-like region, and the defogging effect of the document [5] on the middle region of the image is not good. The method of the invention overcomes the problems in the defogging results, so that the defogging results are defogged and clean, and better results are obtained in the aspects of recovering the image tone and the texture information details.
As shown in fig. 5, the present invention uses the image defogging method to test the fog pattern of the unmanned aerial vehicle under the view angle on the disclosed HSTS data set. As can be seen from the first and third lines of fig. 5, the document [4] and the document [6] make the color of the sky completely different from that of the real image, the document [2] darkens the entire tone of the image, and it can be seen that the lake in the figure cannot reflect the scene, the defogging effect of the document [1] is not good, the defogging effect of the document [5] is relatively good, but the color of the sky is lightened as a whole; in the fourth line image, the document [3] and the document [5] exaggerate the color of the sky. The method of the invention is closest to the real haze-free image in the aspects of image texture information, hue, defogging and the like, and has the best recovery effect.
As shown in fig. 6, the image defogging method is used for testing unpaired fog patterns and fog patterns captured in real life in a data set provided by the image defogging method, and the blue region at the lower right corner of the image is an enlarged display effect of the red region. As can be seen from fig. 6, all of the documents [3], [4] and [6] change the overall color of the image, and the documents [1], [2] and [5] have poor defogging effect on the sky in the first three lines of images. The method of the invention has the cleanest defogging and the truest and natural image tone recovery.
As shown in fig. 7, the present invention was tested on a fog chart of different concentrations in the data set provided by the method of the present invention. The results of fig. 7 show that, on fog patterns with different concentrations and different visibility, the defogging results of the method of the invention are similar to those of a real non-fog pattern, and good results are obtained in terms of image color and detail reduction.
In summary, the method reduces the loss of image details in the process of removing the dense fog, and can restore natural and real color characteristics and tone information. Compared with the previous defogging method, the defogging method can realize a high-quality defogging effect in an unmanned aerial vehicle aerial photographing environment, and phenomena such as overexposure, distortion and the like are not easy to generate.
The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, alternatives, and improvements that fall within the spirit and scope of the invention.
Claims (10)
1. The unmanned aerial vehicle imaging dense fog removing method guided by classification is characterized by comprising the following steps of:
s1, generating a thick fog data set of the unmanned aerial vehicle by designing a fog diffusion method;
s2, designing a defogging network for the foggy image of the unmanned aerial vehicle: designing an image category feature extraction and texture information jump migration structure to realize feature extraction of the unmanned aerial vehicle dense fog image;
s3, training a defogging network: taking the unmanned aerial vehicle dense fog image in the unmanned aerial vehicle dense fog data set generated in the step S1 as input of a defogging network, designing a multi-task loss function to train the defogging network, and obtaining a final trained defogging model;
and S4, defogging the foggy image by using the defogging model in the step S3 to obtain a defogged image.
2. The classification-guided unmanned aerial vehicle imaging dense fog removal method of claim 1, wherein the three channels of RGB of the images in the visclone 2019 unmanned aerial vehicle dataset are subjected to fog adding processing to generate a multi-category unmanned aerial vehicle dense fog dataset with different dense fog thicknesses; the generation method comprises the following steps: designating brightness and setting fixed fog concentration value or randomly initializing brightness and fog concentration value, adjusting selected position and determining fog size, and adding fog to RGB three channels of image.
3. The classification-guided unmanned aerial vehicle imaging dense fog removal method according to claim 2, wherein the fog diffusion method starts diffusion synthesis from a center point of fog, and the farther from the fog center, the weaker the fog concentration; and:
zn＝e (-beta*n)
H fog ＝img f[u][v][:] zn+λ(1-zn)
where u and v represent the length and width of the image, respectively, n represents the distance from the current pixel to the center pixel, Q is the set fogging size, ce []Represents the position of the center point, ce [ 0]]And ce [1]]Representing the row and the ordinate, zn represents the transmittance, beta is a parameter controlling the rate of decay of the transmittance, lambda represents the luminance, img f[u][v][:] Representing the size and the channel number of the normalized image, H fog The pixel data representing the acquired image, i.e., RGB values.
4. The class-guided unmanned aerial vehicle imaging dense fog removal method of claim 3, wherein the unmanned aerial vehicle dense fog dataset comprises 1559 pairs of fog patterns and no fog patterns, 76 non-pairs of fog patterns; by setting random parameters of the density, the image is classified into three kinds of haze, thick haze and lump haze according to the haze density and the haze size in the haze image generation process.
5. The class-guided unmanned aerial vehicle imaging foggy removal method of any of claims 1 to 4, wherein the foggy map of the unmanned aerial vehicle foggy dataset and the corresponding real foggy-free map are used as training data sets to train the defogging network, the unpaired synthetic foggy map and the captured real foggy map are used as test data sets.
6. The classification-guided unmanned aerial vehicle imaging dense fog removal method according to claim 5, wherein the defogging network comprises an image category feature extraction structure and a texture information jump migration structure, and the image category feature extraction structure obtains a feature map A1 containing relevant category feature information through normalization operation and linear operation by using a residual error linear module as a basis;
the feature extraction method of the texture information jump migration structure comprises the following steps: carrying out reflection filling on the feature map A1 to obtain more comprehensive image features to obtain a feature map B1; performing convolution operation with step length of 1 and convolution kernel of 3×3 on the feature map B1 to obtain a feature map B2; performing downsampling operation on the feature map B2 to obtain a feature map B3 with the original half image resolution and 2 times channel number; performing residual migration operation on the feature map B3 for a plurality of times to obtain a feature map B4; performing up-sampling operation on the feature map B4 to obtain a feature map B5 of the original input image with image resolution and channel number recovery; performing convolution operation with the convolution kernel of 3×3 on the feature map B5 to obtain a feature map B6; performing jump connection operation on the feature map B6 and the feature map B2, and then performing reflection filling and convolution operation with a convolution kernel of 3 multiplied by 3 to obtain a feature map C1; the feature map C2 is obtained by performing a convolution operation with a convolution kernel of 3×3 and a Tanh activation function on the feature map C1.
7. The classification-guided unmanned aerial vehicle imaging dense fog removal method of claim 6, wherein the reflection filling operation is to add reflection-symmetric filling to the input data along all axes and fill the input data up, down, left and right with the same size, and for a given convolution kernel size K and filling number P, the R value corresponding to the mth element in the input data is:
r=input data [ m-k+2pl ]
Wherein L is 0 or 1, r represents a filled value;
the jump connection operation is long jump connection, and the output of the first layer and the output of the subsequent layer are added at the pixel level and then output;
the residual migration operation uses a convolution layer to carry out dimension matching and adjustment, and the space size of input and output is kept consistent; the output in the process is subjected to pixel-level addition operation through multiple convolution and jump connection operations, so that the generator can better learn the mapping relation between the target domain and the source domain.
8. The class-guided unmanned aerial vehicle imaging fog removal method of claim 1 or 6, wherein the multitasking loss function is:
wherein K is the total number of categories, N is the total number of samples, g i,j The true class for the ith sample is j, f i,j For the probability of the ith sample being misclassified into the j category, A represents the foggy image domain, B represents the foggy image domain, x represents the foggy image, y represents the foggy image, P data(x) Representing the distribution of image data from domain A, P data(y) Representing the distribution of image data from domain B,representing the slave image data distribution P data(x) Sample x, obtained by sampling>Representing the slave image data distribution P data(y) Sample y, I obtained by middle sampling 1 Representing the L1 norm for calculating the difference between the original image and the converted image, generator G AB Converting fog patterns into haze-free patterns, G AB (x) Representing fog x through generator G AB The resulting haze-free image, D B Indicating whether the inputted image is to be judgedThe foggy image y from the foggy image field B is also the generated foggy image G AB (x) Is a discriminator of (1), generator G BA Representing conversion of haze-free image into haze image, G BA (y) represents the haze-free image y passing through the generator G BA The generated hazy image D A Indicating whether the inputted image is a foggy image x from the foggy image domain a or a generated foggy image G BA A discriminator of (y).
9. The class-guided unmanned aerial vehicle imaging foggy removal method of claim 8, wherein the generated defogging image is evaluated using a no-reference image quality evaluation method and a full-reference image quality evaluation method by selecting a non-paired synthesized foggy map and a captured real foggy map as test data sets on the generated unmanned aerial vehicle foggy data set and the HSTS data set.
10. The classification-guided unmanned aerial vehicle imaging dense fog removal method according to claim 9, wherein the evaluation index of the no-reference image quality evaluation method is time; the evaluation indexes of the full-reference image quality evaluation method comprise peak signal-to-noise ratio, structural similarity and learning perception image block similarity.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311004759.2A CN117011181A (en) | 2023-08-10 | 2023-08-10 | Classification-guided unmanned aerial vehicle imaging dense fog removal method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311004759.2A CN117011181A (en) | 2023-08-10 | 2023-08-10 | Classification-guided unmanned aerial vehicle imaging dense fog removal method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117011181A true CN117011181A (en) | 2023-11-07 |
Family
ID=88565239
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311004759.2A Pending CN117011181A (en) | 2023-08-10 | 2023-08-10 | Classification-guided unmanned aerial vehicle imaging dense fog removal method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117011181A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117649439A (en) * | 2024-01-30 | 2024-03-05 | 山东省海洋资源与环境研究院(山东省海洋环境监测中心、山东省水产品质量检验中心) | Sea weed bed area acquisition method, system, equipment and storage medium |
-
2023
- 2023-08-10 CN CN202311004759.2A patent/CN117011181A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117649439A (en) * | 2024-01-30 | 2024-03-05 | 山东省海洋资源与环境研究院(山东省海洋环境监测中心、山东省水产品质量检验中心) | Sea weed bed area acquisition method, system, equipment and storage medium |
| CN117649439B (en) * | 2024-01-30 | 2024-04-09 | 山东省海洋资源与环境研究院(山东省海洋环境监测中心、山东省水产品质量检验中心) | Sea weed bed area acquisition method, system, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Li et al. | Low-light image enhancement via progressive-recursive network | |
| Rudnev et al. | Nerf for outdoor scene relighting | |
| CN113658057B (en) | Swin converter low-light-level image enhancement method | |
| CN110225260B (en) | Three-dimensional high dynamic range imaging method based on generation countermeasure network | |
| CN113822830B (en) | Multi-exposure image fusion method based on depth perception enhancement | |
| CN115641391A (en) | Infrared image colorizing method based on dense residual error and double-flow attention | |
| CN116033279B (en) | Near infrared image colorization method, system and equipment for night monitoring camera | |
| CN117011181A (en) | Classification-guided unmanned aerial vehicle imaging dense fog removal method | |
| Zhang et al. | A real-time semi-supervised deep tone mapping network | |
| CN116645569A (en) | Infrared image colorization method and system based on generation countermeasure network | |
| CN112419163A (en) | Single image weak supervision defogging method based on priori knowledge and deep learning | |
| CN113034417A (en) | Image enhancement system and image enhancement method based on generation countermeasure network | |
| Ali et al. | LIDN: a novel light invariant image dehazing network | |
| Rao et al. | Seeing in the Dark by Component-GAN | |
| CN115564676A (en) | Underwater image enhancement method and system and readable storage medium | |
| CN117689550A (en) | Low-light image enhancement method and device based on progressive generation countermeasure network | |
| Kumar et al. | Underwater image enhancement using deep learning | |
| Hou et al. | Reconstructing a high dynamic range image with a deeply unsupervised fusion model | |
| CN116452469B (en) | Image defogging processing method and device based on deep learning | |
| CN117237207A (en) | Ghost-free high dynamic range light field imaging method for dynamic scene | |
| Hu et al. | Modified image haze removal algorithm based on dark channel prior | |
| CN116452450A (en) | Polarized image defogging method based on 3D convolution | |
| Mishra et al. | Fusion of multiscale gradient domain enhancement and gamma correction for underwater image/video enhancement and restoration | |
| Fan et al. | Multi-scale dynamic fusion for correcting uneven illumination images | |
| Roy et al. | Diffnat: Improving diffusion image quality using natural image statistics |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |