CN115376066A - Airport scene target detection multi-weather data enhancement method based on improved cycleGAN - Google Patents

Airport scene target detection multi-weather data enhancement method based on improved cycleGAN Download PDF

Info

Publication number
CN115376066A
CN115376066A CN202210989237.1A CN202210989237A CN115376066A CN 115376066 A CN115376066 A CN 115376066A CN 202210989237 A CN202210989237 A CN 202210989237A CN 115376066 A CN115376066 A CN 115376066A
Authority
CN
China
Prior art keywords
network
target detection
cyclegan
improved
weather
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210989237.1A
Other languages
Chinese (zh)
Inventor
阳媛
王紫航
严如强
徐佳文
杨浩然
况余进
蔡杰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Southeast University
Original Assignee
Southeast University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Southeast University filed Critical Southeast University
Priority to CN202210989237.1A priority Critical patent/CN115376066A/en
Publication of CN115376066A publication Critical patent/CN115376066A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/07—Target detection
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
    • Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
    • Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Multimedia (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computational Linguistics (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses an airport scene target detection multi-weather data enhancement method based on improved cycleGAN. On the basis of an original CycleGAN network, a generator network is improved by expanding a residual error network block, cross-layer connection of a coder-decoder and introducing an attention mechanism, a dynamic weighted multi-scale discriminator network is designed, an airport scene small sample target detection data set in normal weather and an auxiliary data set in typical severe weather are used for training, a multi-weather image generation network is obtained, high-quality target detection images in various weather conditions such as rainy days, foggy days and nights are effectively generated, and diversity of data samples is increased. The method provided by the invention is used as an effective data enhancement method for saving the labeling cost, and the performance of the airport scene target detection network under the condition of small samples is improved.

Description

Airport scene target detection multi-weather data enhancement method based on improved cycleGAN
Technical Field
The invention belongs to the field of image generation in computer vision, relates to an airport scene target detection multi-weather data enhancement method based on improved cycleGAN, and particularly relates to a method for generating countermeasure network through improved cycle consistency to generate airport scene target detection data under typical severe weather so as to enhance the data.
Background
As one of the typical tasks in the field of computer vision, object detection has been rapidly developed in recent years, and a large number of excellent network models are developed. However, such models often need a large-scale data set to show better performance, and this condition is difficult to satisfy in practical application scenarios and has extremely high cost, so the data enhancement method becomes an effective means for improving network performance under small sample data. In the visual field, common data enhancement methods include rotation, scaling, turning, clipping and the like of pictures, and these basic data enhancement methods have a good effect on image classification tasks, but have an unsatisfactory effect when applied to target detection tasks. The excellent performance of the generation of the countermeasure network is a big hotspot in recent years, and a fake sample can be generated through the game process between the generator and the discriminator, so that the effect of data enhancement is achieved.
In an airport scene target detection task, various targets such as special vehicles, aircrafts and the like have strong randomness and variable scales, image illumination and the like change along with seasons and weather, a large number of data samples are difficult to collect for marking, and the target detection effect is poor under the condition of small samples. The improved cycleGAN network is used for data generation, the diversity of target detection data is increased, and the method is an effective and feasible means for improving the performance of the target detection network.
Disclosure of Invention
Aiming at the problems, the invention provides an airport scene target detection multi-weather data enhancement method based on improved cycleGAN, which respectively improves a generator network part and a discriminator network part on the basis of generating a countermeasure network by cycle consistency, and simultaneously adjusts an integral loss function, so that the model can generate high-quality picture data under various typical severe weather, original real data and the generated high-quality data are combined, and the diversity of training samples is increased, thereby improving the generalization performance of a target detection network, and finally realizing a more efficient small sample target detection technology.
In order to achieve the purpose, the invention adopts the technical scheme that:
an airport surface target detection multi-weather data enhancement method based on improved cycleGAN comprises the following steps:
s1, image data acquisition and preprocessing, and data set manufacturing;
the specific process of the step S1 is as follows:
acquiring a source image data set through an airport scene monitoring camera, acquiring target detection data in normal weather within a period of time, and marking the target detection data to be used as an original small sample target detection data set;
collecting image data under typical severe weather as an auxiliary weather data set;
performing basic data enhancement operation on the two original data sets to obtain a training data set of the improved cycleGAN network;
s2, constructing an improved attention mechanism generator network;
the specific process of the step S2 is as follows:
expanding a residual error style conversion network in an original cycleGAN network generator, and adding a two-layer convolution residual error network block after each convolution block of an encoding and decoding part;
in a generator of an original cycleGAN network, introducing a jump connection idea in U-Net, namely performing cross-layer connection operation on a part corresponding to coding and decoding;
replacing a residual error style conversion network in an original cycleGAN network generator by using a dense connection network DenseNet;
adding an attention mechanism module at the left end and the right end of the style conversion network DenseNet respectively;
s3, constructing a dynamic weighting multi-scale discriminator network;
the specific process of the step S3 is as follows:
performing downsampling operation on the image through two PatchGAN full convolution network branches with different scales to obtain image discrimination results with different sizes;
s4, antagonistic training;
the specific process of the step S4 is as follows:
adjusting the loss function of the original CycleGAN network on the basis of the improved generator and discriminator network, namely:
L(G,F,D X ,D Y )＝L GAN (G,D Y ,X,Y)+L GAN (F,D X ,Y,X)+λL cyc (G,F)+L Identity (G,F)
wherein
Figure BDA0003803253210000021
In order to cycle the loss of consistency,
Figure BDA0003803253210000022
for constant loss, L GAN (G,D Y X, Y) is the modified generated countermeasure network loss, where the multi-scale discriminator network loss is dynamically weighted, specifically:
Figure BDA0003803253210000023
Figure BDA0003803253210000024
from d A1 ＝2(1-2(L 1 ) And d) and A2 ＝2(1-2(L 2 ) Separately calculate A-distance, thereby obtaining dynamicsWeighting factors
Figure BDA0003803253210000025
Finally have
L GAN (G,D Y ,X,Y)＝αL 1 +(1-α)L 2
L GAN (F,D X Y, X) are the same;
g, F, D in the above loss function X ,D Y Generators of original small sample target detection data X to auxiliary weather data Y, generators of auxiliary weather data Y to original small sample target detection data X, a discriminator taking original small sample target detection data X as a real sample and a discriminator taking auxiliary weather data Y as a real sample; d Y1 、D X1 And D Y2 、D X2 The two discriminators with different scales are respectively used, lambda is a penalty coefficient, and the network parameters are obtained by gradient descent training according to the following formula: g * ,F * ＝argmin G,F max DX,DY L(G,F,D X ,D Y )。
Further, the normal weather in step S1 is daytime and sunny.
Further, the target in step S1 includes a special vehicle and an aircraft.
Further, typical bad weather in the step S1 includes cloudy days, rainy days, foggy days, and nights.
Further, the data enhancement operation in step S1 includes rotation, scaling and flipping.
Further, in the step S2, the attention mechanism module is a spatial channel mixed attention mechanism.
Compared with the prior art, the invention has the beneficial effects that:
the generator network is improved by expanding the cross-layer connection of the residual error network block and the codec and introducing an attention mechanism, and meanwhile, the dynamic weighted multi-scale discriminator network is designed, so that the training is performed under the conditions of a small sample target detection data set and an auxiliary weather data set in an airport scene, high-quality target detection images in rainy days, foggy days, nights and other weather can be effectively generated, the diversity of target detection data samples is increased, and the target detection network performance under the condition of small samples is effectively improved.
Drawings
FIG. 1 is a network architecture diagram of the proposed method of the present invention;
FIG. 2 is a schematic diagram of a cycle consistent generated confrontation network;
FIG. 3 is a diagram of a generator network architecture for the proposed method of the present invention;
fig. 4 is a diagram of a network structure of the arbiter according to the present invention.
Detailed Description
The invention is described in further detail below with reference to the following detailed description and accompanying drawings:
as shown in fig. 1, an original small sample target detection data set and an auxiliary weather data set are respectively used as source domain data X and target domain data Y, an improved CycleGAN network model is trained, a multi-weather image generation network is obtained, and then a multi-weather target detection image is generated. The improved CycleGAN model comprises a generator G and a generator F which respectively correspond to the mapping of data X to data Y and the mapping of data Y to data X; two multi-scale discriminators Dx and Dy discriminate the generation in two directions, respectively, and Dx and Dy each include two sub discriminators. And obtaining the high-quality multi-weather image generation network through iterative training. Fig. 2 is a schematic diagram of an original cycle consistency generation countermeasure network.
Specifically, the main steps are as follows:
s1, acquiring and preprocessing image data, and making a data set;
acquiring a source image data set through an airport scene monitoring camera, acquiring target detection data in normal weather within a period of time, and marking the target detection data to be used as an original small sample target detection data set; collecting image data under the foggy weather condition as an auxiliary weather data set; performing basic data enhancement operation on the two original data sets to obtain a training data set of the improved cycleGAN network;
s2, constructing an improved attention mechanism generator network;
figure 3 is a diagram of an improved generator network architecture. As shown in the figure, the residual error style conversion network in the original CycleGAN network generator is expanded, and a two-layer convolution residual error network block is added after each convolution block of the coding and decoding part; in a generator of an original cycleGAN network, introducing a jump connection idea in U-Net, namely performing cross-layer connection operation on a corresponding part of coding and decoding; replacing a residual error style conversion network in an original cycleGAN network generator by using a dense connection network DenseNet; adding an attention mechanism module at the left end and the right end of the style conversion network DenseNet respectively;
s3, constructing a dynamic weighting multi-scale discriminator network;
fig. 4 is a diagram of an improved arbiter network. As shown in the figure, downsampling operation is carried out on the image through two PatchGAN full convolution network branches with different scales, and image discrimination results with different sizes are obtained;
step 4, antagonistic training;
according to the adjusted network loss function, namely:
L(G,F,D X ,D Y )＝L GAN (G,D Y ,X,Y)+L GAN (F,D X ,Y,X)+λL cyc (G,F)+L Identity (G,F)
optimization goal G * ,F * ＝argmin G,F max DX,DY L(G,F,D X ,D Y ) Setting initialization parameters to train the model, alternately updating the discriminator and the generator, finally completing the training to obtain a generation network, and inputting target detection data in normal weather to obtain correspondingly generated foggy day target detection data.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the present invention in any way, but any modifications or equivalent variations made according to the technical spirit of the present invention are within the scope of the present invention as claimed.

Claims (6)

1. An airport scene target detection multi-weather data enhancement method based on improved CycleGAN is characterized by comprising the following steps:
s1, acquiring and preprocessing image data, and making a data set;
the specific process of the step S1 is as follows:
acquiring a source image data set through an airport scene monitoring camera, acquiring target detection data in normal weather within a period of time, and marking the target detection data to be used as an original small sample target detection data set;
collecting image data under typical severe weather as an auxiliary weather data set;
performing basic data enhancement operation on the two original data sets to obtain a training data set of the improved cycleGAN network;
s2, constructing an improved attention mechanism generator network;
the specific process of the step S2 is as follows:
expanding a residual error style conversion network in an original cycleGAN network generator, and adding a two-layer convolution residual error network block after each convolution block of an encoding and decoding part;
in a generator of an original cycleGAN network, introducing a jump connection idea in U-Net, namely performing cross-layer connection operation on a part corresponding to coding and decoding;
replacing a residual error style conversion network in an original cycleGAN network generator by using a dense connection network DenseNet;
adding an attention mechanism module at the left end and the right end of the style conversion network DenseNet respectively;
s3, constructing a dynamic weighting multi-scale discriminator network;
the specific process of the step S3 is as follows:
performing downsampling operation on the image through two PatchGAN full convolution network branches with different scales to obtain image discrimination results with different sizes;
s4, antagonistic training;
the specific process of the step S4 is as follows:
adjusting the loss function of the original CycleGAN network on the basis of the improved generator and discriminator network, namely:
L(G,F,D X ,D Y )＝L GAN (G,D Y ,X,Y)+L GAN (F,D X ,Y,X)+λL cyc (G,F)+L Identity (G,F)
wherein
Figure FDA0003803253200000011
In order to cycle the loss of consistency,
Figure FDA0003803253200000012
for constant loss, L GAN (G,D Y X, Y) is the modified generated countermeasure network loss, where the multi-scale discriminator network loss is dynamically weighted, specifically:
Figure FDA0003803253200000013
Figure FDA0003803253200000014
from d A1 ＝2(1-2(L 1 ) A) and d A2 ＝2(1-2(L 2 ) Respectively calculate A-distance to obtain dynamic weighting factors
Figure FDA0003803253200000021
Finally is provided with
L GAN (G,D Y ,X,Y)＝αL 1 +(1-α)L 2
L GAN (F,D X Y, X) are the same;
g, F, D in the above loss function X ,D Y Generators of original small sample target detection data X to auxiliary weather data Y, generators of auxiliary weather data Y to original small sample target detection data X, a discriminator taking original small sample target detection data X as a real sample and a discriminator taking auxiliary weather data Y as a real sample; d Y1 、D X1 And D Y2 、D X2 Respectively judging the two different scalesAnd identifying the device, wherein lambda is a penalty coefficient, and the network parameters are obtained by gradient descent training according to the following formula:
G * ,F * ＝arg min G,F max DX,DY L(G,F,D X ,D Y )。
2. the method for enhancing the airport surface target detection multi-weather data based on the improved CycleGAN as claimed in claim 1, wherein:
the normal weather in the step S1 is daytime and sunny.
3. The method for enhancing the airport surface target detection multi-weather data based on the improved CycleGAN as claimed in claim 1, wherein:
the target in the step S1 comprises a special vehicle and an aircraft.
4. The method for enhancing the airport surface target detection multi-weather data based on the improved CycleGAN as claimed in claim 1, wherein:
typical severe weather in the step S1 includes cloudy days, rainy days, foggy days, and nights.
5. The method for enhancing the airport surface target detection multi-weather data based on the improved CycleGAN as claimed in claim 1, wherein:
the data enhancement operation in the step S1 comprises rotation, scaling and turning.
6. The method for enhancing the airport surface target detection multi-weather data based on the improved CycleGAN as claimed in claim 1, wherein:
in the step S2, the attention mechanism module is a spatial channel hybrid attention mechanism.
CN202210989237.1A 2022-08-17 2022-08-17 Airport scene target detection multi-weather data enhancement method based on improved cycleGAN Pending CN115376066A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210989237.1A CN115376066A (en) 2022-08-17 2022-08-17 Airport scene target detection multi-weather data enhancement method based on improved cycleGAN

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210989237.1A CN115376066A (en) 2022-08-17 2022-08-17 Airport scene target detection multi-weather data enhancement method based on improved cycleGAN

Publications (1)

Publication Number Publication Date
CN115376066A true CN115376066A (en) 2022-11-22

Family

ID=84064726

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210989237.1A Pending CN115376066A (en) 2022-08-17 2022-08-17 Airport scene target detection multi-weather data enhancement method based on improved cycleGAN

Country Status (1)

Country Link
CN (1) CN115376066A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116912680A (en) * 2023-06-25 2023-10-20 西南交通大学 SAR ship identification cross-modal domain migration learning and identification method and system
CN118154467A (en) * 2024-05-11 2024-06-07 华东交通大学 Image rain removing method and system based on improved CycleGAN network model

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116912680A (en) * 2023-06-25 2023-10-20 西南交通大学 SAR ship identification cross-modal domain migration learning and identification method and system
CN118154467A (en) * 2024-05-11 2024-06-07 华东交通大学 Image rain removing method and system based on improved CycleGAN network model

Similar Documents

Publication Publication Date Title
Wang et al. SFNet-N: An improved SFNet algorithm for semantic segmentation of low-light autonomous driving road scenes
Jiang et al. Edge-enhanced GAN for remote sensing image superresolution
CN110111335B (en) Urban traffic scene semantic segmentation method and system for adaptive countermeasure learning
CN113052210B (en) Rapid low-light target detection method based on convolutional neural network
Bi et al. Improved VGG model-based efficient traffic sign recognition for safe driving in 5G scenarios
CN110909666B (en) Night vehicle detection method based on improved YOLOv3 convolutional neural network
CN115376066A (en) Airport scene target detection multi-weather data enhancement method based on improved cycleGAN
CN108764298B (en) Electric power image environment influence identification method based on single classifier
CN110443763B (en) Convolutional neural network-based image shadow removing method
CN109993804A (en) A kind of road scene defogging method generating confrontation network based on condition
CN116758130A (en) Monocular depth prediction method based on multipath feature extraction and multi-scale feature fusion
Sun et al. IRDCLNet: Instance segmentation of ship images based on interference reduction and dynamic contour learning in foggy scenes
Cheng et al. A highway traffic image enhancement algorithm based on improved GAN in complex weather conditions
CN114627269A (en) Virtual reality security protection monitoring platform based on degree of depth learning target detection
Cheng et al. Generating high-resolution climate prediction through generative adversarial network
CN118314377A (en) Edge computing-oriented light-weight bimodal interaction target detection method
Wang et al. PACCDU: Pyramid attention cross-convolutional dual UNet for infrared and visible image fusion
Liangjun et al. MSFA-YOLO: A Multi-Scale SAR Ship Detection Algorithm Based on Fused Attention
Li et al. Monocular 3-D Object Detection Based on Depth-Guided Local Convolution for Smart Payment in D2D Systems
Ge et al. MambaTSR: You Only Need 90k Parameters for Traffic Sign Recognition
Zhu et al. HDRD-Net: High-resolution detail-recovering image deraining network
Ran et al. Adaptive fusion and mask refinement instance segmentation network for high resolution remote sensing images
CN111008555B (en) Unmanned aerial vehicle image small and weak target enhancement extraction method
CN113095181A (en) Traffic sign identification method based on Defense-GAN
Alshammari et al. Multi-task learning for automotive foggy scene understanding via domain adaptation to an illumination-invariant representation

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination