CN115376066A - Airport scene target detection multi-weather data enhancement method based on improved cycleGAN - Google Patents

Abstract

The invention discloses an airport scene target detection multi-weather data enhancement method based on improved cycleGAN. On the basis of an original CycleGAN network, a generator network is improved by expanding a residual error network block, cross-layer connection of a coder-decoder and introducing an attention mechanism, a dynamic weighted multi-scale discriminator network is designed, an airport scene small sample target detection data set in normal weather and an auxiliary data set in typical severe weather are used for training, a multi-weather image generation network is obtained, high-quality target detection images in various weather conditions such as rainy days, foggy days and nights are effectively generated, and diversity of data samples is increased. The method provided by the invention is used as an effective data enhancement method for saving the labeling cost, and the performance of the airport scene target detection network under the condition of small samples is improved.

Description

A Multi-weather Data Enhancement for Airport Scene Target Detection Based on Improved CycleGAN method

technical field

The invention belongs to the field of image generation in computer vision, and relates to an airport scene target detection multi-weather data enhancement method based on improved CycleGAN, in particular to an airport scene under typical severe weather generated by an improved cycle consistency generation confrontation network A method for data augmentation based on target detection data.

Background technique

Object detection, as one of the typical tasks in the field of computer vision, has achieved rapid development in recent years, and a large number of excellent network models have emerged. However, such models often require large-scale data sets to show good performance. This condition is difficult to meet in practical application scenarios, and the cost is extremely high. Therefore, data enhancement methods have become an effective means to improve network performance under small sample data. In the field of vision, common data enhancement methods include image rotation, scaling, flipping, cropping, etc. These basic data enhancement methods work well for image classification tasks, but are not ideal for target detection tasks. Generative adversarial networks have also become a hot topic in recent years due to their excellent performance. Through the game process between the generator and the discriminator, it can generate fake samples to achieve the effect of data enhancement.

In the airport scene target detection task, various targets such as special vehicles, aircraft, etc., have strong randomness, variable scale, and image illumination changes with seasons and weather, so it is difficult to collect a large number of data samples for labeling. Target detection under small sample conditions not effectively. Using the improved CycleGAN network for data generation and increasing the diversity of target detection data is an effective and feasible means to improve the performance of target detection networks.

Contents of the invention

In view of the above problems, the present invention provides an airport scene target detection multi-weather data enhancement method based on improved CycleGAN. On the basis of cycle consistency generation confrontation network, the generator network and the discriminator network are respectively improved, and the overall loss is adjusted at the same time. function, so that the model can generate a variety of high-quality image data under typical severe weather, combine the original real data with the generated high-quality data, increase the diversity of training samples and improve the generalization performance of the target detection network, so as to finally achieve More efficient small sample target detection technology.

For realizing above-mentioned object, the technical scheme that the present invention takes is:

A kind of airport scene object detection multi-weather data enhancement method based on improved CycleGAN, described method comprises the steps:

Step S1, image data collection and preprocessing, data set production;

The specific process of the step S1 is:

The source image data set is collected through the airport scene surveillance camera, and the target detection data under normal weather for a period of time is obtained and marked as the original small sample target detection data set;

Collect image data under typical severe weather as an auxiliary weather data set;

Perform basic data enhancement operations on the above two original data sets to obtain a training data set for the improved CycleGAN network;

Step S2, construction of an improved attention mechanism generator network;

The specific process of the step S2 is:

Expand the residual style conversion network in the original CycleGAN network generator, and add a two-layer convolutional residual network block after each convolution block in the encoding and decoding parts;

In the generator of the original CycleGAN network, the idea of skip connection in U-Net is introduced, that is, the corresponding part of the codec is connected across layers;

Use the densely connected network DenseNet to replace the residual style transfer network in the original CycleGAN network generator;

Add an attention mechanism module to the left and right ends of the style transfer network DenseNet;

Step S3, constructing a dynamic weighted multi-scale discriminator network;

The specific process of the step S3 is:

The image is down-sampled through two PatchGAN full convolutional network branches of different scales to obtain image discrimination results of different sizes;

Step S4, confrontational training;

The specific process of the step S4 is:

Adjust the loss function of the original CycleGAN network based on the above improved generator and discriminator network, namely:

L(G,F,D _X ,D _Y )＝L _GAN (G,D _Y ,X,Y)+L _GAN (F,D _X ,Y,X)+λL _cyc (G,F)+L _Identity ( G,F)

为循环一致性损失，

为恒等损失，L_GAN(G,D_Y,X,Y)为修正的生成对抗网络损失，此处对多尺度判别器网络损失进行动态加权，具体地：in

is the cycle consistency loss,

is the constant loss, L _GAN (G,D _Y ,X,Y) is the modified generation confrontation network loss, where the multi-scale discriminator network loss is dynamically weighted, specifically:

最后有Calculate the A-distance from d _A1 = 2(1-2(L ₁ )) and d _A2 = 2(1-2(L ₂ )) to obtain the dynamic weighting factor

Finally there

L _GAN (G,D _Y ,X,Y)=αL ₁ +(1-α)L ₂

L _GAN (F, D _X , Y, X) is the same;

In the above loss function, G, F, D _X , and D _Y are the generators from the original small-sample target detection data X to the auxiliary weather data Y, and the generators from the auxiliary weather data Y to the original small-sample target detection data X. The sample target detection data X is the discriminator of the real sample, and the auxiliary weather data Y is the discriminator of the real sample; D _Y1 , D _X1 and D _Y2 , D _X2 are the discriminators of the above two different scales respectively, and λ is the penalty coefficient , the network parameters are obtained by the following formula through gradient descent training: G ^* , F ^* = argmin _{G, F} max _{DX, DY} L(G, F, D _X , D _Y ).

Further, the normal weather in the step S1 is daytime and sunny.

Further, the targets in the step S1 include special vehicles and aircrafts.

Further, the typical severe weather in the step S1 includes cloudy days, rainy days, foggy days and nights.

Further, the data enhancement operation in the step S1 includes rotation, scaling and flipping.

Further, the attention mechanism module in step S2 is a spatial channel mixed attention mechanism.

Compared with prior art, the beneficial effect of the present invention is:

Improve the generator network by expanding the residual network block, the cross-layer connection of the codec and the introduction of the attention mechanism, and at the same time design a dynamic weighted multi-scale discriminator network, a small sample target detection data set in the airport scene and an auxiliary weather data set Training under conditions can effectively generate high-quality target detection images in rainy, foggy, night and other weather conditions, increase the diversity of target detection data samples, and effectively improve the performance of target detection networks under small sample conditions.

Description of drawings

Fig. 1 is the network structure diagram of the method proposed by the present invention;

Figure 2 is a schematic diagram of the cycle consistency generation confrontation network;

Fig. 3 is the generator network structural diagram of the proposed method of the present invention;

Fig. 4 is a network structure diagram of the discriminator of the method proposed by the present invention.

Detailed ways

Below in conjunction with accompanying drawing and specific embodiment the present invention is described in further detail:

As shown in Figure 1, the original small-sample target detection dataset and auxiliary weather dataset are used as source domain data X and target domain data Y respectively, and the improved CycleGAN network model is trained to obtain a multi-weather image generation network, and then generate multiple Weather object detection images. The improved CycleGAN model includes generator G and generator F, which correspond to the mapping of data X to data Y and data Y to data X respectively; two multi-scale discriminators Dx and Dy respectively discriminate the generation of two directions, and Dx Both D and Dy contain two sub-discriminators. A high-quality multi-weather image generation network can be obtained through iterative training. Figure 2 is a schematic diagram of the original cycle consistency generation confrontation network.

Specifically, the main steps are as follows:

Step S1, image data collection and preprocessing, data set production;

Collect the source image data set through the airport scene monitoring camera, obtain the target detection data under normal weather for a period of time and mark it as the original small sample target detection data set; collect the image data under foggy conditions as the auxiliary weather data set; Perform basic data enhancement operations on the above two original data sets to obtain a training data set for the improved CycleGAN network;

Step S2, construction of an improved attention mechanism generator network;

Figure 3 is a structural diagram of the improved generator network. As shown in the figure, the residual style conversion network in the original CycleGAN network generator is expanded, and a two-layer convolutional residual network block is added after each convolution block in the encoding and decoding parts; in the original CycleGAN network In the generator, the skip connection idea in U-Net is introduced, that is, the corresponding part of the codec is connected across layers; the densely connected network DenseNet is used to replace the residual style conversion network in the original CycleGAN network generator; around the style conversion network DenseNet Add an attention mechanism module at both ends;

Step S3, constructing a dynamic weighted multi-scale discriminator network;

Figure 4 is a structural diagram of the improved discriminator network. As shown in the figure, the image is down-sampled through two PatchGAN full convolutional network branches of different scales to obtain image discrimination results of different sizes;

Step 4, confrontational training;

According to the adjusted network loss function, namely:

L(G,F,D _X ,D _Y )＝L _GAN (G,D _Y ,X,Y)+L _GAN (F,D _X ,Y,X)+λL _cyc (G,F)+L _Identity ( G,F)

Optimize the target G ^* , F ^* = argmin _{G, F} max _{DX, DY} L(G, F, D _X , D _Y ), set the initialization parameters to train the model, update the discriminator and generator alternately, and finally the training is completed to generate Network, input the target detection data in normal weather to get the corresponding foggy target detection data.

The above is only a preferred embodiment of the present invention, and is not intended to limit the present invention in any other form, and any modification or equivalent change made according to the technical essence of the present invention still belongs to the scope of protection required by the present invention .

Claims (6)

1. a kind of airport scene target detection multi-weather data enhancement method based on improved CycleGAN, it is characterized in that, described method comprises the steps: Step S1, image data collection and preprocessing, data set production; The specific process of the step S1 is: The source image data set is collected through the airport scene surveillance camera, and the target detection data under normal weather for a period of time is obtained and marked as the original small sample target detection data set; Collect image data under typical severe weather as an auxiliary weather data set; Perform basic data enhancement operations on the above two original data sets to obtain a training data set for the improved CycleGAN network; Step S2, construction of an improved attention mechanism generator network; The specific process of the step S2 is: Expand the residual style conversion network in the original CycleGAN network generator, and add a two-layer convolutional residual network block after each convolution block in the encoding and decoding parts; In the generator of the original CycleGAN network, the idea of skip connection in U-Net is introduced, that is, the corresponding part of the codec is connected across layers; Use the densely connected network DenseNet to replace the residual style transfer network in the original CycleGAN network generator; Add an attention mechanism module to the left and right ends of the style transfer network DenseNet; Step S3, constructing a dynamic weighted multi-scale discriminator network; The specific process of the step S3 is: The image is down-sampled through two PatchGAN full convolutional network branches of different scales to obtain image discrimination results of different sizes; Step S4, confrontational training; The specific process of the step S4 is: Adjust the loss function of the original CycleGAN network based on the above improved generator and discriminator network, namely: L(G,F,D _X ,D _Y )＝L _GAN (G,D _Y ,X,Y)+L _GAN (F,D _X ,Y,X)+λL _cyc (G,F)+L _Identity ( G,F) in

is the cycle consistency loss,

is the constant loss, L _GAN (G,D _Y ,X,Y) is the modified generation confrontation network loss, where the multi-scale discriminator network loss is dynamically weighted, specifically:

Calculate the A-distance from d _A1 = 2(1-2(L ₁ )) and d _A2 = 2(1-2(L ₂ )) to obtain the dynamic weighting factor

Finally there L _GAN (G,D _Y ,X,Y)=αL ₁ +(1-α)L ₂ L _GAN (F, D _X , Y, X) is the same; In the above loss function, G, F, D _X , and D _Y are the generators from the original small-sample target detection data X to the auxiliary weather data Y, and the generators from the auxiliary weather data Y to the original small-sample target detection data X. The sample target detection data X is the discriminator of the real sample, and the auxiliary weather data Y is the discriminator of the real sample; D _Y1 , D _X1 and D _Y2 , D _X2 are the discriminators of the above two different scales respectively, and λ is the penalty coefficient , the network parameters are obtained by the following formula through gradient descent training: G ^* ,F ^* =arg min _G,F max _DX,DY L(G,F,D _X ,D _Y ). 2. a kind of airport scene target detection multi-weather data enhancement method based on improved CycleGAN according to claim 1, is characterized in that: The normal weather in the step S1 is daytime and sunny. 3. a kind of airport scene target detection multi-weather data enhancement method based on improved CycleGAN according to claim 1, is characterized in that: The targets in the step S1 include special vehicles and aircraft. 4. a kind of airport scene object detection multi-weather data enhancement method based on improved CycleGAN according to claim 1, is characterized in that: Typical severe weather in the step S1 includes cloudy days, rainy days, foggy days and nights. 5. a kind of airport scene object detection multi-weather data enhancement method based on improved CycleGAN according to claim 1, is characterized in that: The data enhancement operations in step S1 include rotation, scaling and flipping. 6. a kind of airport scene object detection multi-weather data enhancement method based on improved CycleGAN according to claim 1, is characterized in that: The attention mechanism module in the step S2 is a spatial channel hybrid attention mechanism.

Images