CN111738939B

CN111738939B - Complex scene image defogging method based on semi-training generator

Info

Publication number: CN111738939B
Application number: CN202010490188.8A
Authority: CN
Inventors: 王野; 孙亮; 葛宏伟; 谭国真
Original assignee: Dalian University of Technology
Current assignee: Dalian University of Technology
Priority date: 2020-06-02
Filing date: 2020-06-02
Publication date: 2022-02-15
Anticipated expiration: 2040-06-02
Also published as: CN111738939A

Abstract

A complex scene image defogging method based on a semi-training generator belongs to the field of image defogging applied to complex environments and comprises a training process and a using process. In the training process, firstly, a CycleGAN network is used for training on any image defogging data set, a real-time defogging image is output every 50 times of training, the current model is stored, and the training is finished for 2000 times; secondly, repeating the above process ten times; and finally, selecting a fog-free gray image with the best fog removal effect and no color information from the saved effect images, and taking a generator G in the corresponding saved model as a final sketch module. When the haze image processing device is used, the haze image of any scene is input into the sketch module, and the haze-removed gray image can be output. The application range of the method is not limited to the training data set, the method has strong adaptability, visibility and reality, can be applied to any scene, and can help the intelligent system to play a certain role in the environment affected by dense fog.

Description

Complex scene image defogging method based on semi-training generator

Technical Field

The invention belongs to the field of image defogging applied to complex environments, and relates to an image conversion type single image defogging method based on a semi-training generator.

Background

In the field of computer vision, the quality of an image has a great influence on the completion effect of tasks such as target detection, image recognition and the like. However, images acquired in real-world application environments are often affected by airborne suspended matter (fog, haze, dust, etc.). These effects can blur the image, eventually making it difficult to extract valid features of the image. The research image defogging technology can restore the image affected by fog into an image similar to a fog-free state, so that intelligent systems such as an automatic driving system, a monitoring system and a target recognition system can normally operate in severe weather such as haze. Therefore, how to achieve efficient image defogging has become an important part of research in the field of computer vision. For an intelligent system which needs to be deployed in multiple environments, the image defogging algorithm is required to be capable of simultaneously dealing with various environments, and the research on the algorithm is another huge challenge.

Current single image defogging algorithms can be roughly classified into three categories, one is defogging algorithms based on image enhancement, the second is algorithms based on atmospheric scattering models, and the third is defogging algorithms based on image conversion. The defogging algorithm based on image enhancement mainly depends on a general image enhancement technology, content of interest of people is enhanced, other content is inhibited to improve the visibility of an image, and the characteristics of fog are not processed, so that phenomena of color distortion, supersaturation and the like often occur, bad influence is caused on the contrary, and the defogging effect is not good enough. The defogging algorithm based on the atmospheric scattering model has higher authenticity and better effect than the image enhancement method because the used physical model is used for image degradation and reduction. The method based on the prior theory has the limitation of scenes, and simultaneously has unsatisfactory effect on the edge processing of the scene depth mutation position in the image. The effect which can be achieved by the method based on the three-dimensional model and the deep learning is the best at present, but a large standard data set is required, and the data set is extremely difficult to obtain in a real environment, so that the method is limited to a laboratory environment and has poor adaptability. The method based on image conversion solves the problem of dependence of deep learning on a data set by relying on a CycleGAN framework, has good adaptability and has very good engineering practical value.

Since 2017, Zhu et al proposed cycleGAN (Zhu J Y, Park T, Isola P, et al, Ungained Image-to-Image transformation using Cycle-dependent additive Networks [ J]2017.) network enables training without requiring paired data, researchers are based on CThe ycegan network framework proposes some defogging algorithms. Cycle-Dehaze by Engin (Engin D,

A,Kemal Ekenel H.Cycle-dehaze:Enhanced CycleGAN for single image dehazing[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition works 2018:825-833.) the loss of cyclic perception is increased on the basis of CycleGAN, and after the image is output, the image is enlarged by using Laplace transform to obtain a high-definition defogged image. The loss of cyclic perception of Cycle-Dehaze is determined by vgg16 (Simnyan K, Zisserman A. Very Deep conditional Networks for Large-Scale Image Recognition [ J)]Computer Science,2014.) performs high-dimensional feature transformation on the input image and the reconstructed image, and then performs variance calculation on the high-dimensional vector to obtain another cyclic loss constraint. Yang (Yang X, Xu Z, Luo J. Towards technical image dehazing by graphics-based differentiation and adaptation [ C)]// third-second AAAI conference on intellectual identity interference.2018.) the discrimination performance of the network is enhanced by the loss of multi-scale discriminators, the training effect is improved, and the cycleGAN is enhanced. Similar to Yang's thought, ZHao (ZHao J, ZHang J, Li Z, et al. DD-CycleGAN: unpaired image dehazing vitamin double-discrete genetic additive network [ J]Engineering Applications of intellectual Intelligence,2019,82:263-271.) directly uses the dual discriminator structure for intensive training, which also achieves good results.

However, the methods enhance the defogging effect by enhancing the general performance of CycleGAN, and do not research the characteristics of the fog, the fog characteristics of some images are difficult to distinguish, the model is insufficient in image analysis, the fog cannot be completely separated from the images, the fog is interfered by residual fog information in image conversion, the defogging effect is not particularly ideal, and the corresponding scenes still need to be trained for good image defogging.

Disclosure of Invention

Aiming at the problems that the practicability of the existing method is not good enough and the requirements of simultaneously coping with various environments are difficult to meet, the invention provides a defogging method for a complex scene image based on a Semi-Training generator, which is realized based on a Semi-Training Color Stripping defogging network (STCSDN), wherein the STCSDN relies on a classical cycleGAN network to complete the defogging task through a new Training strategy. The STCSDN mainly depends on two important properties in the defogging process of the convolutional neural network, wherein the learning speed of the convolutional neural network to outline and shadow information is faster than that of color information and content information, and the STCSDN is insensitive to the fog concentration. Based on property one, by undertraining the loop countermeasure network, a semi-trained generator is obtained as a sketch module of the STCSDN. The sketch module can extract a fog-free gray image from a foggy color image, and the gray image only contains the outline and shadow information of the image, so that the original color information interfered by the fog information is abandoned. Since the sketch module does not learn the specific content of the image, the application scope is not limited to the trained data set. Therefore, the method has extremely strong adaptability, visibility and reality, the sketch module trained on any data set can be applied to any other scene, and the intelligent system can be deployed and used in various scenes simultaneously. Based on property two, the sketch module can extract information such as outlines of images from dense fog which almost covers a scene, so that fog-free gray level images with use values are obtained, and the intelligent system can also play a certain role when being influenced by the dense fog.

In order to achieve the purpose, the invention adopts the technical scheme that:

a color stripping single image defogging method based on a semi-training generator performs model training on a traditional CycleGAN network through a brand-new training strategy to obtain an image defogging model capable of adapting to any scene, comprises a training process and a using process, and comprises the following steps:

and in the training process, insufficient training is carried out on the circulation confrontation network to obtain a draft module:

step 1: training on any image defogging dataset by using a classical CycleGAN network;

step 2: in the training process, outputting a real-time defogging image every 50 times of training and storing the current model;

and step 3: finishing training for a certain number of times; the training times in a single training process are the times of the network starting to learn the color information, and are usually 2000 times;

and 4, step 4: repeating the step 1-the step 3 for a plurality of times; the more the repetition times are in the training process, the more easily a model with better effect can be obtained, and the more 10 times are usually adopted;

and 5: selecting a fog-free gray image with the best defogging effect and no color information from the stored effect images, and taking a generator G in the corresponding storage model as a final sketch module; the sketch module does not learn the specific content of the image, the application range of the sketch module is not limited to a data set in the training process, the sketch module can be applied to any other scene, and the sketch module can extract the outline and shadow information of the image from dense fog which almost covers the scene to obtain a fog-free gray image with use value.

In the using process, a sketch module obtained in the training process can extract a fog-free gray image from a foggy color image:

step 1: the foggy image of any scene (not limited to the scene to which the training set belongs) is input into the sketch module, the dehazed gray image can be output, the gray image only contains the outline and shadow information of the image, and the original color information interfered by the foggy information can be discarded.

The invention trains the CycleGAN network and the similar network in an insufficient training mode, and selects a model which is only trained for about hundreds of times as a final defogging model, wherein the model has the following characteristics: the output image only contains primary information such as outline information and shadow information of a scene in the original fog image, and high-level information such as fog and color information does not exist; since the model learns only low-level common information such as contour information, it is possible to recognize information such as a contour even in an image of a scene not belonging to the training data set, and to obtain a defogged grayscale image.

The invention has the beneficial effects that: the color stripping single image defogging method of the semi-training generator can output a grayscale defogged image. The gray defogged image contains information such as the outline, shadow, brightness and the like of the original image, so that the content in the original image can be identified by human eyes and an intelligent processing system. The image extracted by the method retains the real information of the image and maximally highlights the scenery visible in the scene. Because the sketch module only operates the most basic information such as outlines and the like in the images and does not react to the advanced knowledge of the content, the sketch module trained on a single scene can be suitable for any scene and can meet the deployment requirements of complex and changeable environments and various scenes at the same time. Because the sketch is insensitive to the concentration of the fog, the method can extract information such as outlines of the image from the dense fog which almost covers the scene, obtain a gray image with use value, and help the intelligent system to play a certain efficiency in the environment affected by the dense fog.

Drawings

FIG. 1 is a diagram of the STCSDN network framework of the present invention;

FIG. 2 is a schematic diagram of a model retention strategy;

FIG. 3 is a schematic diagram of a model screening strategy;

FIG. 4 is a schematic view of a model defogging process;

FIG. 5 is a diagram showing the defogging effect of STCSDN;

fig. 6 is a diagram showing defogging effect of the STCSDN fog scene.

Detailed Description

The invention provides a complex scene image defogging method based on a semi-training generator. The specific embodiments discussed are merely illustrative of implementations of the invention and do not limit the scope of the invention. Embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

A semi-training generator-based color stripping single image defogging method is characterized in that the overall framework of the method is based on a CycleGAN network, as shown in figure 1, a defogging data set is obtained through insufficient training of the CycleGAN network, and the defogging data set is used for extracting lines, shadows and brightness information of an image and acquiring a grayscale image without color information. The gray image describes the content in the original image only by the outline information, similar to the sketch in the drawing, so we call this module the sketch module. The method comprises the following specific steps:

(1) and in the training process, insufficient training is carried out on the circulation confrontation network to obtain a draft module:

network training process: training is carried out on a data set of any image defogging field by using a traditional CycleGAN network as shown in figure 1, the parameters use default CycleGAN parameters, a generator G converts a foggy image into a fogless image, and a generator F converts the fogless image into a fogless image. The training process is as shown in fig. 2, outputting a real-time fog-free image after every 50 times of training, storing the current model, and training to 2000 times of iteration to terminate the training. Training 10 sets of cyclegans was repeated to obtain 400 fog-free images and corresponding models.

And (3) model screening process: as shown in fig. 3, the fog-free gray image with the best defogging effect and no color information is manually selected from the saved effect maps, and the generator G of the corresponding model which is saved is used as a final sketch module.

In the training process, the cycleGAN is trained on a data set which is I-HAZE and has few samples and clear and simple scenery in a random training mode, the intermediate process is intensively stored and the effect diagram is output, 10 groups of cycleGAN are trained each time, and each group of training is terminated by 2000 times of iteration. One generated image is output every 50 iterations, and the nodes are maintained. After training is finished, the image with the best effect and the defogging direction generator of the corresponding node are manually selected to serve as a sketch module. As shown in fig. 2, for example, the images at 700 th, 750 th and 850 th times are already clear, so that the nodes corresponding to the images are tried to test the high-resolution foggy image, and the node with the most ideal effect is selected as the sketch module to be selected.

(2) The using process, namely the model defogging process: as shown in fig. 4, the fog-free gray image with a size of 2000 × 2000 can be output by inputting the fog-free image of any scene (not limited to the scene to which the training set belongs) into the sketch module.

The above-mentioned embodiments only express the embodiments of the present invention, but not should be understood as the limitation of the scope of the invention patent, it should be noted that, for those skilled in the art, many variations and modifications can be made without departing from the concept of the present invention, and these all fall into the protection scope of the present invention.

Claims

1. A single image defogging method for a complex scene based on a semi-training generator is characterized by comprising the following steps:

training process:

and step 3: finishing training for a certain number of times;

and 4, step 4: repeating the step 1 to the step 3 for multiple times, wherein the more the repetition times are, the better the obtained effect is;

and 5: selecting a fog-free gray image with the best defogging effect and no color information from the stored effect images, and taking a generator G in the corresponding storage model as a final sketch module;

the way of training the CycleGAN network is insufficient training, and the sketch module has the following characteristics: the output image only contains the primary information of the scenery in the original fog image, and no high-level information exists; the sketch module does not learn the specific content of the image, the application range of the sketch module is not limited to the data set in the training process, and the sketch module can be applied to any other scenes; the primary information comprises contour information and shadow information, and the high-level information comprises color information;

the use process comprises the following steps:

step 1: inputting the foggy image of any scene into the sketch module, and outputting the defogged gray image; since the sketch module only learns the low-level common information of the contour information, the image of the scene which does not belong to the training data set can be identified, and the defogged gray image can be obtained.

2. The method for defogging the single image in the complex scene based on the semi-training generator as claimed in claim 1, wherein the number of times of training in the single training process is 2000 times of learning the color information by the network.