CN114841907A - A method for multi-scale generative adversarial fusion networks for infrared and visible light images - Google Patents

Abstract

The invention discloses a method for generating a confrontation fusion network in a multi-scale mode facing infrared and visible light images, which comprises the steps of selecting a plurality of infrared and visible light image pairs from a standard training set, inputting the image pairs into an edge-preserving filter, and obtaining a basic layer and a detail layer; and inputting the basic layer into a gradient filter to obtain a gradient map and a new basic layer, adding the gradient map and the original detail layer to obtain a new detail layer, calculating to obtain network parameters of the discriminator, training, and finally obtaining an output which is a final fusion image. The image obtained by fusion retains the target information and the texture information of the source image to the maximum extent, improves the quality of the fused image, and provides more convenient prerequisite for subsequent target detection and identification.

Description

A method for multi-scale generative adversarial fusion networks for infrared and visible light images

technical field

The invention belongs to the technical field of image decomposition and image fusion in digital image processing, and in particular relates to a multi-scale generation confrontation fusion network method for infrared and visible light images.

Background technique

Image fusion is a branch of the field of information fusion, which is a cross-domain research involving sensor imaging, image preprocessing, computer vision, and artificial intelligence. With the rapid development of multi-type imaging sensors, the problem of limited image target information provided by a single sensor has been effectively solved. For the same scene, by fusing two or more source images from the same or different imaging sensors, a fusion image with rich information and high definition can be obtained. The visible light sensor uses the reflected light of the object to image, and the obtained image has the characteristics of high resolution and rich details. But in poor lighting conditions, the resulting images are less clear. When the infrared sensor is imaging, it uses the thermal radiation information of the target to image, and the penetrating power is strong. At the same time, it can solve the problem that the imaging effect of the visible light sensor is not good when the light is insufficient or there is an object occluded. The target can be detected, but the detail information and contrast information of the resulting image are insufficient. Infrared and visible light image fusion technology can complement the respective advantages of the two types of images, ensuring that the final fusion image is full of thermal radiation information, contrast information and detail information, so as to better understand the image target information, and finally realize the all-weather operation of the system. In recent years, important progress has been made in multi-scale image fusion technology. Generally, the fusion scheme of infrared and visible light images based on multi-scale transformation includes three steps. First, each source image is decomposed into a series of multi-scale representations, then the multi-scale representations of the source images are fused according to the given fusion rules, and finally the corresponding multi-scale inverse transformation is performed on the fused images. At the same time, with the rapid development of deep learning, unsupervised deep learning has been expanded in the field of fusion and achieved certain results. Although this method is more suitable for multi-source image fusion without reference images, it puts forward higher requirements for the design of network structure and loss function. Therefore, unsupervised generative adversarial network fusion methods have gradually attracted the attention of researchers.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a multi-scale generative adversarial fusion network method for infrared and visible light images, by performing multi-scale decomposition on the source image, and then inputting the decomposed corresponding components into the generative adversarial network to obtain the final fusion image , so that the fused image retains the target information and texture information of the source image to the greatest extent, improves the quality of the fused image, and provides a more convenient prerequisite for subsequent target detection and recognition.

The technical solution adopted in the present invention is a method for multi-scale generation of an adversarial fusion network for infrared and visible light images, which is specifically implemented according to the following steps:

Step 1. Select several infrared and visible light image pairs from the standard training set, and then input the image pairs into the edge preserving filter to obtain the base layer and the detail layer;

Step 2. Input the base layer obtained in step 1 into the gradient filter to obtain a gradient map and a new base layer, and add the gradient map and the detail layer of step 1 as a new detail layer;

Step 3. Input the base layer and detail layer obtained in step 2 to the generator network G, and obtain the fusion image corresponding to the source image pair after passing through the generator network G, calculate the generator loss function L _G , and perform the generator network G parameters. update to obtain the final generator network parameters, input the source image and the fusion image to the discriminator network D for classification, calculate the discriminator loss function L _D , update the discriminator network parameters, and obtain the final discriminator network parameters;

Step 4. Start training the network, and determine whether the iteration is over, that is, whether the current number of iterations has reached the set number of iterations, and the network parameters obtained when the number of iterations reaches the set number of iterations are used as the final network parameters, and the network parameters are saved;

Step 5. Load the generator network parameters obtained in step 4 into the generator network in the test network, and perform multi-scale decomposition of the tested infrared and visible light source images, that is, the filtering operations in steps 1 and 2, and then decompose The corresponding base layer and detail layer are spliced together as the input of the test network, and the obtained output is the final fusion image.

The present invention is also characterized in that,

The filtering formula in step 1 is as follows:

in:

为滤波后图像，q是I_q的一个像素点，s是q的像素集合，p是q领域中的一个像素，

是部分输入图像块，

是

周边图像块，

是空间滤波器内核，

是距离滤波器内核，空间内核和距离内核通常都以高斯的方式表示；In formula (1), I _q is the input image,

is the filtered image, q is a pixel of I _q , s is the set of pixels of q, p is a pixel in the field of q,

is a partial input image patch,

Yes

surrounding image blocks,

is the spatial filter kernel,

is the distance filter kernel, the spatial kernel and the distance kernel are usually expressed in Gaussian form;

In formula (3), I _d0 is the detail layer obtained by bilateral filtering, and I _b0 is the obtained base layer.

in step 2

The gradient value of all pixels in the image is obtained by the following formula:

Then define a threshold Gmax, if the gradient value of the pixel is greater than the threshold, set it as white, otherwise it is black, and thus obtain the gradient map _IG ;

The base layer I _b0 obtained in step 1 is subjected to gradient filtering to obtain a gradient map I _G , and then the original base layer I _b0 is subtracted from the gradient map to obtain a new base layer I _b , and the gradient map I _G is added to the original detail layer I _d0 A new level of detail I _d is available.

In step 3, the generator network structure consists of a dual-stream network and a convolutional neural network followed by it. The upper and lower network structures in the dual-stream network are the same, both are six-layer convolutional neural networks, the first four layers have the same structure, and the network structure is 3 ×3 convolutional layer, batch normalization layer and activation layer, the activation function of this activation layer is Leaky Relu; the latter two layers have the same structure and consist of 5×5 convolutional layer, batch normalization layer and activation layer , the activation function of this activation layer is Leaky Relu, the network structure behind the two-stream network consists of a 1×1 convolution layer and an activation layer, the activation function of this activation layer is tanh, and the output of this layer of convolutional neural network is The final fused image.

The generator loss function _LG in step 3 is:

L _G =λL _content +L _Gen , (6)

where L _content is the content loss after comparing the generator input and output, L _Gen is the adversarial loss between the generator and the discriminator, and λ is a constant;

为梯度算子，ξ为常量；Among them, H and W are the height and width of the image input by the generator, ||·|| ₂ is the calculation of the second norm, I _f is the output of the generator, i.e. the fusion image, I _b is the base layer of the input generator, I _d is the detail layer of the input generator,

is the gradient operator, and ξ is a constant;

L _Gen = E[log(1-D _V (G(I _b , I _d )))]+E[log(1-D _I (G(I _b , I _d )))](8)

D _V (G(I _b , I _d )) represents the discriminator discriminant value input with infrared image or fusion image, G(I _b , I _d ) represents the fusion image generated by the generator, D _I (G(I _{b )} , I _d )) represents the discriminator discriminant value that takes the visible light image or the fusion image as input.

Calculate the generator loss function L _G , and use SGD to update the network parameters to achieve the purpose of optimization, and obtain the network parameters of the generator.

In step 3, the two discriminator networks D _I and D _V have the same network structure, both composed of five-layer convolutional neural networks, the first four layers have the same structure, and the network structure is 3 × 3 convolutional layers, batch normalization layer and activation layer, the activation function of the activation layer is Leaky Relu; the last layer is a fully connected layer, the output is the classification result of the input, so as to predict whether the output is a fusion image or a source image;

The loss function of the discriminator whose input is the infrared image and the fused image is:

Wherein, D _I (I _I ) represents the discriminator discriminant value that takes infrared image as input, D _I (G(I _b , I _d )) represents the discriminator discriminant value that takes fusion image as input;

The loss function of the discriminator whose input is the visible light image and the fused image is:

Wherein, D _V ( _IV ) represents the discriminator discriminant value that takes the visible light image as input, and D _V (G(I _b , I _d )) represents the discriminator discriminant value that takes the fusion image as input;

和

更新网络参数的优化方法是SGD最终得到判别器的网络参数。Set a threshold for the output of the discriminator. When the output value of the discriminator is greater than the preset threshold, continue to update the network parameters until it is less than the preset threshold. In this process, after passing through the discriminator D _I and D _V , calculate the corresponding The discriminator loss function of

and

The optimization method for updating the network parameters is that SGD finally obtains the network parameters of the discriminator.

The beneficial effect of the present invention is that a multi-scale generation confrontation fusion network method for infrared and visible light images combines multi-scale decomposition and generation confrontation network, which not only optimizes the source image, but also integrates neural networks with good fusion effects. The network is applied to the fusion process. A method for multi-scale generative adversarial fusion network for infrared and visible light images. First, the base layer and detail layer of the image are obtained through edge filter holder and gradient filtering. The obtained image components retain the information we need to the greatest extent. Then, the two branch networks of the generator network in the generative adversarial network are used to fuse the base layer (structure information) and the detail layer (detail information) after multi-scale decomposition respectively, and finally the generated base layer image and detail layer image are added to get the final result. The fusion image is generated, and the two discriminator structures in the generative adversarial network are used to classify and discriminate the two source images and the fusion image. The image obtained by the fusion of the present invention retains the target information and texture information of the source image to the greatest extent, improves the quality of the fusion image, and thus provides more convenient conditions for subsequent target detection and recognition.

Description of drawings

Fig. 1 is the overall flow chart of the method for multi-scale generation confrontation fusion network for infrared and visible light images of the present invention;

Fig. 2 is the base layer and detail layer diagram after the source image bilateral filtering of the present invention;

Fig. 3 is the base layer after bilateral filtering of the present invention to obtain new base layer and detail layer diagram after gradient filtering;

Fig. 4 is the network structure diagram of the generator in the generation confrontation network of the present invention;

FIG. 5 is a network structure diagram of the discriminator in the Generative Adversarial Network of the present invention.

Detailed ways

The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

The method of the present invention is oriented to the multi-scale generation confrontation fusion network of infrared and visible light images; firstly, the source image is decomposed by the edge filter holder and gradient filtering to obtain the base layer and the detail layer of the image, and then the base layer and the detail layer are input into In the generative adversarial network, the generator network is fused, and then the fused image and the two source images are respectively input into the discriminator for discrimination to optimize the network parameters, and the final fused image is obtained to realize image fusion. The overall network structure of the algorithm is shown in Figure 1. The fusion process of infrared and visible light images based on the multi-scale decomposition of the generative adversarial network is mainly divided into the following three stages;

1) Multi-scale decomposition of source image

The multi-scale decomposition of the source image is mainly divided into three steps. First, the image is input to the edge-preserving filter (bilateral filtering) to obtain the base layer and the detail layer. The obtained base layer and detail layer are shown in Figure 2, and then the base layer is passed through Gradient filtering obtains a gradient map and a new base layer. Finally, the gradient map and the detail layer are added as a new detail layer, and a new base layer and a new detail layer are obtained as shown in Figure 3. The principles of bilateral filtering and gradient filtering are as follows:

Bilateral filtering is an edge-preserving filter, which can achieve the effect of maintaining edges and reducing noise and smoothness. Like other filtering principles, bilateral filtering also uses a weighted average method, where the weighted average of the brightness values of surrounding pixels represents the intensity of a pixel, and the weighted average used is based on a Gaussian distribution. Most importantly, the weights of bilateral filtering take into account not only the Euclidean distance of pixels (like ordinary Gaussian low-pass filtering, which only considers the effect of position on the central pixel), but also the radiance differences in the pixel range domain (such as volume Similarity, color intensity, depth distance, etc. between the pixels in the product kernel and the center pixel), these two weights are considered at the same time when calculating the center pixel. The filtering formula is as follows:

为滤波后图像In formula (1), I _q is the input image,

is the filtered image

Step 2, input the base layer obtained in step 1 to the gradient filter to obtain a gradient map and a new base layer, and add the gradient map and the detail layer of step 1 as a new detail layer. The principle of gradient filtering is as follows:

Gradient is simply derivation, three different filters: Sobel, Scharr and Laplacian; Sobel, Scharr are actually the first or second derivative; Scharr is the optimization of Sobel; Laplacian is the second derivative. The Sobel filter is used here, the purpose is to let the high frequency pass, block the low frequency, and make the edge more obvious to enhance the image. The specific principle is as follows:

The Sobel operator is a discrete difference operator, which is used to calculate the approximation of the gray level of the image brightness function. Using this operator at any point in the image will yield the corresponding grayscale vector or its normal vector.

The operator consists of two sets of 3x3 matrices, which are horizontal and vertical, respectively. By convolving them with the image plane, the approximation of the horizontal and vertical luminance differences can be obtained respectively. If A represents the original image, Gx and Gy represent the gray value of the image detected by the horizontal and vertical edges, respectively, and the formula is as follows:

G _X =G _x *A and G _Y =G _y *A (4)

2) Generative Adversarial Network Parameter Acquisition

Obtaining the network parameters of the discriminator: The base layer and the detail layer obtained in 1) are spliced together in the dimension of the image channel as the input of the generator. The network structure diagram of the generator is shown in Figure 4, which is composed of a two-stream network. It is followed by a convolution block. The upper and lower two networks of the dual-stream network are the same and consist of six-layer convolutional neural networks. The first four layers have the same structure. The network structure is a 3×3 convolutional layer, a batch normalization layer and an activation layer (the activation function is Leaky Relu); The latter two layers have the same structure, consisting of a 5×5 convolutional layer, a batch normalization layer, and an activation layer (the activation function is Leaky Relu). The network structure followed by the two-stream network consists of a 1×1 convolutional layer and an activation layer (the activation function is tanh), and the output of this layer is the final fusion image. Then the respective fusion results (the fusion map of the base layer and the detail layer) are spliced to obtain the final fusion image. In this process, after passing through the generator _G , the generator loss function LG is calculated, and at the same time, SGD (stochastic gradient descent) is used to update the network parameters to achieve the purpose of optimization, and the network parameters of the generator are obtained.

和

更新网络参数的优化方法是SGD(随机梯度下降)，最终得到判别器的网络参数。Discriminator network parameter acquisition: Because the source image is a pair, two discriminators are used, one is used to obtain the probability P _I that the fusion image is an infrared image, and the other is used to obtain the probability P _V that the fusion image is a visible light image. The two discriminator network structures are exactly the same, as shown in Figure 5. It consists of five layers of convolutional neural networks. The first four layers have the same structure. The network structure is a 3×3 convolution layer, a batch normalization layer and an activation layer (the activation function is Leaky Relu); the last layer is a fully connected layer. , the output is the classification result of the input. Two probabilities are obtained from the input source image and the fusion image. When the probability is greater than the preset threshold, the network parameters are continuously updated until the probability is less than the preset threshold. In this process, after passing through the discriminator D _I and D _V , the corresponding discriminator loss function is calculated

and

The optimization method for updating the network parameters is SGD (Stochastic Gradient Descent), and finally the network parameters of the discriminator are obtained.

和

设计如下：The loss function includes the generator loss function _LG and the loss functions of the two discriminators

and

The design is as follows:

The purpose of the generator loss function is to save more information about the source image, which consists of two parts: content loss and adversarial loss:

L _G =λL _content +L _Gen , (5)

where L _content is the content loss after comparing the generator input and output, L _Gen is the adversarial loss between the generator and the discriminator, and λ is a constant;

为梯度算子，ξ为常量；where H and W are the height and width of the image input by the generator, ||·|| ₂ is the calculation of the second norm, I _f is the output of the generator, i.e. the fusion image, I _b is the base layer of the input generator, and I _d is The detail layer of the input generator,

is the gradient operator, and ξ is a constant;

L _Gen = E[log(1-D _V (G(I _b , I _d )))]+E[log(1-D _I (G(I _b , I _d )))] (7)

D _V (G(I _b , I _d )) represents the discriminator discriminant value input with infrared image or fusion image, G(I _b , I _d ) represents the fusion image generated by the generator, D _I (G(I _{b )} , I _d )) represents the discriminator discriminant value that takes the visible light image or the fusion image as input.

Using two discriminators can effectively reduce the information loss of the fusion result, and its role is to allow the generator to save more source image information; the definition is as follows:

Among them, D _V ( _IV ) represents the discriminator discriminant value that takes the visible light image as input, D _V (G(I _b , I _d )) represents the discriminator discriminant value that takes the fusion image as input, D _I (I _I ) represents the discriminator discriminant value with infrared image as input, D _I (G(I _b , I _d )) represents the discriminator discriminant value with fusion image as input.

3) Fusion test network

Input the generator network parameters obtained in 2) into the generator network, first perform multi-scale decomposition on the test image, and then splicing the corresponding base layer and detail layer obtained from the decomposition and input them into the generator network, so that the generator's The output is the final fused image.

The method of the present invention for the multi-scale generation confrontation fusion network of infrared and visible light images, the flowchart is shown in Figure 1, and the specific implementation is carried out according to the following steps:

With reference to Figure 2, step 1, select several infrared and visible light image pairs from the standard training set, and then input the image pairs to the edge preserving filter (bilateral filtering) to obtain the base layer and the detail layer;

The filtering formula in step 1 is as follows:

in:

为滤波后图像，q是I_q的一个像素点，s是q的像素集合，p是q领域中的一个像素，

是部分输入图像块，

是

周边图像块，

是空间滤波器内核，

是距离滤波器内核，空间内核和距离内核通常都以高斯的方式表示；In formula (1), I _q is the input image,

is the filtered image, q is a pixel of I _q , s is the set of pixels of q, p is a pixel in the field of q,

is a partial input image patch,

Yes

surrounding image blocks,

is the spatial filter kernel,

is the distance filter kernel, the spatial kernel and the distance kernel are usually expressed in Gaussian form;

In formula (3), I _d0 is the detail layer obtained by bilateral filtering, and I _b0 is the obtained base layer.

With reference to Figure 3, step 2, input the base layer obtained in step 1 into the gradient filter to obtain a gradient map and a new base layer, and add the gradient map and the detail layer of step 1 as a new detail layer;

in step 2

The principle of gradient filtering is as follows:

Gradient filtering is simply the derivation of the image, including three different filters: Sobel, Scharr and Laplacian; Sobel and Scharr are the first derivative, Scharr is the optimization of Sobel, and Laplacian is the second derivative. The Sobel filter is used here, the purpose is to let high-frequency information pass, block low-frequency information, and make the edge more obvious to enhance the image. That

The specific principles are as follows:

The Sobel operator is a discrete difference operator, which is used to calculate the approximation of the gray level of the image brightness function. Using this operator at any point in the image will yield the corresponding grayscale vector or its normal vector.

The operator consists of two sets of 3x3 matrices, which are horizontal and vertical, respectively. By convolving them with the image plane, the approximation of the horizontal and vertical luminance differences can be obtained respectively. If A represents the original image, Gx and Gy represent the gray value of the image detected by the horizontal and vertical edges, respectively, and the formula is as follows:

G _x =G _x *A and G _y =G _y *A (4)

The gradient value of all pixels in the image is obtained by the following formula:

Then define a threshold Gmax (defined as 150 here), if the gradient value of the pixel is greater than the threshold, set it as white, otherwise it is black, and thus obtain the gradient map _IG ;

The base layer I _b0 obtained in step 1 is subjected to gradient filtering to obtain a gradient map I _G , and then the original base layer I _b0 is subtracted from the gradient map to obtain a new base layer I _b , and the gradient map I _G is added to the original detail layer I _d0 A new level of detail I _d is available.

With reference to Figure 4 and Figure 5, in step 3, the base layer and detail layer obtained in step 2 are input to the generator network G, and after the generator network G, the fusion image corresponding to the source image pair is obtained, and the generator loss function L _G is calculated, Update the generator network G parameters to obtain the final generator network parameters, input the source image and the fusion image to the discriminator network D for classification, calculate the discriminator loss function L _D , and update the discriminator network parameters to obtain The final discriminator network parameters;

In step 3, the generator network structure consists of a dual-stream network and a convolutional neural network followed by it. The upper and lower network structures in the dual-stream network are the same, both are six-layer convolutional neural networks, the first four layers have the same structure, and the network structure is 3 ×3 convolutional layer, batch normalization layer and activation layer, the activation function of this activation layer is Leaky Relu; the latter two layers have the same structure and consist of 5×5 convolutional layer, batch normalization layer and activation layer , the activation function of this activation layer is Leaky Relu, the network structure behind the two-stream network consists of a 1×1 convolution layer and an activation layer, the activation function of this activation layer is tanh, and the output of this layer of convolutional neural network is The final fused image.

The generator loss function _LG in step 3 is:

L _G =λL _content +L _Gen , (6)

where L _content is the content loss after comparing the generator input and output, L _Gen is the adversarial loss between the generator and the discriminator, and λ is a constant;

为梯度算子，ξ为常量；Among them, H and W are the height and width of the image input by the generator, ||·|| ₂ is the calculation of the second norm, I _f is the output of the generator, i.e. the fusion image, I _b is the base layer of the input generator, I _d is the detail layer of the input generator,

is the gradient operator, and ξ is a constant;

L _Gen = E[log(1-D _V (G(I _b , I _d )))]+E[log(1-D _I (G(I _b , I _d )))] (8)

D _V (G(I _b , I _d )) represents the discriminator discriminant value input with infrared image or fusion image, G(I _b , I _d ) represents the fusion image generated by the generator, D _I (G(I _{b )} , I _d )) represents the discriminator discriminant value that takes the visible light image or the fusion image as input.

Calculate the generator loss function L _G , and use SGD (Stochastic Gradient Descent) to update the network parameters to achieve the purpose of optimization, and obtain the network parameters of the generator.

In step 3, the two discriminator networks D _I and D _V have the same network structure, both composed of five-layer convolutional neural networks, the first four layers have the same structure, and the network structure is 3 × 3 convolutional layers, batch normalization layer and activation layer, the activation function of the activation layer is Leaky Relu; the last layer is a fully connected layer, which outputs the classification result of the input, so as to predict whether the output is a fusion image or a source image (there are two possibilities for the source image here , infrared and visible light images. Which one refers to depends on whether the discriminator is _DI or _DV , _DI refers to infrared images, and _DV refers to visible light images);

The loss function of the discriminator whose input is the infrared image and the fused image is:

Wherein, D _I (I _I ) represents the discriminator discriminant value that takes infrared image as input, D _I (G(I _b , I _d )) represents the discriminator discriminant value that takes fusion image as input;

The loss function of the discriminator whose input is the visible light image and the fused image is:

Wherein, D _V ( _IV ) represents the discriminator discriminant value that takes the visible light image as input, and D _V (G(I _b , I _d )) represents the discriminator discriminant value that takes the fusion image as input;

和

更新网络参数的优化方法是SGD(随机梯度下降)，最终得到判别器的网络参数。Set a threshold for the output of the discriminator. When the output value of the discriminator is greater than the preset threshold, continue to update the network parameters until it is less than the preset threshold. In this process, after passing through the discriminator D _I and D _V , calculate the corresponding The discriminator loss function of

and

The optimization method for updating the network parameters is SGD (Stochastic Gradient Descent), and finally the network parameters of the discriminator are obtained.

Step 4. Start training the network, and determine whether the iteration is over, that is, whether the current number of iterations has reached the set number of iterations, and the network parameters obtained when the number of iterations reaches the set number of iterations are used as the final network parameters, and the network parameters are saved;

Specifically, the generator network parameters obtained in step 4 are loaded into the generator network in the test network, and the tested infrared and visible light source images are subjected to multi-scale decomposition, that is, the filtering operations in steps 1 and 2, and then the decomposition The corresponding base layer and detail layer are spliced together as the input of the test network, and the obtained output is the final fusion image.

Step 5. Load the generator network parameters obtained in step 4 into the generator network in the test network, and perform multi-scale decomposition of the tested infrared and visible light source images, that is, the filtering operations in steps 1 and 2, and then decompose The corresponding base layer and detail layer are spliced together as the input of the test network, and the obtained output is the final fusion image.

Claims (6)

1. The method for multi-scale generation confrontation fusion network for infrared and visible light images, characterized in that it is specifically implemented according to the following steps: Step 1. Select several infrared and visible light image pairs from the standard training set, and then input the image pairs into the edge preserving filter to obtain the base layer and the detail layer; Step 2. Input the base layer obtained in step 1 into the gradient filter to obtain a gradient map and a new base layer, and add the gradient map and the detail layer of step 1 as a new detail layer; Step 3. Input the base layer and the detail layer obtained in step 2 to the generator network G, obtain the fusion image corresponding to the source image pair after passing through the generator network G, calculate the generator loss function L _G , and perform the generator network G parameters. update to obtain the final generator network parameters, input the source image and the fusion image to the discriminator network D for classification, calculate the discriminator loss function L _D , update the discriminator network parameters, and obtain the final discriminator network parameters; Step 4. Start training the network, and determine whether the iteration is over, that is, whether the current number of iterations has reached the set number of iterations, and the network parameters obtained when the number of iterations reaches the set number of iterations are used as the final network parameters, and the network parameters are saved; Step 5. Load the generator network parameters obtained in step 4 into the generator network in the test network, and perform multi-scale decomposition of the tested infrared and visible light source images, that is, the filtering operations in steps 1 and 2, and then decompose The corresponding base layer and detail layer are spliced together as the input of the test network, and the obtained output is the final fusion image. 2. the method for multi-scale generation confrontation fusion network oriented to infrared and visible light images according to claim 1, is characterized in that, in described step 1, filtering formula is as follows:

in:

In formula (1), I _q is the input image,

is the filtered image, q is a pixel of I _q , s is the set of pixels of q, p is a pixel in the field of q,

is a partial input image patch,

Yes

surrounding image blocks,

is the spatial filter kernel,

is the distance filter kernel, the spatial kernel and the distance kernel are usually expressed in Gaussian form;

In formula (3), I _d0 is the detail layer obtained by bilateral filtering, and I _b0 is the obtained base layer. 3. The method for multi-scale generation confrontation fusion network for infrared and visible light images according to claim 2, wherein in the step 2 The gradient value of all pixels in the image is obtained by the following formula:

Then define a threshold Gmax, if the gradient value of the pixel is greater than the threshold, set it as white, otherwise it is black, and thus obtain the gradient map _IG ; The base layer I _b0 obtained in step 1 is subjected to gradient filtering to obtain a gradient map I _G , and then the original base layer I _b0 is subtracted from the gradient map to obtain a new base layer I _b , and the gradient map I _G is added to the original detail layer I _d0 A new level of detail I _d is available. 4. The method for multi-scale generation confrontation fusion network for infrared and visible light images according to claim 3, characterized in that, in the step 3, the generator network structure is composed of a two-stream network and a convolutional neural network followed by it. , the upper and lower networks in the dual-stream network have the same structure, both are six-layer convolutional neural networks, the first four layers have the same structure, and the network structure is a 3×3 convolutional layer, a batch normalization layer and an activation layer. The activation function is Leaky Relu; the latter two layers have the same structure, consisting of a 5×5 convolution layer, a batch normalization layer and an activation layer. The activation function of the activation layer is Leaky Relu, and the network structure behind the two-stream network consists of 1 A ×1 convolutional layer and an activation layer are composed. The activation function of the activation layer is tanh, and the output of this layer of convolutional neural network is the final fusion image. 5. The method for multi-scale generation confrontation fusion network for infrared and visible light images according to claim 4, characterized in that, in the step 3, the generator loss function _LG is: L _G =λL _content +L _Gen , (6) where L _content is the content loss after comparing the generator input and output, L _Gen is the adversarial loss between the generator and the discriminator, and λ is a constant;

Among them, H and W are the height and width of the image input by the generator, ||·|| ₂ is the calculation of the second norm, I _f is the output of the generator, i.e. the fusion image, I _b is the base layer of the input generator, I _d is the detail layer of the input generator,

is the gradient operator, and ξ is a constant; L _Gen = E[log(1-D _V (G(I _b , I _d )))]+E[log(1-D _I (G(I _b , I _d )))] (8) D _V (G(I _b , I _d )) represents the discriminator discriminant value input with infrared image or fusion image, G(I _b , I _d ) represents the fusion image generated by the generator, D _I (G(I _{b )} , I _d )) represents the discriminator discriminant value input with visible light image or fusion image; Calculate the generator loss function L _G , and use SGD to update the network parameters to achieve the purpose of optimization, and obtain the network parameters of the generator. 6. The method for multi-scale generation adversarial fusion network for infrared and visible light images according to claim 5, characterized in that, in the step 3, the two discriminator networks D _I and D _V have the same network structure, both of which have the same network structure. It consists of a five-layer convolutional neural network. The first four layers have the same structure. The network structure is a 3×3 convolution layer, a batch normalization layer and an activation layer. The activation function of the activation layer is Leaky Relu; the last layer is full The connection layer outputs the classification result of the input, thereby predicting whether the output is a fusion image or a source image; The loss function of the discriminator whose input is the infrared image and the fused image is:

Wherein, D _I (I _I ) represents the discriminator discriminant value that takes infrared image as input, D _I (G(I _b , I _d )) represents the discriminator discriminant value that takes fusion image as input; The loss function of the discriminator whose input is the visible light image and the fused image is:

Wherein, D _V ( _IV ) represents the discriminator discriminant value that takes the visible light image as input, and D _V (G(I _b , I _d )) represents the discriminator discriminant value that takes the fusion image as input; Set a threshold for the output of the discriminator. When the output value of the discriminator is greater than the preset threshold, continue to update the network parameters until it is less than the preset threshold. In this process, after passing through the discriminator D _I and D _V , calculate the corresponding The discriminator loss function of

and

The optimization method for updating the network parameters is that SGD finally obtains the network parameters of the discriminator.

Images