CN112561846A - Method and device for training image fusion model and electronic equipment - Google Patents

Abstract

The application discloses a method for training an image fusion model and electronic equipment, and belongs to the field of image processing. The method comprises the following steps: acquiring a training data set, wherein the training data set comprises an image pair; inputting an image pair into an initial image fusion model, wherein the image fusion model is a deep neural network model comprising a shallow feature extraction network, a deep feature extraction network, a global feature fusion network and a feature reconstruction network; sequentially processing a shallow feature extraction network, a deep feature extraction network, a global feature fusion network and a feature reconstruction network to obtain a fusion image of the image pair; and calculating a difference value between the fused image and the image pair according to a preset loss function, updating the network parameters of the image fusion model according to the calculated difference value until the calculated difference value is smaller than a preset threshold value, and obtaining the trained image fusion model, wherein the loss function is used for calculating a structural loss value and a content loss value. The method and the device can improve the objective authenticity of the fusion image.

Description

Method, apparatus and electronic device for training image fusion model

technical field

The present application belongs to the technical field of image processing, and in particular relates to a method, apparatus and electronic device for training an image fusion model.

Background technique

With the rapid development of sensor imaging technology, it is difficult for single sensor imaging to meet the needs of daily applications, and multi-sensor imaging has led the technological innovation. Image fusion refers to the comprehensive processing of image information detected by multiple sensors to achieve a more comprehensive and reliable description of the detection scene.

Infrared and visible light are the most widely used image types in the field of image processing. Infrared images can efficiently capture the thermal radiation of the scene and identify bright targets in the scene. The visible light image has high resolution characteristics and can present detailed texture information of the scene. The image information of the two is highly complementary. sex. Therefore, by fusing the infrared image and the visible light image, a fusion image with rich scene information can be obtained, and the scene background and target can be described clearly and accurately.

At present, for the fusion task of infrared image and visible light image, the fusion algorithm based on multi-scale decomposition, such as wavelet transform fusion algorithm, is usually used to extract and decompose the image features, and then classify the features. Different feature fusion strategies are used to obtain multiple sets of fused features after inverse transformation, and transform the fused features back into the fused image.

However, the classical image fusion algorithm based on multi-scale decomposition usually uses a fixed transformation and feature extraction level to decompose and reconstruct the image. The artificially formulated feature fusion rules cause the fusion results to easily introduce artificial visual artifacts, destroy the objective authenticity of image information, and affect the effect of fusion images.

SUMMARY OF THE INVENTION

The purpose of the embodiments of the present application is to provide a method for training an image fusion model, which can solve the problem that the fusion of infrared images and visible light images in the prior art requires the use of artificially formulated feature fusion rules, resulting in the fusion results easily introducing artificial visual artifacts and damaging The objective authenticity of image information affects the effect of fused images.

In order to solve the above technical problems, this application is implemented as follows:

In a first aspect, an embodiment of the present application provides a method for training an image fusion model, the method comprising:

Obtaining a training data set, the training data set includes an image pair, and the image pair includes an infrared image and a visible light image in the same scene;

Inputting the image pair into an initial image fusion model, where the image fusion model is a deep neural network model comprising a shallow feature extraction network, a deep feature extraction network, a global feature fusion network, and a feature reconstruction network;

Through the sequential processing of the shallow feature extraction network, the deep feature extraction network, the global feature fusion network, and the feature reconstruction network, the fusion image of the image pair is obtained;

Calculate the difference value between the fused image and the image pair according to the preset loss function, and update the network parameters of the image fusion model according to the calculated difference value, until the calculated difference value is less than the preset threshold, and the training is completed The image fusion model of , the preset loss function is used to calculate the structure loss value and the content loss value.

In a second aspect, an embodiment of the present application provides an apparatus for training an image fusion model, the apparatus comprising:

a data acquisition module, used for acquiring a training data set, the training data set includes image pairs, and the image pairs include infrared images and visible light images in the same scene;

A data input module for inputting the image pair into an initial image fusion model, where the image fusion model is a deep neural network model including a shallow feature extraction network, a deep feature extraction network, a global feature fusion network, and a feature reconstruction network ;

a data processing module, configured to obtain a fusion image of the image pair through the sequential processing of the shallow feature extraction network, the deep feature extraction network, the global feature fusion network, and the feature reconstruction network;

A parameter adjustment module, configured to calculate the difference value between the fusion image and the image pair according to a preset loss function, and update the network parameters of the image fusion model according to the calculated difference value, until the calculated difference value is less than the predetermined value. A threshold is set to obtain a trained image fusion model, and the preset loss function is used to calculate the structure loss value and the content loss value.

In a third aspect, embodiments of the present application provide an electronic device, the electronic device includes a processor, a memory, and a program or instruction stored on the memory and executable on the processor, the program or instruction being The processor implements the steps of the method according to the first aspect when executed.

In a fourth aspect, an embodiment of the present application provides a readable storage medium, where a program or an instruction is stored on the readable storage medium, and when the program or instruction is executed by a processor, the steps of the method according to the first aspect are implemented .

In a fifth aspect, an embodiment of the present application provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is configured to run a program or an instruction, and implement the first aspect the method described.

In the embodiments of the present application, the fusion of infrared images and visible light images is realized based on a neural network model. The neural network model can decompose and extract more abundant and suitable image features by simulating the structure of human eyes, which can improve the accuracy of feature extraction. sex.

In addition, the present application forces the neural network to fit the most suitable image feature fusion rules and strategies by setting a preset loss function, so as to improve the fusion performance and efficiently fuse image information without adding artificial fusion rules and strategies. The preset loss function can be used to calculate the structure loss value and the content loss value. On the basis of keeping the structure information of the infrared image and the visible light image in the fusion image as much as possible, the fusion image can reflect the infrared image and the visible light image. Detailed content information.

Therefore, using the image fusion model trained in the present application to fuse infrared images and visible light images can avoid the introduction of artificial visual artifacts into the fusion image, improve the objective authenticity of the fusion image, and enable the fusion image to better retain structural features. and content features to further improve the effect of fused images.

Furthermore, the trained image fusion model is an end-to-end model. After the trained image fusion model is obtained, the image pair to be fused is input into the trained image fusion model, and the trained image can be passed through. The fusion model outputs fusion images, which can improve the convenience of image fusion operations.

Description of drawings

1 is a flow chart of steps of a method embodiment of a training image fusion model of the present application;

2 is a schematic diagram of an infrared image and a visible light image under a certain scene;

3 is a schematic diagram of a network structure comprising at least two layers of aggregated residual dense blocks according to the present application;

Fig. 4 is the model structure schematic diagram of a kind of image fusion model of the present application;

5 is a schematic flow chart of an image pair processed by each network module of an image fusion model according to the present application;

6 is a schematic diagram of a calculation flow of a pixel-level loss function of the present application;

7 is a schematic diagram of the calculation flow of a feature-level loss function of the present application;

8 is a schematic structural diagram of an embodiment of an apparatus for training an image fusion model of the present application;

9 is a schematic structural diagram of an electronic device of the present application;

FIG. 10 is a schematic diagram of a hardware structure of an electronic device implementing an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

The terms "first", "second" and the like in the description and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances so that the embodiments of the present application can be practiced in sequences other than those illustrated or described herein, and distinguish between "first", "second", etc. The objects are usually of one type, and the number of objects is not limited. For example, the first object may be one or more than one. In addition, "and/or" in the description and claims indicates at least one of the connected objects, and the character "/" generally indicates that the associated objects are in an "or" relationship.

The method for training an image fusion model provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings through specific embodiments and application scenarios thereof.

Referring to FIG. 1, a flow chart of steps of a method embodiment of a method for training an image fusion model of the present application is shown, including the following steps:

Step 101, obtaining a training data set, the training data set includes an image pair, and the image pair includes an infrared image and a visible light image in the same scene;

Step 102, inputting the image pair into an initial image fusion model, where the image fusion model is a deep neural network model comprising a shallow feature extraction network, a deep feature extraction network, a global feature fusion network, and a feature reconstruction network;

Step 103, through the sequential processing of the shallow feature extraction network, the deep feature extraction network, the global feature fusion network, and the feature reconstruction network, to obtain a fusion image of the image pair;

Step 104: Calculate the difference value between the fused image and the image pair according to a preset loss function, and update the network parameters of the image fusion model according to the calculated difference value, until the calculated difference value is less than a preset threshold, The trained image fusion model is obtained, and the preset loss function is used to calculate the structure loss value and the content loss value.

The present application provides a method for training an image fusion model, where the image fusion model is a deep neural network model, and the deep neural network model includes a shallow feature extraction network, a deep feature extraction network, a global feature fusion network, and a feature reconstruction network. The present application realizes the fusion of infrared images and visible light images based on a neural network model. The neural network model can decompose and extract more abundant and suitable image features by simulating the neuron structure of the human eye, which can improve the accuracy of feature extraction.

The image fusion model can be obtained by performing unsupervised or supervised training on an existing neural network according to a large amount of training data and machine learning methods. It should be noted that the embodiment of the present application does not limit the model structure and training method of the image fusion model. The image fusion model may be a deep neural network model that fuses multiple neural networks. The neural network includes but is not limited to at least one of the following or a combination, superposition and nesting of at least two: CNN (Convolutional Neural Network, convolutional neural network), LSTM (Long Short-TermMemory, long short-term memory) network, RNN (Simple Recurrent Neural Network, recurrent neural network), attention neural network, etc.

In the embodiment of the present application, the collected training data set is composed of a large number of image pairs, and each image pair includes an infrared image and a visible light image in the same scene. Referring to FIG. 2 , a schematic diagram of an infrared image and a visible light image in a certain scene is shown. As shown in Figure 2, Figures a and b are infrared images and visible light images in a certain scene, respectively, and Figures a and b can form an image pair.

Input each image pair in the training data set into the initial image fusion model in turn, and through the sequential processing of the shallow feature extraction network, the deep feature extraction network, the global feature fusion network, and the feature reconstruction network, the fusion image of the image pair can be obtained. .

Among them, the shallow feature extraction network is used to extract the basic feature map of the shallow layer of the image. The deep feature extraction network is used to further extract multi-level deep feature information from the basic shallow feature map extracted by the shallow feature extraction network. The global feature fusion network is used to further integrate the extracted image feature information, so that the global feature maps extracted by the fusion network can be integrated, and the information content of the multi-scale feature maps of the network can be deepened. The feature reconstruction network is used to reconstruct the fused image features to obtain a fused image of the image pair.

Finally, the difference value between the fused image and the image pair is calculated according to the preset loss function, and the network parameters of the image fusion model are updated according to the calculated difference value, until the calculated difference value is less than the preset threshold value, obtaining The trained image fusion model.

Among them, the structural loss value is used to represent the structural similarity of the two images, and this application uses the structural loss value to better preserve the structural information in the infrared image and the visible light image in the fusion image. The content loss value is used to represent the content similarity of two images. The purpose of this application is to use the content loss value to effectively calculate the pixel grayscale information between image pairs, reduce errors as much as possible, and enable the fusion image to save infrared images and visible light. Detailed content information in the image.

The present application forces the neural network to fit the most suitable image feature fusion rules and strategies by setting a preset loss function, so as to improve the fusion performance and efficiently fuse image information without adding artificial fusion rules and strategies. The preset loss function can be used to calculate the structure loss value and the content loss value. On the basis of keeping the structure information of the infrared image and the visible light image in the fusion image as much as possible, the fusion image can reflect the infrared image and the visible light image. Detailed content information.

Therefore, using the image fusion model trained in the present application to fuse infrared images and visible light images can avoid the introduction of artificial visual artifacts into the fusion image, improve the objective authenticity of the fusion image, and enable the fusion image to better retain structural features. and content features to further improve the effect of fused images.

In addition, the trained image fusion model is an end-to-end model. After the trained image fusion model is obtained, the image pair to be fused is input into the trained image fusion model, and then the trained image fusion model can be used. The model outputs fused images, which can improve the convenience of image fusion operations.

In an optional embodiment of the present application, in step 103, the fusion of the image pair is obtained by sequentially processing the shallow feature extraction network, the deep feature extraction network, the global feature fusion network, and the feature reconstruction network. images, including:

Step S11, performing channel concatenation processing on the image pair to obtain concatenated images;

Step S12, inputting the cascaded images into the shallow feature extraction network to extract a shallow feature map;

Step S13, inputting the shallow feature map into the deep feature extraction network to extract deep feature information;

Step S14, inputting the shallow feature map and the depth feature information into the global feature fusion network to integrate the shallow feature map and the depth feature information to obtain an integrated image;

Step S15: Input the integrated image into the feature reconstruction network to perform feature reconstruction on the integrated image to obtain a fusion image.

It should be noted that, before inputting the image pair into the initial image fusion model, the image pair can be precisely aligned, and then the aligned image pair can be input into the initial image fusion model to further improve the fusion effect.

After inputting the image pair into the initial image fusion model, the input infrared image and the visible light image are first connected through the channel cascade operation to obtain a cascaded image, which can be understood as the channel cascade of the infrared image and the visible light image. Image container I _v,r . In the embodiment of the present application, the infrared image is represented by I _r , the visible light image is represented by I _v , and the channel cascade mathematical expression 1-1 of both I _r and I _v is as follows:

I _v,r =Concat(I _v ,I _r ) (1-1)

Among them, Concat is the image channel cascade function. Then, the cascaded images I _v,r are input into the shallow feature extraction network (this application will denote the shallow feature extraction network as BFEnet), and basic shallow feature extraction is performed to obtain the shallow features of the cascaded images I _v,r Atlas.

Further, the shallow feature extraction network may be a convolutional neural network. In an example, the shallow feature extraction network may include two convolutional layers, and the size of the convolution kernel used in each convolutional layer is 3×3 pixels, so as to extract from the concatenated I _v,r The basic shallow feature map of the image, the operation of the convolution layer is expressed as formula 1-2:

为BFEnet模块中第i个卷积层提取的浅层特征图谱。in,

Shallow feature map extracted for the ith convolutional layer in the BFEnet module.

It can be understood that the above shallow feature extraction network includes two convolution layers, and the size of the convolution kernel used in each convolution layer is 3×3 pixels, which is only an application example of the embodiment of the present application. The embodiments of the present application do not limit the number of convolutional layers included in the shallow feature extraction network and the size of the convolution kernel used by each convolutional layer.

The shallow feature map is then passed to a deep feature extraction network (denoted as RXDBs in this application) to further extract multi-level deep feature information in BFEnet. The module structure of the deep feature extraction network improves infrared images and visible light images in a targeted manner, and further improves the feature circulation inside the module, so that the shallow and deep features inside the module become part of the output of the module, so that the entire network module has Better image feature extraction capability.

Next, the extracted feature map is input into a global feature fusion network (this application will denote the global feature fusion network as GFF) to perform global feature fusion, and further integrate the extracted image feature information.

In an optional embodiment of the present application, the deep feature extraction network includes at least two layers of aggregated residual dense blocks for extracting deep feature information, and the shallow feature map and the deep feature information are input The global feature fusion network integrates the shallow feature map and the depth feature information to obtain an integrated image, including:

Step S21, performing feature fusion on the depth feature information extracted from each layer of aggregated residual dense blocks to obtain global features;

Step S22: Perform feature fusion on the global feature and the shallow feature map extracted by the shallow feature extraction network to obtain an integrated image.

In the embodiment of the present application, the deep feature extraction network RXDBs includes at least two layers of aggregated residual dense blocks RXDB for extracting deep feature information, and the global feature fusion network GFF includes a dense feature fusion module (this application fuses dense features The module is denoted as DFF) and the global residual learning module (the global residual learning module is denoted as GRL in this application). In the embodiment of the present application, the depth feature information extracted from each layer of aggregated residual dense blocks is input into the dense feature fusion module DFF for feature fusion to obtain global features; and the global features and the shallow feature extraction network extracted by the shallow feature extraction network are The layer feature map is input into the global residual learning module GRL for feature fusion to obtain an integrated image.

Referring to FIG. 3, a schematic diagram of a network structure including at least two layers of aggregated residual dense blocks according to the present application is shown, and the structure calculation process is represented by formulas 1-3:

为第i个聚集残余致密块提取的图像特征。稠密特征融合模块(DFF)通过整合所有聚集残余致密块的特征得到全局特征。进一步地，为了避免模型过拟合并且提高计算效率，本申请在DFF模块内部进行特征图谱降维，具体计算如公式1-4所示：in,

Image features extracted for the ith aggregated residual dense block. The dense feature fusion module (DFF) obtains global features by integrating the features of all aggregated residual dense blocks. Further, in order to avoid model overfitting and improve computational efficiency, the present application performs feature map dimension reduction inside the DFF module, and the specific calculation is shown in formulas 1-4:

表示由6层聚集残余致密块提取的特征图谱，

为GFF模块中DFF得到的图像特征。全局残余学习模块GRL充分利用前面所有特征，其中包含BFENet首个卷积层所获得的基础浅层特征图谱，使得融合网络所提取的全局特征图谱得以整合，深化网络多尺度特征图谱的信息量，得到整合后的图像。全局残余学习模块GRL计算如公式1-5所示：in,

represents the feature map extracted from the 6-layer aggregated residual dense block,

It is the image feature obtained by DFF in the GFF module. The global residual learning module GRL makes full use of all the previous features, including the basic shallow feature map obtained by the first convolutional layer of BFENet, so that the global feature map extracted by the fusion network can be integrated, and the information content of the multi-scale feature map of the network can be deepened. Get the integrated image. The global residual learning module GRL is calculated as shown in Equation 1-5:

为GFF模块中GRL得到的图像特征。in,

It is the image feature obtained by GRL in the GFF module.

It should be noted that the deep feature extraction network includes 6 layers of aggregated residual dense blocks, which is only an application example of this embodiment of the present application. Unrestricted.

Finally, the integrated image is input into a feature reconstruction network (referred to as Rbnet in this application) to perform feature reconstruction on the integrated image to obtain a fusion image.

The feature reconstruction network may be a convolutional neural network. In an example, the feature reconstruction network includes three convolutional layers, the size of the convolution kernel of the first two convolutional layers is 3×3 pixels, and the size of the convolutional kernel of the last convolutional layer is 1×1 pixel. , in order to reconstruct the local features of the fused image. The calculation formula of the feature reconstruction network is shown in 1-6:

Among them, If represents the _fused image.

It can be understood that the above feature reconstruction network includes 3 convolution layers, the size of the convolution kernel of the first two layers is 3 × 3 pixels, and the size of the convolution kernel of the last layer is 1 × 1 pixels, which is only an embodiment of this application. An example of application. The embodiments of the present application do not limit the number of convolutional layers included in the feature reconstruction network and the size of the convolutional kernel used by each convolutional layer.

Referring to FIG. 4 , a schematic structural diagram of an image fusion model of the present application is shown. And referring to FIG. 5 , a schematic flow chart of an image pair processed by each network module of the image fusion model according to the present application is shown.

The image fusion model of the present application sequentially processes the image pairs through the shallow feature extraction network, the deep feature extraction network, the global feature fusion network, and the feature reconstruction network to obtain a fusion image of the image pair. In this process, The middle layer of the image fusion model can reuse image features to increase the reusability and circulation of features. While improving the efficient fusion of image features, it can further reduce the size and computational complexity of the network model.

In an optional embodiment of the present application, the preset loss function includes a structure loss function and a content loss function, and the difference value between the fused image and the image pair is calculated according to the preset loss function, include:

Step S31: Perform a weighted average calculation on the first structure loss and the second structure loss through the structure loss function to obtain a structure loss value of the fusion image, where the first structure loss is the infrared image and the fusion image The structure loss between the visible light image and the fusion image is the second structure loss;

Step S32: Perform a weighted average calculation on the first content loss and the second content loss through the content loss function to obtain a content loss value of the fused image, where the first content loss is the infrared image and the fused image The content loss between the two, the second content loss is the content loss between the visible light image and the fusion image;

Step S33: Perform weighted calculation on the structure loss value and the content loss value of the fused image to obtain a difference value between the fused image and the image pair.

It should be noted that, this embodiment of the present application does not limit the sequence of step S31 and step S32.

In the embodiment of the present application, the preset loss function is used to calculate the structure loss value and the content loss value. The structure loss value is used to better preserve the structural information in the infrared and visible images in the fused image. The content loss value is used to preserve detailed content information in the infrared and visible images in the fused image.

An efficient network loss function can quickly reduce the gap between the results obtained by the neural network and the expected real target. For this reason, in view of the characteristics of the infrared image and visible light image fusion task, this application designs a preset loss function as shown in formula 1-7:

L=αL _ssim +βL _content (1-7)

Among them, α and β are trainable parameters, L _ssim is the structure loss function, which is used to calculate the structure loss value, and L _content is the content loss function, which is used to calculate the content loss value. L _ssim and L _content are shown in equations 1-8 and 1-9, respectively:

L _ssim =1-SSIM(A,B) (1-8)

L _content =||AB|| ₂ (1-9)

SSIM(A,B) represents the structural similarity (SSIM) between image A and image B. ||·|| ₂ represents the l ₂ norm operation, which is used to calculate the Euclidean distance between image A and image B.

In practical applications, the brightness information of an object is related to the illuminance and reflection coefficient, and the structure of the object in the scene and the illuminance are independent of each other, and the reflection coefficient is related to the object target itself. Therefore, the SSIM definition can reflect the structural information in an image by separating the influence of illumination on the object. The brightness contrast is often measured by the average gray level of the two images. The average gray level and standard deviation of the images are as follows Equations 1-10 and 1-11 are shown.

In the above formula, x _i represents the gray value of the corresponding position of the image, N represents the number of pixels included in the image, μ _x is the average gray value of the calculated image, and σ _x is the calculated image standard deviation.

Therefore, the present application can smoothly introduce the scene highlight target in the infrared image while retaining the background information with rich texture details, and can generate a smoother and more natural fusion image.

In this embodiment of the present application, the structure loss between the infrared image and the fused image, and the structural loss between the visible light image and the fused image are weighted and averaged by the structure loss function, and the structure loss value of the fused image is obtained.

The content loss between the infrared image and the fused image, and the content loss between the visible light image and the fused image are weighted and averaged by a content loss function to obtain a content loss value of the fused image.

The weighted calculation is performed on the structure loss value and the content loss value of the fused image to obtain the difference value between the fused image and the image pair, and the network parameters of the image fusion model are iteratively updated according to the difference value, Until the calculated difference value is smaller than the preset threshold, the trained image fusion model can be obtained.

Further, the embodiment of the present application designs two different calculation strategies for the preset loss function, which are a pixel-level loss function and a feature-level loss function, respectively. Each of them will be described below.

In an optional embodiment of the present application, the weighted average calculation is performed on the first structure loss and the second structure loss by using the structure loss function to obtain the structure loss value of the fused image, including:

Perform a weighted average calculation on the pixel-level first structure loss and the second structure loss through the structure loss function to obtain a pixel-level structure loss value;

The weighted average calculation is performed on the first content loss and the second content loss through the content loss function to obtain the content loss value of the fused image, including:

Perform a weighted average calculation on the pixel-level first content loss and the second content loss through the content loss function to obtain a pixel-level content loss value;

The weighted calculation is performed on the structure loss value and the content loss value of the fused image to obtain the difference value between the fused image and the image pair, including:

A weighted calculation is performed on the pixel-level structure loss value and the pixel-level content loss value to obtain a pixel-level difference value between the fused image and the image pair.

The calculation strategy of pixel-wise loss function L _Pixel-wise is a weighted combination of L _ssim and L _content directly acting on the input image (infrared image and visible light image) and output image (fused image). The calculation formula is shown in 1-12:

L _Pixel-wise = αL _ssim + βL _content (1-12)

The L _ssim in 1-12 is used to calculate the pixel-level structural similarity between the input image and the output image, that is, to calculate the pixel-level structural loss value of the fused image, which is determined by the pixel-level first structural loss L _ssim . _-r and pixel-level second structural loss L _ssim-v weighted average. Among them, the first pixel-level structural loss L _ssim-r is calculated as shown in Equation 1-13, and the pixel-level second structural loss L _ssim-v is calculated as shown in Equation 1-14. The pixel-level structural loss of the fused image As shown in Equation 1-15:

L _ssim-r =1-SSIM(I _r ,I _f ) (1-13)

L _ssim-v =1-SSIM(I _v ,I _f ) (1-14)

L _ssim =λL _ssim-r +(1-λ)L _ssim-v (1-15)

Similarly, L _content in 1-12 is used to calculate the pixel-level content similarity between the input image and the output image, that is, to calculate the pixel-level content loss value of the fused image, which is determined by the pixel-level first content. A weighted average of the loss L _content-r and the pixel-level second content loss L _content-v . The calculation of the pixel-level first content loss L _content-r is shown in formula 1-16, the calculation of the pixel-level second content loss L _content-v is shown in formula 1-17, and the pixel-level content loss of the fused image As shown in Equation 1-18:

L _content-r =||I _f -I _r || ₂ (1-16)

L _content-v =||I _f -I _v || ₂ (1-17)

L _content =δL _content-r +(1-δ)L _content-v (1-18)

Referring to FIG. 6 , a schematic diagram of a calculation flow of a pixel-level loss function of the present application is shown.

The pixel-level calculation strategy provides a relatively rough estimation of the network loss function. In order to obtain more detailed loss information, the present application preferably adopts the feature-level loss function calculation strategy in network training.

In an optional embodiment of the present application, before calculating the difference value between the fused image and the image pair according to a preset loss function, the method further includes:

Step S41, using a preset neural network to extract a depth feature map from the infrared image, the visible light image, and the fused image, respectively;

Step S42, based on the depth feature map of the infrared image and the depth feature map of the fused image, calculate the first structural loss of the feature level, and based on the depth feature map of the visible light image and the depth feature map of the fused image , calculate the feature-level second structure loss;

Step S43, based on the depth feature map of the infrared image and the depth feature map of the fused image, calculate the first content loss of the feature level, and based on the depth feature map of the visible light image and the depth feature map of the fused image , calculate the feature-level second content loss;

The weighted average calculation of the first structure loss and the second structure loss through the structure loss function to obtain the structure loss value of the fused image, including:

A weighted average calculation is performed on the first structure loss of the feature level and the second structure loss of the feature level through the structure loss function to obtain a structure loss value of the feature level;

The weighted average calculation is performed on the first content loss and the second content loss through the content loss function to obtain the content loss value of the fused image, including:

Perform a weighted average calculation on the feature-level first content loss and the feature-level second content loss by using the content loss function to obtain a feature-level content loss value;

The weighted calculation is performed on the structure loss value and the content loss value of the fused image to obtain the difference value between the fused image and the image pair, including:

A weighted calculation is performed on the feature-level structure loss value and the feature-level content loss value to obtain a feature-level difference value between the fused image and the image pair.

The feature-level computing strategy uses a preset neural network to perform feature extraction on the image pairs that need to calculate the loss, and obtains two sets of corresponding deep feature maps, and evaluates the loss function from a new perspective, that is, the loss function is calculated in the calculated feature map.

The preset neural network is used to extract the depth feature map of the image, and the preset network may be a pre-trained neural network. In an example, the preset network may be a VGG19 network. The VGG network is proposed by the VisualGeometry Group of Oxford. The VGG network has two structures, namely VGG16 and VGG19. The difference between the two networks is the depth. It can be understood that the embodiment of the present application does not limit the types of the preset neural network.

The calculation of the feature-level loss function L _Feature-wise is shown in Equation 1-19:

L _Feature-wise = αL _ssim + βL _content (1-19)

The L _ssim in 1-19 is used to calculate the feature-level structural similarity of the input image and the output image, that is, to calculate the feature-level structural loss value of the fused image, which is determined by the feature-level first structural loss L _{ssim -r} and feature-level second structural loss L _ssim-v weighted average obtained. Among them, the first feature-level structural loss L _ssim-r is calculated as shown in formula 1-20, and the second feature-level structural loss L _ssim-v is calculated as shown in formula 1-21. The feature-level structural loss of the fused image As shown in Equation 1-22:

L _ssim =λL _ssim-r +(1-λ)L _ssim-v (1-22)

分别表示VGG19网络从红外图像、可见光图像和融合图像中提取的深度特征图谱。Among them, the feature map of the 'ReLU3_2' layer of the VGG19 network is identified as m∈{1,2,...,M}, M=256.

denote the deep feature maps extracted by the VGG19 network from infrared images, visible light images, and fused images, respectively.

Similarly, L _content in 1-19 is used to calculate the feature-level content similarity between the input image and the output image, that is, to calculate the feature-level content loss value of the fused image, which is determined by the feature-level first content. The loss L _content-r and the feature-level second content loss L _content-v are weighted averaged. The calculation of the feature-level first content loss L _content-r is shown in formula 1-23, the calculation of the feature-level second content loss L _content-v is shown in formula 1-24, and the feature-level content loss of the fused image As shown in Equation 1-25:

L _content =δL _content-r +(1-δ)L _content-v (1-25)

Referring to FIG. 7 , a schematic diagram of the calculation flow of a feature-level loss function of the present application is shown.

In an application example of this application, the parameter settings of the convolution layer of the network framework related modules in network training are shown in Table 1, and the parameter settings of the internal convolution layer of the aggregated residual dense block RXDB are shown in Table 2. In order to reduce the computational cost, the embodiment of the present application can normalize the pixel value of each training image block pair, and set the learning rate to a constant piecewise decay. When the number of network training iterations reaches [5000, 10000, 14000 , 18000, 20000, 24000], set the network training learning rate to the corresponding value of [0.01, 0.007, 0.005, 0.0025, 0.001, 0.0001, 0.00005].

Table 1

Table 2

It can be understood that the above parameter setting is only an application example of the present application, and the present application does not limit the parameter setting of the image fusion model in practical application.

In an optional embodiment of the present application, after obtaining the trained image fusion model, the method further includes:

Step S51, inputting the image pair to be fused into the trained image fusion model, and the image pair to be fused includes an infrared image and a visible light image in the same scene;

Step S52, outputting a fused image through the image fusion model completed by the training.

After the training of the image fusion model is completed, the image fusion process can be performed by using the trained image fusion model. In the embodiment of the present application, the image fusion model is an end-to-end model, and the infrared image and the visible light image in the same scene to be fused are used as the image pair to input the image fusion model after training, and then the fusion model can be output.

Further, before inputting the image pairs to be fused into the trained image fusion model, the image pairs can be precisely aligned, and then the aligned image pairs can be input into the trained image fusion model to improve the fusion effect.

In summary, the present application provides a method for training an image fusion model, and the trained image fusion model can effectively realize the fusion of infrared images and visible light images, and effectively improve image feature analysis and performance without introducing artificial fusion rules. The extraction effect further enhances the richness of fusion features, making the fusion image have better imaging performance. While retaining the background information of rich texture details, it smoothly introduces the scene highlight target in the infrared image to generate a smooth and natural fusion image. . At the same time, the introduction of artificial visual artifacts can be avoided, the objective authenticity of the fused image can be provided, and the effect of the fused image can be improved.

It should be noted that, in the method for training an image fusion model provided by the embodiments of the present application, the execution subject may be a device for training an image fusion model, or the control of the method for executing a method for training an image fusion model in the device for training an image fusion model. module. In the embodiments of the present application, the apparatus for training an image fusion model provided by the embodiments of the present application is described by taking the method for training an image fusion model performed by an apparatus for training an image fusion model as an example.

Referring to FIG. 8 , a schematic structural diagram of an embodiment of an apparatus for training an image fusion model of the present application is shown, and the apparatus includes:

A data acquisition module 801, configured to acquire a training data set, where the training data set includes an image pair, and the image pair includes an infrared image and a visible light image in the same scene;

The data input module 802 is used to input the image pair into an initial image fusion model, where the image fusion model is a deep neural network including a shallow feature extraction network, a deep feature extraction network, a global feature fusion network, and a feature reconstruction network Model;

The data processing module 803 is configured to obtain the fusion image of the image pair through the sequential processing of the shallow feature extraction network, the deep feature extraction network, the global feature fusion network, and the feature reconstruction network;

A parameter adjustment module 804, configured to calculate the difference value between the fused image and the image pair according to a preset loss function, and update the network parameters of the image fusion model according to the calculated difference value, until the calculated difference value is less than A preset threshold is used to obtain a trained image fusion model, and the preset loss function is used to calculate the structure loss value and the content loss value.

Optionally, the data processing module 803 includes:

a cascade processing sub-module for performing channel cascade processing on the image pair to obtain cascaded images;

a shallow layer extraction sub-module for inputting the concatenated images into the shallow layer feature extraction network to extract a shallow layer feature map;

a deep extraction sub-module for inputting the shallow feature map into the deep feature extraction network to extract deep feature information;

A feature fusion submodule, configured to input the shallow feature map and the deep feature information into the global feature fusion network to integrate the shallow feature map and the deep feature information to obtain an integrated image;

The feature reconstruction sub-module is used for inputting the integrated image into the feature reconstruction network to perform feature reconstruction on the integrated image to obtain a fusion image.

Optionally, the preset loss function includes a structure loss function and a content loss function, and the parameter adjustment module includes:

The structure loss value calculation sub-module is used to perform a weighted average calculation on the first structure loss and the second structure loss through the structure loss function to obtain the structure loss value of the fusion image, and the first structure loss is the infrared The structure loss between the image and the fused image, and the second structure loss is the structure loss between the visible light image and the fused image;

A content loss value calculation submodule, configured to perform a weighted average calculation on the first content loss and the second content loss through the content loss function to obtain a content loss value of the fused image, where the first content loss is the infrared The content loss between the image and the fused image, the second content loss is the content loss between the visible light image and the fused image;

The difference value calculation sub-module is configured to perform weighted calculation on the structure loss value and the content loss value of the fused image to obtain the difference value between the fused image and the image pair.

Optionally, the structure loss value calculation sub-module is specifically configured to perform a weighted average calculation on the pixel-level first structure loss and the second structure loss through the structure loss function to obtain a pixel-level structure loss value;

The content loss value calculation submodule is specifically configured to perform a weighted average calculation on the pixel-level first content loss and the second content loss through the content loss function to obtain a pixel-level content loss value;

The difference value calculation sub-module is specifically configured to perform weighted calculation on the pixel-level structure loss value and the pixel-level content loss value to obtain a pixel-level difference value between the fused image and the image pair .

Optionally, the device further includes:

a feature extraction module for extracting depth feature maps from the infrared image, the visible light image, and the fused image respectively by using a preset neural network;

a first calculation module, configured to calculate the first structural loss at the feature level based on the depth feature map of the infrared image and the depth feature map of the fusion image, and the depth feature map based on the visible light image and the fusion image The depth feature map of , calculates the feature-level second structure loss;

The second calculation module is configured to calculate the first content loss at feature level based on the depth feature map of the infrared image and the depth feature map of the fused image, and based on the depth feature map of the visible light image and the fused image The depth feature map of , calculates the feature-level second content loss;

The structure loss value calculation sub-module is specifically configured to perform a weighted average calculation on the first structure loss of the feature level and the second structure loss of the feature level through the structure loss function, so as to obtain the structure loss value of the feature level ;

The content loss value calculation submodule is specifically configured to perform a weighted average calculation on the first content loss at the feature level and the second content loss at the feature level through the content loss function to obtain a content loss value at the feature level ;

The difference value calculation submodule is specifically configured to perform weighted calculation on the feature-level structure loss value and the feature-level content loss value to obtain the feature-level difference value between the fused image and the image pair .

Optionally, the deep feature extraction network includes at least two layers of aggregated residual dense blocks for extracting deep feature information, and the feature fusion sub-module includes:

The first fusion unit is used to perform feature fusion on the depth feature information extracted from each layer of aggregated residual dense blocks to obtain global features;

The second fusion unit is configured to perform feature fusion on the global feature and the shallow feature map extracted by the shallow feature extraction network to obtain an integrated image.

Optionally, the device further includes:

an image input module, configured to input the image pair to be fused into the image fusion model that has been trained, and the image pair to be fused includes an infrared image and a visible light image in the same scene;

The image fusion module is used for outputting the fusion image through the image fusion model completed by the training.

The present application provides a device for training an image fusion model. The image fusion model trained by the device can effectively realize the fusion of infrared images and visible light images, and effectively improve image feature analysis and performance without introducing artificial fusion rules. The extraction effect further enhances the richness of fusion features, making the fusion image have better imaging performance. While retaining the background information of rich texture details, it smoothly introduces the scene highlight target in the infrared image to generate a smooth and natural fusion image. . At the same time, the introduction of artificial visual artifacts can be avoided, the objective authenticity of the fused image can be provided, and the effect of the fused image can be improved.

The apparatus for training an image fusion model in this embodiment of the present application may be an apparatus, or may be a component, an integrated circuit, or a chip in a terminal. The apparatus may be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device may be a mobile phone, a tablet computer, a notebook computer, a palmtop computer, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (personal digital assistant). assistant, PDA), etc., the non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc. This application Examples are not specifically limited.

The apparatus for training an image fusion model in the embodiment of the present application may be an apparatus having an operating system. The operating system may be an Android (Android) operating system, an ios operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.

The apparatus for training an image fusion model provided in this embodiment of the present application can implement each process implemented by the method embodiment in FIG. 1 , and to avoid repetition, details are not repeated here.

Optionally, as shown in FIG. 9 , an embodiment of the present application further provides an electronic device 900, including a processor 901, a memory 902, a program or instruction stored in the memory 902 and executable on the processor 901, When the program or instruction is executed by the processor 901, each process of the above-mentioned method embodiment for training an image fusion model can be achieved, and the same technical effect can be achieved. To avoid repetition, details are not repeated here.

It should be noted that the electronic devices in the embodiments of the present application include the aforementioned mobile electronic devices and non-mobile electronic devices.

FIG. 10 is a schematic diagram of a hardware structure of an electronic device implementing an embodiment of the present application.

The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010, etc. part.

Those skilled in the art can understand that the electronic device 1000 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 1010 through a power management system, so that the power management system can manage charging, discharging, and power functions. consumption management and other functions. The structure of the electronic device shown in FIG. 10 does not constitute a limitation on the electronic device, and the electronic device may include more or less components than the one shown, or combine some components, or arrange different components, which will not be repeated here. .

The processor 1010 is configured to acquire a training data set, the training data set includes image pairs, and the image pairs include infrared images and visible light images in the same scene; the image pairs are input into the initial image fusion model, and the The image fusion model is a deep neural network model including a shallow feature extraction network, a deep feature extraction network, a global feature fusion network, and a feature reconstruction network; after the shallow feature extraction network, the deep feature extraction network, and the global feature fusion network , the sequential processing of the feature reconstruction network, the fusion image of the image pair is obtained; the difference value between the fusion image and the image pair is calculated according to the preset loss function, and the image fusion value is updated according to the calculated difference value. The network parameters of the model are calculated until the calculated difference value is less than the preset threshold, and the image fusion model after training is obtained, and the preset loss function is used to calculate the structure loss value and the content loss value.

The present application realizes the fusion of infrared images and visible light images based on a neural network model. The neural network model can decompose and extract more abundant and suitable image features by simulating the neuron structure of the human eye, which can improve the accuracy of feature extraction.

In addition, the present application forces the neural network to fit the most suitable image feature fusion rules and strategies by setting a preset loss function, so as to improve the fusion performance and efficiently fuse image information without adding artificial fusion rules and strategies. The preset loss function can be used to calculate the structure loss value and the content loss value. On the basis of keeping the structure information of the infrared image and the visible light image in the fusion image as much as possible, the fusion image can reflect the infrared image and the visible light image. Detailed content information.

Therefore, using the image fusion model trained in the present application to fuse infrared images and visible light images can avoid the introduction of artificial visual artifacts into the fusion image, improve the objective authenticity of the fusion image, and enable the fusion image to better retain structural features. and content features to further improve the effect of fused images.

Optionally, the processor 110 is further configured to perform channel concatenation processing on the image pair to obtain a concatenated image; input the concatenated image into the shallow feature extraction network to extract a shallow feature map; The shallow feature map is input into the deep feature extraction network to extract deep feature information; the shallow feature map and the deep feature information are input into the global feature fusion network to extract the shallow feature map and the depth feature information. The feature information is integrated to obtain an integrated image; the integrated image is input into the feature reconstruction network to perform feature reconstruction on the integrated image to obtain a fusion image.

The image fusion model of the present application sequentially processes the image pairs through the shallow feature extraction network, the deep feature extraction network, the global feature fusion network, and the feature reconstruction network to obtain a fusion image of the image pair. In this process, The middle layer of the image fusion model can reuse image features to increase the reusability and circulation of features. While improving the efficient fusion of image features, it can further reduce the size and computational complexity of the network model.

Optionally, the preset loss function includes a structure loss function and a content loss function, and the difference value between the fused image and the image pair is calculated according to the preset loss function, and the processor 110 is further configured to pass The structure loss function performs a weighted average calculation on the first structure loss and the second structure loss to obtain the structure loss value of the fusion image, and the first structure loss is the structure between the infrared image and the fusion image loss, the second structural loss is the structural loss between the visible light image and the fused image; the first content loss and the second content loss are weighted and averaged through the content loss function to obtain the fused image , the first content loss is the content loss between the infrared image and the fused image, and the second content loss is the content loss between the visible light image and the fused image; The structure loss value and the content loss value of the fused image are weighted and calculated to obtain the difference value between the fused image and the image pair.

Optionally, the processor 110 is further configured to perform a weighted average calculation on the pixel-level first structure loss and the second structure loss by using the structure loss function to obtain a pixel-level structure loss value; Perform a weighted average calculation on the pixel-level first content loss and the second content loss to obtain a pixel-level content loss value; perform weighted calculation on the pixel-level structure loss value and the pixel-level content loss value to obtain the pixel-level structure loss value and the pixel-level content loss value. Pixel-level difference values between the fused image and the pair of images.

The embodiment of the present application designs two different calculation strategies for the preset loss function, namely, a pixel-level loss function and a feature-level loss function, which are suitable for different requirements.

Optionally, the processor 110 is further configured to use a preset neural network to extract a depth feature map from the infrared image, the visible light image, and the fused image, respectively; based on the depth feature map of the infrared image and the depth of the fused image. feature map, calculate the first structure loss at feature level, and calculate the second structure loss at feature level based on the depth feature map of the visible light image and the depth feature map of the fusion image; based on the depth feature map of the infrared image and the depth feature map of the fused image, calculate the first content loss at the feature level, and calculate the second content loss at the feature level based on the depth feature map of the visible light image and the depth feature map of the fused image; The structure loss function performs a weighted average calculation on the first structure loss of the feature level and the second structure loss of the feature level to obtain the structure loss value of the feature level; A weighted average calculation is performed on the first content loss and the second content loss at the feature level to obtain the content loss value at the feature level; the weighted calculation is performed on the structure loss value at the feature level and the content loss value at the feature level to obtain the content loss value at the feature level. feature-level difference value between the fused image and the image pair.

The pixel-level calculation strategy provides a relatively rough estimation of the network loss function. In order to obtain more detailed loss information, the present application preferably adopts the feature-level loss function calculation strategy in network training to improve the accuracy of the fusion image.

Optionally, the deep feature extraction network includes at least two layers of aggregated residual dense blocks for extracting depth feature information, and the processor 110 is further configured to perform feature fusion on the depth feature information extracted from each layer of aggregated residual dense blocks, Obtaining a global feature; performing feature fusion on the global feature and the shallow feature map extracted by the shallow feature extraction network to obtain an integrated image.

Optionally, the processor 110 is further configured to input the image pair to be fused into the image fusion model completed by the training, and the image pair to be fused includes an infrared image and a visible light image in the same scene; The image fusion model of the output fusion image.

The trained image fusion model is an end-to-end model. After the trained image fusion model is obtained, the image pair to be fused is input into the trained image fusion model, and the trained image fusion model can be output. Fusion of images can improve the convenience of image fusion operations.

It should be understood that, in this embodiment of the present application, the input unit 1004 may include a graphics processor (Graphics Processing Unit, GPU) 1041 and a microphone 1042. camera) to process the image data of still pictures or videos. The display unit 1006 may include a display panel 1061, which may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 1007 includes a touch panel 1071 and other input devices 1072 . The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts, a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, physical keyboards, function keys (such as volume control keys, switch keys, etc.), trackballs, mice, and joysticks, which are not described herein again. Memory 1009 may be used to store software programs as well as various data, including but not limited to application programs and operating systems. The processor 1010 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the like, and the modem processor mainly processes wireless communication. It can be understood that, the above-mentioned modulation and demodulation processor may not be integrated into the processor 1010.

Embodiments of the present application further provide a readable storage medium, where a program or an instruction is stored on the readable storage medium, and when the program or instruction is executed by a processor, each process of the above-mentioned method embodiment for training an image fusion model is implemented, and The same technical effect can be achieved, and in order to avoid repetition, details are not repeated here.

Wherein, the processor is the processor in the electronic device described in the foregoing embodiments. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and the like.

An embodiment of the present application further provides a chip, where the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run a program or an instruction to implement the above method for training an image fusion model Each process of the embodiment can achieve the same technical effect, and to avoid repetition, it will not be repeated here.

It should be understood that the chip mentioned in the embodiments of the present application may also be referred to as a system-on-chip, a system-on-chip, a system-on-a-chip, or a system-on-a-chip, or the like.

It should be noted that, herein, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements, It also includes other elements not expressly listed or inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element. Furthermore, it should be noted that the scope of the methods and apparatus in the embodiments of the present application is not limited to performing the functions in the order shown or discussed, but may also include performing the functions in a substantially simultaneous manner or in the reverse order depending on the functions involved. To perform functions, for example, the described methods may be performed in an order different from that described, and various steps may also be added, omitted, or combined. Additionally, features described with reference to some examples may be combined in other examples.

From the description of the above embodiments, those skilled in the art can clearly understand that the method of the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is better implementation. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or in a part that contributes to the prior art, and the computer software product is stored in a storage medium (such as ROM/RAM, magnetic disk, CD-ROM), including several instructions to make a terminal (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in the various embodiments of this application.

The embodiments of the present application have been described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific embodiments, which are merely illustrative rather than restrictive. Under the inspiration of this application, without departing from the scope of protection of the purpose of this application and the claims, many forms can be made, which all fall within the protection of this application.

Claims (15)

1. a method for training an image fusion model, wherein the method comprises: Obtaining a training data set, the training data set includes an image pair, and the image pair includes an infrared image and a visible light image in the same scene; Inputting the image pair into an initial image fusion model, where the image fusion model is a deep neural network model comprising a shallow feature extraction network, a deep feature extraction network, a global feature fusion network, and a feature reconstruction network; Through the sequential processing of the shallow feature extraction network, the deep feature extraction network, the global feature fusion network, and the feature reconstruction network, the fusion image of the image pair is obtained; Calculate the difference value between the fused image and the image pair according to the preset loss function, and update the network parameters of the image fusion model according to the calculated difference value, until the calculated difference value is less than the preset threshold, and the training is completed The image fusion model of , the preset loss function is used to calculate the structure loss value and the content loss value. 2 . The method according to claim 1 , characterized in that, through the sequential processing of the shallow feature extraction network, the deep feature extraction network, the global feature fusion network, and the feature reconstruction network, the image pair is obtained. 3 . Fused images, including: Perform channel concatenation processing on the image pair to obtain concatenated images; inputting the concatenated images into the shallow feature extraction network to extract a shallow feature map; Inputting the shallow feature map into the deep feature extraction network to extract deep feature information; Inputting the shallow feature map and the deep feature information into the global feature fusion network to integrate the shallow feature map and the deep feature information to obtain an integrated image; The integrated image is input into the feature reconstruction network to perform feature reconstruction on the integrated image to obtain a fusion image. 3. The method according to claim 1, wherein the preset loss function comprises a structure loss function and a content loss function, and the calculation of the difference between the fused image and the image pair according to the preset loss function. Difference values, including: The first structure loss and the second structure loss are weighted and averaged by the structure loss function to obtain the structure loss value of the fusion image, and the first structure loss is the difference between the infrared image and the fusion image. structural loss, the second structural loss is the structural loss between the visible light image and the fusion image; A weighted average calculation is performed on the first content loss and the second content loss through the content loss function to obtain the content loss value of the fused image, where the first content loss is the difference between the infrared image and the fused image. content loss, the second content loss is the content loss between the visible light image and the fusion image; A weighted calculation is performed on the structure loss value and the content loss value of the fused image to obtain a difference value between the fused image and the image pair. 4. The method according to claim 3, wherein the weighted average calculation is performed on the first structure loss and the second structure loss by the structure loss function to obtain the structure loss value of the fused image, comprising: Perform a weighted average calculation on the pixel-level first structure loss and the second structure loss through the structure loss function to obtain a pixel-level structure loss value; The weighted average calculation is performed on the first content loss and the second content loss through the content loss function to obtain the content loss value of the fused image, including: Perform a weighted average calculation on the pixel-level first content loss and the second content loss through the content loss function to obtain a pixel-level content loss value; The weighted calculation is performed on the structure loss value and the content loss value of the fused image to obtain the difference value between the fused image and the image pair, including: A weighted calculation is performed on the pixel-level structure loss value and the pixel-level content loss value to obtain a pixel-level difference value between the fused image and the image pair. 5. The method according to claim 3, wherein before calculating the difference value between the fused image and the image pair according to a preset loss function, the method further comprises: Using a preset neural network to extract a depth feature map from the infrared image, the visible light image, and the fusion image, respectively; A first structural loss at feature level is calculated based on the depth feature map of the infrared image and the depth feature map of the fused image, and features are calculated based on the depth feature map of the visible light image and the depth feature map of the fused image The second structural loss of the stage; A feature-level first content loss is calculated based on the depth feature map of the infrared image and the depth feature map of the fused image, and features are calculated based on the depth feature map of the visible light image and the depth feature map of the fused image Level 2 loss of content; The weighted average calculation of the first structure loss and the second structure loss through the structure loss function to obtain the structure loss value of the fused image, including: A weighted average calculation is performed on the first structure loss of the feature level and the second structure loss of the feature level through the structure loss function to obtain a structure loss value of the feature level; The weighted average calculation is performed on the first content loss and the second content loss through the content loss function to obtain the content loss value of the fused image, including: Perform a weighted average calculation on the feature-level first content loss and the feature-level second content loss by using the content loss function to obtain a feature-level content loss value; The weighted calculation is performed on the structure loss value and the content loss value of the fused image to obtain the difference value between the fused image and the image pair, including: A weighted calculation is performed on the feature-level structure loss value and the feature-level content loss value to obtain a feature-level difference value between the fused image and the image pair. 6. The method according to claim 2, wherein the deep feature extraction network comprises at least two layers of aggregated residual dense blocks for extracting deep feature information, the shallow feature map and the depth The feature information is input into the global feature fusion network to integrate the shallow feature map and the deep feature information to obtain an integrated image, including: Perform feature fusion on the depth feature information extracted from each layer of aggregated residual dense blocks to obtain global features; Feature fusion is performed on the global feature and the shallow feature map extracted by the shallow feature extraction network to obtain an integrated image. 7. method according to claim 1, is characterized in that, after described obtaining the image fusion model that the training completes, also comprises: Inputting the image pair to be fused into the trained image fusion model, and the image pair to be fused includes an infrared image and a visible light image in the same scene; The image fusion model completed by the training outputs a fusion image. 8. A device for training an image fusion model, wherein the device comprises: a data acquisition module, used for acquiring a training data set, the training data set includes image pairs, and the image pairs include infrared images and visible light images in the same scene; A data input module for inputting the image pair into an initial image fusion model, where the image fusion model is a deep neural network model including a shallow feature extraction network, a deep feature extraction network, a global feature fusion network, and a feature reconstruction network ; a data processing module, configured to obtain a fusion image of the image pair through the sequential processing of the shallow feature extraction network, the deep feature extraction network, the global feature fusion network, and the feature reconstruction network; A parameter adjustment module, configured to calculate the difference value between the fusion image and the image pair according to a preset loss function, and update the network parameters of the image fusion model according to the calculated difference value, until the calculated difference value is less than the predetermined value. A threshold is set to obtain a trained image fusion model, and the preset loss function is used to calculate the structure loss value and the content loss value. 9. The device according to claim 8, wherein the data processing module comprises: a cascade processing sub-module for performing channel cascade processing on the image pair to obtain cascaded images; a shallow layer extraction sub-module for inputting the concatenated images into the shallow layer feature extraction network to extract a shallow layer feature map; a deep extraction sub-module for inputting the shallow feature map into the deep feature extraction network to extract deep feature information; A feature fusion submodule, configured to input the shallow feature map and the deep feature information into the global feature fusion network to integrate the shallow feature map and the deep feature information to obtain an integrated image; The feature reconstruction sub-module is used for inputting the integrated image into the feature reconstruction network to perform feature reconstruction on the integrated image to obtain a fusion image. 10. The apparatus according to claim 8, wherein the preset loss function comprises a structure loss function and a content loss function, and the parameter adjustment module comprises: The structure loss value calculation sub-module is used to perform a weighted average calculation on the first structure loss and the second structure loss through the structure loss function to obtain the structure loss value of the fusion image, and the first structure loss is the infrared The structure loss between the image and the fused image, and the second structure loss is the structure loss between the visible light image and the fused image; A content loss value calculation submodule, configured to perform a weighted average calculation on the first content loss and the second content loss through the content loss function to obtain a content loss value of the fused image, where the first content loss is the infrared The content loss between the image and the fused image, the second content loss is the content loss between the visible light image and the fused image; The difference value calculation sub-module is configured to perform weighted calculation on the structure loss value and the content loss value of the fused image to obtain the difference value between the fused image and the image pair. 11. The apparatus according to claim 10, wherein the structure loss value calculation sub-module is specifically configured to perform a weighted average calculation on the pixel-level first structure loss and the second structure loss through the structure loss function , get the pixel-level structure loss value; The content loss value calculation submodule is specifically configured to perform a weighted average calculation on the pixel-level first content loss and the second content loss through the content loss function to obtain a pixel-level content loss value; The difference value calculation sub-module is specifically configured to perform weighted calculation on the pixel-level structure loss value and the pixel-level content loss value to obtain a pixel-level difference value between the fused image and the image pair . 12. The apparatus of claim 11, wherein the apparatus further comprises: a feature extraction module for extracting depth feature maps from the infrared image, the visible light image, and the fused image respectively by using a preset neural network; a first computing module, configured to calculate the first structural loss at feature level based on the depth feature map of the infrared image and the depth feature map of the fusion image, and the depth feature map based on the visible light image and the fusion image The depth feature map of , calculates the feature-level second structure loss; The second calculation module is configured to calculate the first content loss at feature level based on the depth feature map of the infrared image and the depth feature map of the fused image, and based on the depth feature map of the visible light image and the fused image The depth feature map of , calculates the feature-level second content loss; The structure loss value calculation sub-module is specifically configured to perform a weighted average calculation on the first structure loss of the feature level and the second structure loss of the feature level through the structure loss function, so as to obtain the structure loss value of the feature level ; The content loss value calculation submodule is specifically configured to perform a weighted average calculation on the feature-level first content loss and the feature-level second content loss through the content loss function to obtain a feature-level content loss value ; The difference value calculation submodule is specifically configured to perform weighted calculation on the feature-level structure loss value and the feature-level content loss value to obtain the feature-level difference value between the fused image and the image pair . 13. The apparatus according to claim 9, wherein the deep feature extraction network comprises at least two layers of aggregated residual dense blocks for extracting deep feature information, and the feature fusion sub-module comprises: The first fusion unit is used to perform feature fusion on the depth feature information extracted from each layer of aggregated residual dense blocks to obtain global features; The second fusion unit is configured to perform feature fusion on the global feature and the shallow feature map extracted by the shallow feature extraction network to obtain an integrated image. 14. The apparatus of claim 8, wherein the apparatus further comprises: an image input module, configured to input the image pair to be fused into the image fusion model that has been trained, and the image pair to be fused includes an infrared image and a visible light image in the same scene; The image fusion module is used for outputting the fusion image through the image fusion model completed by the training. 15. An electronic device, characterized in that it comprises a processor, a memory, and a program or instruction stored on the memory and executable on the processor, and the program or instruction is implemented when executed by the processor The steps of training an image fusion model method according to any one of claims 1 to 7.

Images