CN114627035A - Multi-focus image fusion method, system, device and storage medium - Google Patents

Abstract

The invention discloses a multi-focus image fusion method, system, device and storage medium, belonging to the technical field of multi-focus image fusion. The method includes: acquiring a multi-focus image to be fused; inputting the multi-focus image into a pre-trained fusion Fusion is performed in the network model to obtain a fusion image; the fusion network model is constructed by the following methods: constructing a fusion network model by using an atrous convolutional network and a dense convolutional neural network based on an attention mechanism; Extracting a single defect that cannot effectively highlight the importance of salient features, realizes the preservation of rich detail information during multi-focus image fusion, and the fusion image obtained by fusion has better visual quality.

Description

A multi-focus image fusion method, system, device and storage medium

technical field

The invention relates to a multi-focus image fusion method, system, device and storage medium, and belongs to the technical field of multi-focus image fusion.

Background technique

In the process of digital image acquisition, due to the limitation of the depth of field of the optical sensor lens, it is difficult for the camera system to obtain panoramic depth images during imaging, and the problem that the scene within the depth of field is clear and the scene outside the depth of field is blurred often occurs. The region is not conducive to subsequent image understanding and application, and may also cause certain errors. The multi-focus image fusion technology can effectively solve this problem. It combines complementary images of the same target scene with different focus areas to effectively solve the problem of clear imaging of the whole scene, so that it contains more information on the same image and achieves practical use. At present, multi-focus image fusion technology has played a vital role in the fields of machine vision, remote sensing monitoring and military medicine.

Existing multi-focus image fusion techniques can be mainly divided into transform domain methods, spatial domain methods and deep learning methods. The transform domain method usually decomposes the original image into different transform coefficients, and then fuses these transform coefficients through the corresponding fusion rules, and finally inversely transforms the fused coefficients to obtain a fused image; the spatial domain method directly converts the source image. Pixels or regions are fused to extract clear pixels in the focal region; however, both the transform domain method and the spatial domain method need to manually design the activity level measurement of significant information and the fusion rules, which limits the universality of the fusion algorithm to a certain extent; Due to the powerful feature extraction and data representation capabilities of deep learning, multi-focus image fusion technology based on deep learning has also become popular; most of the current deep learning-based multi-focus image fusion technology The key is to use convolutional neural networks to The system accurately detects the focus area in the focus source image, and then fuses the focus areas from different source images to generate a clear image of the whole scene; compared with the traditional multi-focus image fusion technology, the multi-focus image fusion technology based on deep learning can achieve a certain The fusion quality is improved, but there are still some limitations, which are mainly reflected in: 1. The feature extraction scale of the source image is single; 2. The importance of salient features cannot be effectively highlighted.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a multi-focus image fusion method, system, device and storage medium, which solves the defect in the prior art that the source image feature extraction is single and cannot effectively highlight the importance of salient features, and realizes multi-focus image fusion. While retaining rich detail information, the visual quality of the fusion image obtained by fusion is better.

To achieve the above object, the present invention adopts the following technical solutions to realize:

In a first aspect, the present invention provides a multi-focus image fusion method, including:

Obtain the multi-focus image to be fused;

Input the multi-focus image into the pre-trained fusion network model for fusion to obtain the fusion image;

The fusion network model is constructed by the following method: constructing a fusion network model by using an atrous convolutional network and a dense convolutional neural network based on an attention mechanism.

In combination with the first aspect, further, it also includes the steps of establishing a training data set, including:

Images are extracted from the public dataset MS-COCO, and their size is cropped to a uniform size to obtain labeled images, which form a training dataset.

In combination with the first aspect, further, it also includes the step of preprocessing the training data set, including:

Gaussian blurring different regions of the label images in the training dataset.

In combination with the first aspect, it further includes the step of setting the loss function of the fusion network model, including:

Set the following loss function:

L=L _mse + αL _ssim + βL _per

L _ssim =1-SSIM(O,T)

表示融合图像通过VGG16提取的特征图第i通道中坐标为(x,y)的像素值，

表示标签图像通过VGG16提取的特征图第i通道中坐标为(x,y)的像素值，C_f、H_f和W_f分别表示任意特征图的通道数、高度和宽度。where L is the loss function, L _mse is the mean square loss function, L _ssim is the structural similarity loss function, L _per is the perceptual loss function, α and β are the balance parameters, O is the fusion image, T is the label image, SSIM ( O,T) represents the structural similarity between O and T,

Indicates the pixel value with coordinates (x, y) in the i-th channel of the feature map extracted by VGG16 of the fusion image,

Represents the pixel value with coordinates (x, y) in the i-th channel of the feature map extracted by VGG16 from the label image, and C _f , H _f and W _f represent the channel number, height and width of any feature map, respectively.

In combination with the first aspect, further, the fusion network model is trained by the following methods:

The fusion network model is trained using the constructed training data set in Pytorch, and the batch size is set to 8 during the training process.

In combination with the first aspect, further, it also includes the steps of optimizing the parameters of the fusion network model, including:

The parameters of the fusion network model are optimized by the Adam optimizer, where the initial learning rate of the Adam optimizer is set to 0.001.

In a second aspect, the present invention also provides a multi-focus image fusion system, including:

Acquisition module: used to acquire the multi-focus image to be fused;

Fusion module: It is used to input the multi-focus image into the pre-trained fusion network model for fusion to obtain a fusion image.

In a third aspect, the present invention also provides a multi-focus image fusion device, including a processor and a storage medium;

the storage medium is used for storing instructions;

The processor is adapted to operate in accordance with the instructions to perform the steps of the method according to any one of the first aspects.

In a fourth aspect, the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, implements the steps of any one of the methods in the first aspect.

Compared with the prior art, the beneficial effects achieved by the present invention are:

The invention provides a multi-focus image fusion method, system, device and storage medium. The multi-focus image is input into a pre-trained fusion network model for fusion to obtain a fusion image, and the fusion of the multi-focus image is realized; wherein the hole convolution The network expands the receptive field by increasing the expansion rate, and then extracts the multi-scale features in the source image (the multi-focus image to be fused) more comprehensively; the dense convolutional neural network can effectively solve the gradient disappearance problem of the deep network, and in order to further Highlighting the importance of salient features, an attention mechanism is introduced into the dense convolutional neural network to adaptively select salient features, thereby improving the fusion performance; in summary, the solution of the present invention can retain richness when fusing multi-focus images The visual quality of the fusion image obtained by fusion is better.

Description of drawings

1 is one of the flowcharts of a multi-focus image fusion method provided by an embodiment of the present invention;

2 is a schematic structural diagram of a fusion network model provided by an embodiment of the present invention;

FIG. 3 is the second flowchart of a multi-focus image fusion method provided by an embodiment of the present invention.

Detailed ways

The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention, and cannot be used to limit the protection scope of the present invention.

Example 1

As shown in FIG. 1, a multi-focus image fusion method provided by an embodiment of the present invention includes:

S1. Acquire a multi-focus image to be fused.

Acquire a partially blurred multi-focus image for subsequent fusion.

S2. Input the multi-focus image into the pre-trained fusion network model for fusion to obtain a fusion image.

Construct training dataset: Construct a labeled training dataset (multi-focus image dataset) based on the public dataset MS-COCO.

In practical applications, it is difficult to obtain multi-focus images and their paired panoramic deep images, for which the present invention constructs a set of simulated multi-focus image datasets (ie, training datasets).

Select 8000 high-definition natural images in the public dataset MS-COCO, uniformly crop them to 128×128 and use them as label images.

Preprocess the training data set: perform Gaussian blurring on the label image in different regions, specifically, do Gaussian blurring on the complementary region of the label image.

In order to simulate different degrees of blur caused by different depths of field, the present invention divides the selected 8000 label images into 4 groups on average, and uses Gaussian blur with Gaussian blur radius of 2, 4, 6 and 8 for blurring processing.

Construct a fusion network model using atrous convolutional networks, convolutional layers, and attention-based dense convolutional neural networks.

As shown in Figure 2, the fusion network model includes three parts, namely feature extraction, feature fusion and image reconstruction.

The feature extraction part includes two network branches, the two network branches share weights, each network branch consists of a multi-branch parallel atrous convolutional network, a 1×1 convolutional layer and a dense volume with an attention mechanism Constructed neural network.

The feature channel of the multi-branch parallel atrous convolutional network is set to 192, and the network consists of three atrous convolutions with kernel size of 3×3 and dilation rate of 1, 2 and 3, respectively.

The input channels and output channels of the dense convolutional neural network with attention mechanism are 64 and 256, respectively, and consist of 1 dense block and 3 Squeeze-Excitation-Block, where the dense block contains 3 3×3 convolutional layers, The output of each layer is cascaded as the input to the next layer.

A 1×1 convolutional layer is used to adjust the dimension of feature channels.

The feature fusion part consists of a "splicing" operation and a 1×1 convolutional layer, which mainly realizes feature fusion.

The feature fusion part performs "splicing" and 1 × 1 convolution operations on the features of the multi-focus image obtained by the feature extraction part to achieve feature fusion to obtain fusion features, where the input channel and output channel of the 1 × 1 convolution layer are 512 and 64.

The image reconstruction part mainly generates the fusion image from the fusion features. This part consists of 4 3×3 convolutional layers, and the number of feature channels is 64, 64, 64 and 3 respectively; each convolutional layer except the last layer adopts ReLU as activation function.

In order to make the reconstructed image more accurate, set the loss function of the above fusion network model, and set the following loss function:

L=L _mse + αL _ssim + βL _per

L _ssim =1-SSIM(O,T)

表示融合图像通过VGG16提取的特征图第i通道中坐标为(x,y)的像素值，

表示标签图像通过VGG16提取的特征图第i通道中坐标为(x,y)的像素值，C_f、H_f和W_f分别表示任意特征图的通道数、高度和宽度。where L is the loss function, L _mse is the mean square loss function, L _ssim is the structural similarity loss function, L _per is the perceptual loss function, α and β are the balance parameters, O is the fusion image, T is the label image, SSIM ( O,T) represents the structural similarity between O and T,

Indicates the pixel value with coordinates (x, y) in the i-th channel of the feature map extracted by VGG16 of the fusion image,

Represents the pixel value with coordinates (x, y) in the i-th channel of the feature map extracted by VGG16 from the label image, and C _f , H _f and W _f represent the channel number, height and width of any feature map, respectively.

In this embodiment, both α and β are 0.5.

The fusion network model is trained using the constructed training data set. During the training process, the hyperparameters of the fusion network model include batch size, initial learning rate, number of iterations (epoch), and learning rate decay. Strategy.

In this embodiment, Pytorch (a scientific computing library based on python) is used to implement the training of the fusion network model, and the program running environment is RTX 3080/10GB RAM, Intel Core i7-10700K@3.80GHz.

The batch size in the training process is set to 8, the parameters are optimized by the Adam optimizer, the initial learning rate of the optimizer is set to 0.001, and the cosine annealing decay method is used to adjust the learning rate, and the network is trained for a total of 500 epochs.

The fusion image can be obtained by inputting the multi-focus image into the pre-trained fusion network model for fusion.

In order to prove the effectiveness of the fusion technology proposed in the present invention, 8 mainstream multi-focus image fusion methods were selected to compare the Lytro multi-focus color image data set with the present invention, namely the NSCT method, the SR method, the IMF method, and the MWGF method. , CNN method, DeepFuse method, DenseFuse method (including DenseFuse-ADD method and DenseFuse-L1 method) and IFCNN-MAX method. All comparative methods were tested using default parameters provided in the literature.

In this embodiment, four objective indicators are used as quantitative indicators, namely, average gradient (AG), spatial frequency (SF), visual information fidelity (VIF) and edge retention (Q ^AB/F ). Table 1 shows The average index value on the Lytro data set; as can be seen from Table 1, the present invention obtains the best results on the three indexes of AG, SF and VIF, and also obtains the second best result after the CNN for the Q ^AB/F index Suboptimal results of the method; from the results, the present invention is a feasible and efficient multi-focus image fusion method.

Table 1 Average metric values on the Lytro dataset

Example 2

A multi-focus image fusion system provided by an embodiment of the present invention includes:

Acquisition module: used to acquire the multi-focus image to be fused;

Fusion module: It is used to input the multi-focus image into the pre-trained fusion network model for fusion to obtain a fusion image.

The fusion network model is constructed by the following method: constructing a fusion network model by using an atrous convolutional network and a dense convolutional neural network based on an attention mechanism.

Example 3

A multi-focus image fusion device provided by an embodiment of the present invention includes a processor and a storage medium;

the storage medium is used for storing instructions;

The processor is configured to operate in accordance with the instructions to perform the steps of the following methods:

Obtain the multi-focus image to be fused;

The multi-focus image is input into the pre-trained fusion network model for fusion, and the fusion image is obtained.

The fusion network model is constructed by the following method: constructing a fusion network model by using an atrous convolutional network and a dense convolutional neural network based on an attention mechanism.

Example 4

The computer-readable storage medium provided by the embodiment of the present invention stores a computer program on it, and when the program is executed by a processor, implements the steps of the following method:

Obtain the multi-focus image to be fused;

The multi-focus image is input into the pre-trained fusion network model for fusion, and the fusion image is obtained.

The fusion network model is constructed by the following method: constructing a fusion network model by using an atrous convolutional network and a dense convolutional neural network based on an attention mechanism.

Example 5

As shown in FIG. 3 , a multi-focus image fusion method provided by an embodiment of the present invention includes:

S1. Construct a training data set, and preprocess the training data set.

S2. Use the attention mechanism and dense convolutional neural network to build a fusion network model.

S3. Set the loss function of the fusion network model, and optimize the network parameters.

S4, using the training data set to train the fusion network model to obtain a trained fusion network model.

S5. Obtain the multi-focus image to be fused, and input the multi-focus image into the trained fusion network model to obtain a fusion image.

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

The above are only the preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, without departing from the technical principle of the present invention, several improvements and modifications can also be made. These improvements and modifications It should also be regarded as the protection scope of the present invention.

Claims (9)

1. A multi-focus image fusion method, comprising:

acquiring a multi-focus image to be fused;

inputting the multi-focus images into a pre-trained fusion network model for fusion to obtain a fusion image;

the fusion network model is constructed by the following method: and constructing a fusion network model by utilizing the void convolution network and the intensive convolution neural network based on the attention mechanism.

2. The method of claim 1, further comprising the step of creating a training data set, comprising:

and extracting images from the public data set MS-COCO, cutting the images into uniform sizes to obtain label images, and forming a training data set by the label images.

3. The method of claim 2, further comprising the step of preprocessing the training data set, comprising:

and performing Gaussian blur processing on different areas of the label images in the training data set.

4. The method of claim 2, further comprising a step of setting a loss function of the fusion network model, comprising:

the following loss function is set:

L＝L_mse+αL_ssim+βL_per

L_ssim＝1-SSIM(O,T)

where L is a loss function, L_mseIs a mean square loss function, L_ssimIs a loss function of structural similarity, L_perIs a perceptual loss function, alpha and beta are balance parameters, O is a fusion image, T is a tag image, SSIM (O, T) represents the structural similarity between O and T,

represents the pixel value with coordinates (x, y) in the ith channel of the feature map extracted by the fused image through VGG16,

a pixel value C representing coordinates (x, y) in the i channel of the feature map extracted by the VGG16 from the label image_f、H_fAnd W_fRespectively representing the number of channels, height and width of any feature map.

5. The method of claim 2, wherein the fusion network model is trained by:

and training the fusion network model by using the constructed training data set in the Pythrch, wherein the size of the batch in the training process is set as 8.

6. The method of claim 1, further comprising the step of optimizing parameters of the fusion network model, comprising:

and optimizing parameters of the fusion network model by adopting an Adam optimizer, wherein the initial learning rate of the Adam optimizer is set to be 0.001.

7. A multi-focus image fusion system, comprising:

an acquisition module: the multi-focus image fusion method comprises the steps of obtaining a multi-focus image to be fused;

a fusion module: and the fusion image processing method is used for inputting the multi-focus image into a pre-trained fusion network model for fusion to obtain a fusion image.

8. A multi-focus image fusion device is characterized by comprising a processor and a storage medium;

the storage medium is used for storing instructions;

the processor is configured to operate in accordance with the instructions to perform the steps of the method according to any one of claims 1 to 6.

9. Computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 6.

Images