AU2020100178A4

AU2020100178A4 - Multiple decision maps based infrared and visible image fusion

Info

Publication number: AU2020100178A4
Application number: AU2020100178A
Authority: AU
Inventors: Shuying Huang; Jiaxiang Liu; Wenying Wen; Yong Yang
Original assignee: Wen Wenying Dr
Current assignee: Wen Wenying Dr
Priority date: 2020-02-04
Filing date: 2020-02-04
Publication date: 2020-03-19
Anticipated expiration: 2028-02-04

Abstract

Abstract: A VGG-based multiple decision maps fusion method is proposed for infrared and visible image fusion. In our method, firstly, the infrared and visible images are fed into a pre-trained model of VGG-16 to extract features. Then, we design a feature representation method to construct saliency maps by those features. Next, multiple decision maps are constructed by saliency maps and guided filter. Finally, the final fusion image is achieved by weighting the source images based on multiple decision maps. To the best of our knowledge, it is the first introduction of decision map in the field of infrared and visible image fusion. Experimental results demonstrate that the proposed method can obtain better performance compared with state-of-the-art infrared and visible image fusion methods.

Description

BACKGROUND AND PURPOSE [0001] The infrared and visible image fusion is an important technique for computer vision. Infrared image captures the information of the target in low light and nighttime scene, but is not clear enough with few details. The visible image can record the texture details of the scene; however, the visible image has poor anti-interference ability. Different images of same scene can be processed, and significant features can be extracted and then integrated into a single image for human and machine perception through image fusion. The fusion result of infrared and visible images can retain both infrared target information and the visible texture information.

[0002] For decades, many kinds of image fusion methods have been proposed. Some classic methods include sparse representation (SR), discrete wavelet transform (DWT), nonsubsampled contourlet transform (NSCT), guided filter (GFF), convolutional sparse representation, and so on. Most of these methods are subject to complex fusion rules and pixel activity level measurement that are costly to calculate.

[0003] Deep learning can automatically leam pattern features and incorporate feature learning into the process of building model. Many image fusion methods have been developed based on deep learning. Liu et al. proposed the convolutional neural network (CNN) based image fusion method, which achieved good experimental result. However, the shallow features of this network are ignored, and some details of source images are missing. Ma et al. proposed a generative adversarial network (FusionGAN) for image fusion. In the FusionGAN, Generator generates fused images, and discriminator determines whether the result is qualified. However, due to the difficulty of the GAN convergence, its training and testing process are relatively unstable, and some texture information disappears. Li et al. proposed a CNN-based network called DenseFuse, in which the dense block is introduced to extract features. DenseFuse generated a fused image by encoding and decoding network. However, this method requires a long training time, and the features extracted are still not comprehensive enough because the training process is not targeted to infrared and

2020100178 18 Feb 2020 visible image. In general, image fusion based on deep learning requires a long training time and standard reference images, which are actually difficult to be obtained for infrared and visible images.

[0004] We propose a feature representation method. In our method, we firstly extract the different layer feature maps of infrared and visible image through the pre-trained model. Then, based on the feature maps, we design a feature representation way to build saliency maps. Thirdly, we employ the combined saliency map as the guidance image and adopt the guided filter on the source images to construct the decision maps for two kinds of images. Finally, we define a fusion strategy for multiple decision maps to obtain fusion image.

FEATURE EXTRACTION ALGORITHM DESCRIPTION [0001] In the deep learning methods, if considering the depth network to learn the characteristics of infrared and visible images, it is difficult to obtain standard reference images in this field. Therefore, instead of using the end-to-end fusion strategy, we utilize the learning ability of the pre-trained model to extract the salient regions of the two source images. Since the depth model of VGG-16 is trained on a large number of datasets, it has a strong feature extraction capability. VGG-16 has 13 convolution layers, and the ReLU activation function is not shown for brevity. For the VGG-16 convolutional layer, it contains five identical scales after four downsampling operation. We select the first layer under the identical scales to extract more spatial features. The convolution operations of VGG-16 is shown as follows. Here, let df denote the feature map extracted by the z-th layer and c is the channel, ce {1,2,-,C}.

Equation 1

2020100178 18 Feb 2020 where φ_ι. represents a convolution layer in the VGG-16 network, and ze {1,2,3,4,5} denotes the convll, conv_2_l, conv_3_l, conv_4_l, conv_5_l of different scales, respectively. As we know, in deep learning, the neural network learns features spontaneously. The shallow layers in the network tend to detect the edge of the image, and the detected content is comprehensive. With the deepening of the network level, the detected features become more abstract, and a lot of spatial information is ignored. So, the shallower layers convl l, conv_2_l extract more spatial information, and deeper layers conv_3_l, conv_4_l, conv_5_l extract more semantic information. In addition, in the VGG-16 model, there are downsampling operation in the network layer, which will result in the feature space size of the upper layer being too small. It is not conducive to the edge extraction of the effective information region, so convl l and conv_2_l are considered to extract features.

FEATURE REPRESENTATION ALGORITHM DESCRIPTION [0001] To exact the saliency features better, we design a feature representation method based on the feature maps. Saliency maps contain the saliency features of infrared and visible images and can be used to assist in further obtaining the decision maps. The feature representation adopt L2-norm to construct saliency maps of the infrared source image and the visible source image I₂.

[0002] Feature representation of source images are obtained by utilizing L2-norm to measure the pixel activity level. Through the pixel activity level, we can determine the saliency features in the source images and get the saliency maps. The activity level measurement is an important step in image fusion. It is proved the LI-norm of feature map d_k ^{l c} (x, y) can used to calculate the activity level of the source images. However, LI-norm causes sparseness of feature selection and some details are ignored, while L2-norm makes it easy to select more features. Therefore, L2-norm of feature map d_k ^{1 c} (x, y) is used to estimate the activity level of image. The larger the

2020100178 18 Feb 2020 value of L2-norm, the better the sharpness of the image and the richer the details. The pixel activity level can be reflected on the activity level map.

[0003] The activity level map T^l _k can be obtained by the following formula:

Equation 2 [0004] The activity level map reflects the activity level coefficient of image. The larger the coefficient, the more active this pixel is. We compare the activity level coefficient of infrared and visible images to get saliency features. Therefore, the saliency maps S(x,y) can be obtained by activity level maps of infrared and visible images.

if^'(x,y)> iF‘(x,y) otherwise

Equation 3 where A['(x,y) and A₂'(x,y) are activity level maps obtained by the z-th convolution layer of infrared and visible image input network, respectively.

[0005] Due to the impact of pooling operations in the network, the size of the saliency map of infrared and visible image is different in different layers. Therefore, the upsampling operation is used to ensure the same size of the two saliency maps. In order to explore the differences of saliency map of different layer, Figure 1 shows the significant area map of the camp image obtained with different convolutions of VGG-16. Figure 1 (a) is a saliency map obtained by convll layer, which contains more complete spatial information. The saliency map Figure 1(b) obtained from conv_2_l contains more texture information, while Figure 1 (c) and (d) obtained from conv_3_l and conv_4_l respectively contains more abstract semantic information and less details. Therefore, two saliency maps A and B are obtained with convl l and conv_2_l. Saliency maps A record spatial information of the source image, while

2020100178 18 Feb 2020 saliency map B record texture details of the source image. Therefore, we combine saliency maps A and B to get combined saliency map.

[0006] Combined saliency map S_F (x, y) is obtained by intersection strategy of the two saliency maps obtained with convll and conv_2_l. The mathematical formula is described as follows:

S_F (x, y) = S_A (x, y)A S_B (x, y) Equation 4 [0007] In order to prove that convl l and conv_2_l layers of extracted features are more conducive to the infrared and visible image fusion, we set up experiments to verify the effectiveness of the features extracted from convl l and conv_2_l by five metrics.

MULTIPLE DECISION MAPS AND FUSION STRATEGY [0001] Most fusion methods fuse two kinds of source images by using a single decision map to capture complementary information between images. Because the information of infrared image and visible image is not complementary, the fusion result obtained by only using one decision map cannot get complete and effective information of the two images. Therefore, we employ the combined saliency map to construct multiply decision maps to realize the fusion of the source images. As shown in the part of multiple decision map fusion, the fusion decision maps are constructed by employing the guided filtering method, which makes use of the combined saliency map as the guidance image to filter the two source images. Because the guided filtering algorithm can maintain the edges of output image similar to the guidance image. The gradient information of the filtered image can be determined by the gradient information of the guidance image. Therefore, the saliency areas in the filtered image can be consistent with the combined saliency map. Due to the different acquisition modes of the source images, the information of the images is not completely complementary. Therefore, we propose a fusion strategy of multiple

2020100178 18 Feb 2020 decision maps which can extract effective information from the source images respectively. The multiple decision maps can be obtained by the following formulas. The combined saliency map S_F (x, y) is used as the guidance image, and infrared image I, and visible image I₂ are used as input image respectively.

D = G_r (ή, S_F) Equation 5

D² - G_r (l₂,S_F) Equation 6

Here, G_rf(·) represents the guided filtering operation. S_F is the combined saliency map as the guidance image. I, and I₂ represent infrared image and visible image respectively . η, q, r₂, and e₂ are parameters of the guided filter, η = r₂ = 10, q = e₂=0.1. D¹ and D² are decision maps, which are normalized between zero and one, as shown below.

D' - minfD') . .

D =-------:----------—, z = 1,2 Equation 7 ^Λ max(D')-min(D') ⁴ [0002] In the fusion strategy, two initial fusion images are obtained from the infrared and visible images based on the multiple decision maps, and then the final fusion result is achieved by weighting those initial fusion images. The fusion strategy is defined as:

F(x, f) = Σ^α(Α (% y^^DN (V f) + Λ ί^χ’ ^)(¹ - ^D, (*> f)) Equation 8 / = 1.2 where /, (x, y) and I₂ (x, y) denote the infrared image and the visible image, D'_Nare the normalized decision maps. When z = 1, one decision map reconstructs the first initial fusion image, and when z = 2, another decision map reconstructs the second initial fusion image. These two initial fusion images are fused by a weighted-sum

Claims

1. The procedures of image fusion are as follows:

[0001] The proposed method is introduced in detail. The schematic diagram of the proposed fusion method is illustrated in Figure 1. From Figure 1, we can see that the proposed fusion method consists of four steps.

[0002] Stepl: The two kinds of source images are fed into the pre-trained VGG model for feature extraction.

[0003] Step2: We design a feature representation way to construct saliency maps by calculating the L2-norm of the channel direction of feature maps to estimate the activity level of pixels.

[0004] Step3: We build multiple decision maps by saliency maps, which is used as the guidance image to filter the infrared and visible image by guided filter.

[0005] Step4: Multiple decision maps are used for obtaining the initialized fusion results, and then they are fused with a weighted-sum fusion strategy to achieve the final fused image.

2. The procedures of image feature extraction are as follows:

[0001] For the VGG-16 convolutional layer, it contains five identical scales after four downsampling operation. We select the first layer under the identical scales to extract more spatial features. The convolution operations of VGG-16 is shown as follows. Here, let off denote the feature map extracted by the z-th layer and c is the channel, ce {1,2,--,C}.

a'_k ^c = φ. (I J Equation 1

2020100178 18 Feb 2020 where φ_ί (·) represents a convolution layer in the VGG-16 network, and ze {1,2,3,4,5} denotes the convll, conv_2_l, conv_3_l, conv_4_l, conv_5_l of different scales, respectively, convl l and conv_2_l are considered to extract features.

3. The procedures of image feature representation are as follows:

[0001] Stepl: The feature representation adopt L2-norm to construct saliency maps of the infrared source image and the visible source image I₂.

[0002] The activity level map J-^l _k can be obtained by the following formula:

T'_k(x,y) =|| a;'^:'(x,y) ||₂ Equation 2 [0003] Step2: The activity level map reflects the activity level coefficient of image.

The larger the coefficient, the more active this pixel is. We compare the activity level coefficient of infrared and visible images to get saliency features. Therefore, the saliency maps S(x,y) can be obtained by activity level maps of infrared and visible images.

Cl, if ^(x,y)> F‘(x, [0, otherwise where (x, y) and (x, y) are activity level maps obtained by the z-th convolution layer of infrared and visible image input network, respectively.

[0004] Step3: Combined saliency map S_F (x,y) is obtained by intersection strategy of the two saliency maps obtained with convll and conv_2_l. The mathematical formula is described as follows:

Equation 3

2020100178 18 Feb 2020

S_F (x,y) = S_A(x,y)OS_B(x,y) Equation 4

4. The procedures of multiple decision maps and fusion strategy are as follows: [0001] We employ the combined saliency map to construct multiply decision maps to realize the fusion of the source images. As shown in the part of multiple decision map fusion in Figure 1, the fusion decision maps are constructed by employing the guided filtering method, which makes use of the combined saliency map as the guidance image to filter the two source images. Because the guided filtering algorithm can maintain the edges of output image similar to the guidance image. The gradient information of the filtered image can be determined by the gradient information of the guidance image. Therefore, the saliency areas in the filtered image can be consistent with the combined saliency map. Due to the different acquisition modes of the source images, the information of the images is not completely complementary. Therefore, we propose a fusion strategy of multiple decision maps which can extract effective information from the source images respectively. The multiple decision maps can be obtained by the following formulas. The combined saliency map S_F (x,y) is used as the guidance image, and infrared image I_t and visible image I₂ are used as input image respectively.

D' =G_{r q} (ή, S_F) Equation 5

D² = G_r^₂(Ι₂,$ρ) Equation 6

Here, G_{r e}(·) represents the guided filtering operation. S_F is the combined saliency map as the guidance image. I_t and I₂ represent infrared image and visible image respectively . η, q, r₂, and e₂ are parameters of the guided filter, η=Γ₂=10, q = e₂=0.1. D^l and D² are decision maps, which are normalized between zero and one, as shown below.

2020100178 18 Feb 2020

D' =-------_:----------—, / = 1,2 Equation 7 max(D‘) - min(D‘) [0002] In the fusion strategy, two initial fusion images are obtained from the infrared and visible images based on the multiple decision maps, and then the final fusion result is achieved by weighting those initial fusion images. The fusion strategy is defined as:

F(x, y) = (ή (x, y)D'_N (x,y) +I₂ (x, y)(l - . (x, y)) Equation 8 /=1,2 where ή (x, y) and I₂ (x, y) denote the infrared image and the visible image, D'_Nare the normalized decision maps. When /=1, one decision map reconstructs the first initial fusion image, and when z = 2, another decision map reconstructs the second initial fusion image. These two initial fusion images are fused by a weighted-sum fusion strategy to obtain the fused image.