KR102553934B1 - Apparatus for Image Fusion High Quality - Google Patents

Abstract

The high-definition image convergence device fuses an infrared image and a visible ray image based on intermediate features that reflect the main structure of the source image and compensation features that represent the details of the source image, resulting in converged image quality. This has an enhancing effect.

Description

High-definition image fusion device {Apparatus for Image Fusion High Quality}

The present invention relates to a high-definition image convergence device, and more particularly, to an infrared image and an infrared image based on intermediate features that reflect the main structure of a source image and compensation features that represent details of the source image. The present invention relates to a high-definition image fusion device capable of fusing visible light images and improving the quality of the fused image.

Image fusion is widely used in a variety of applications ranging from remote sensing, medical diagnosis, security and surveillance.

In general, fusion tasks can contribute to object detection recognition challenges in the context of mapping in night environments, medical systems, photography, and remote sensing.

Image fusion is designed to obtain a more comprehensive and informative image by integrating multiple source images from various sensors. For example, thermal and visible light imaging systems are widely adopted to enhance surveillance in both civilian and military applications.

Recently, numerous convergence algorithms for infrared and visible ray images have been proposed. As a low-level fusion technique, pixel-level image fusion directly processes the pixels of an image from multiple sensors.

However, in practical applications, pixel-level fusion generally has problems of high computational intensity and unacceptable artifacts.

Korean Registered Patent No. 10-0551826

In order to solve this problem, the present invention provides an infrared image and a visible ray image based on intermediate features that reflect the main structure of the source image and compensation features that represent details of the source image. An object of the present invention is to provide a high-definition image fusion device in which the quality of the fused image is improved by fusion.

A high-definition image fusion device according to the features of the present invention for achieving the above object,

a feature extractor for extracting features from source images of an infrared image and a visible ray image, respectively;

An encoder unit composed of one or more convolutional layers functioning as a feature extractor and outputting intermediate features reflecting the main structure of the source image and compensation features representing details of the source image. ;

When the intermediate feature and the compensation feature are fused, an attention map of the source image for reflecting the feature of the source image is extracted from a feature map using the intermediate feature, and the generated attention map is used. a fusion unit generating a fused intermediate feature and generating a fused compensation feature using the compensation feature; and

and a decoder unit including one or more deconvolution layers and outputting a final fusion image by decoding the fused intermediate features and the fused compensation features.

According to the configuration described above, the present invention can significantly reduce computational complexity and load due to redundant information when converging a visible ray image and an infrared image, and has an effect of preventing unacceptable artifacts.

1 is a diagram showing the configuration of a high-definition video convergence device according to an embodiment of the present invention.
2 is a diagram illustrating a training step between an encoder unit and a decoder unit according to an embodiment of the present invention.
3 is a diagram illustrating visualization of an attention map and some acquired features according to an embodiment of the present invention.
4 is a diagram showing a training total loss curve in a training phase according to an embodiment of the present invention.
5 is a diagram illustrating a sample image.
6 is a diagram illustrating a fusion strategy for intermediate features according to an embodiment of the present invention.
7 is a diagram illustrating a fusion strategy for compensation features according to an embodiment of the present invention.
8 is a diagram illustrating a study on whether or not a skip connection is present according to an embodiment of the present invention.

Throughout the specification, when a certain component is said to "include", it means that it may further include other components without excluding other components unless otherwise stated.

1 is a diagram showing the configuration of a high-definition image fusion device according to an embodiment of the present invention, and FIG. 2 is a diagram showing a training step between an encoder unit and a decoder unit according to an embodiment of the present invention.

The high-definition image fusion device 100 according to an embodiment of the present invention includes an encoder unit 110, a fusion unit 120, and a decoder unit 130.

The high-definition image convergence device 100 represents a residual block method encoder decoder convergence device for convergence of an infrared image and a visible ray image.

The encoder unit 110 is composed of one or more convolutional layers functioning as a feature extractor, and includes intermediate features reflecting the main structure of the source image and compensation features representing details of the source image. ) is output.

When fusing the intermediate features and compensation features, the fusion unit 120 extracts an attention map of the source image for reflecting the features of the source image from the feature map using the intermediate feature, and converts the generated attention map to to generate a fused intermediate feature, and to generate a fused compensation feature using a reward feature.

The decoder unit 130 is composed of one or more deconvolution layers, decodes the fused intermediate feature and the fused compensation feature, and outputs a final fused image.

Various representative features can be effectively extracted from the source image.

In addition, the fusion of these features can output features of the same type because the same convolutional layer has the same weight.

Unlike existing frameworks that use only intermediate features, the framework of the present invention uses all features generated in the convolution layer for fusion.

Specifically, the two source images (visible ray image and infrared image) are separately encoded using a trained model to extract a series of features.

라고 정의한다. 여기서, k는 k번째 입력 소스 영상을 의미한다.Intermediate features generated by the residual block

define it as Here, k denotes the kth input source image.

k = 1 is an infrared (IR) image, and k = 2 is a visible ray image. i is the total number of intermediate features of the input image. res means the residual layer output.

로 정의된다.Also, the output characteristics of the first convolution (conv1) and the second convolution (conv2) are

is defined as

m is the total number of features in the conv1 layer, and n is the total number of features in the conv2 layer. These features are called reward features.

3 is a diagram illustrating visualization of an attention map and some acquired features according to an embodiment of the present invention.

Figure 3 (a) is a source image, Figure 3 (b) is an attention map, Figure 3 (c) is some features obtained from the first convolution layer and the residual block.

It can be seen that the intermediate features (red boxes) convey the main structure of the source image, while the compensating features (green boxes) retain some details of the source image.

As shown in (c) of FIG. 3, some of the features obtained from the first convolution layer and a residual block for visualization are selected.

It can be seen that the middle features with red boxes reflect the main structure of the source image. Accordingly, these deep features may provide background information of the fusion result. In contrast, compensation features with green boxes convey various textural details of the source image.

Therefore, the shallow features generated in the first convolutional layer and the second convolutional layer are equally important in reconstructing the final fusion result.

The fusion unit 120 is composed of two fusion strategies for intermediate features and reward features, namely, attention-based feature fusion and reward feature fusion.

으로 표현된다.The purpose of the decoder unit 130 is to decode the above-mentioned two fused parts to restore a final fused image. Similarly, the last (dconv1) and penultimate (dconv2) deconvolutions feature each

is expressed as

training phase

As shown in FIG. 2, the SEDRFuse network of the present invention includes an encoder unit 110 and a decoder unit 130 without a fusion part in the training stage.

The SEDRFuse network is a concept including an encoder unit 110, a fusion unit 120, and a decoder unit 130.

The SEDRFuse network is trained with available KAIST and FLIR data sets.

The training step is to accurately reconstruct the original data set with minimal reconstruction loss. That is, the smaller the reconstruction error, the more representative the extracted feature. The basic units of the training network are the functions Convolutional Layer, Deconvolutional Layer, Residual Block, Skip Connections, and Rectified Linear Units (ReLUs). . A pooling layer is removed to avoid loss of useful details in the original data set.

(1) Encoder unit

The encoder unit 110 is composed of three convolutional layers and one residual block. The size of the input training data is 256 × 256 (height and width).

The first convolutional layer does not change the input size, while the second and third convolutional layers (downsampling) half the input size.

In order to compensate for missing image details in the convolution process, we further reuse previous features by mimicking ResNet.

In this SEDRFuse network, the encoder unit 110 adds one residual block after the last convolutional layer. All convolution operations act as feature extractors while completely maintaining the texture and structure information of the original image.

The output of the encoder has 256 intermediate features of size 64 × 64.

The encoder unit 110 adds one residual block after the last convolution layer.

(2) decoder unit

The decoder unit 130 selects symmetric deconvolution for the corresponding convolution in the encoder unit 110 to obtain an output image having the same size as the input.

Deconvolution is generally used to reconstruct an original image from intermediate features extracted through upsampling. The kernel size of the deconvolution layer must be the same as the convolution layer to match exactly.

In this SEDRFuse network, all kernel sizes are set to 3 × 3. In addition, the decoder unit 130 has only two types of units, that is, a Deconvolutional Layer and a ReLU function.

(3) Residual Block

As shown in FIG. 2, one residual block is added to the encoder unit 110 for two purposes.

The first is to ensure optimal training convergence in the deep network used to extract more representative intermediate features.

The second is to make the most of the previous features generated in the third convolutional layer.

The selection of the number of residual blocks is determined by experimental performance.

The residual block is composed of F(x) composed of an input feature x and one or more weight layers, and rectified linear units (Rectified Linear Units) are formed by merging the output of F(x) and the input feature x in an element-wise manner. , ReLUs) function.

(4) Skip Connections

While the convolution operation preserves the underlying video content, texture details of the video may be lost. In addition, deconvolution may restore only structural details of video contents from extracted features having a certain amount of information loss during downsampling in the encoder unit 110 .

In general, since the output of the decoder unit 130 is a filtered version of the input image, image fusion performance is not satisfactory. Therefore, the present invention uses skip connection to transfer texture feature information from a convolution layer to a corresponding deconvolution layer in an element-by-element selection-maximum manner. These skip connections make the proposed framework easier to train and speed up convergence.

-norm 전략을 사용하면, 융합 과정에서 두드러진 특징을 구별할 수 없어 낮은 대비와 낮은 밝기의 융합 결과가 나타낸다.We describe two strategies for merging intermediate features and compensating features in the fusion step. As can be seen in (c) of FIG. 3, the middle feature represents the main structure of the source image. Therefore, a simple weighted average or

Using the -norm strategy, salient features cannot be distinguished during the fusion process, resulting in fusion results with low contrast and low brightness.

In order to preserve the brightness information of the original image, an attention-based feature fusion strategy that fuses intermediate features is designed.

Figure 3 (b) shows the attention map of the source video. Prominent entities in the source image are assigned higher attention scores suitable for fusing these intermediate features.

Nonetheless, as the number of convolutional layers increases, a lot of detail is lost in intermediate features. It can be seen that the features generated by the previous shallow layer (green box in FIG. 3(c)) contain details of the source image. Therefore, in order to retain these visual details in the fused result, we reuse reward features using a skip linking strategy.

1) Attention-Based Feature Fusion

Recently, a model based on the attention mechanism has been introduced to CNN architecture training. It is applied to the region of interest in many visual tasks, especially in visual scenes. The goal of infrared imaging and visible imaging fusion is to simultaneously retain visual detail and prominent thermal radiation areas.

Therefore, motivated by the previous work, we obtain the attention map of the original video using these intermediate features. The output of the residuals in the framework of the present invention is a series of intermediate features.

Each of them shows a special kind of information about the source image. The convergence unit 120 generates an attention map from a feature map in order to accurately reflect salient features of the source image. Each feature map has a unique weight given by the Softmax operation that calculates the probability of the channel direction.

The Softmax function can calculate the probability that each intermediate feature contributes to the attention map, so it is suitable for calculating the activity level measure for the source image. The fusion unit 120 calculates the probability weight map by using the Softmax function according to Equation 1 below.

는 각 특징의 확률 가중치 맵이고, (x, y)는 모든 중간 특징 채널의 동일한 위치를 나타내고, i는 채널 번호이고, 상기 잔차 블록에 의해 생성된 중간 특징들을

라고 정의하고, k는 k번째 입력 소스 영상이고, k=1은 적외선(IR)영상이고, k=2는 가시광선 영상이고, i는 입력 영상의 중간 특징의 총 개수이고, res는 잔차 레이어 출력이다.here,

is the probability weight map of each feature, (x, y) denotes the same location of all intermediate feature channels, i is the channel number, intermediate features generated by the residual block

, where k is the kth input source image, k = 1 is an infrared (IR) image, k = 2 is a visible ray image, i is the total number of intermediate features of the input image, and res is the residual layer output. am.

The Softmax function can be expressed as Equation 2 below.

는 벡터 시퀀스의 요소이다.here,

is an element of a vector sequence.

The fusion unit 120 generates an attention map for the source image by multiplying and summing the corresponding probability weights for all intermediate features by Equation 3 below.

는 소스 영상의 활동 수준 측정을 반영하는 어텐션 맵이다. 본 발명은 두드러진 메커니즘(Salient Mechanism)에 따라 특징 수준 융합(Feature-Level Fusion) 전에 이러한 중간 특징을 최적화하기 위해 어텐션 맵을 사용한다.here,

is an attention map that reflects the activity level measurement of the source video. The present invention uses the attention map to optimize these intermediate features before feature-level fusion according to the Salient Mechanism.

The fusing unit 120 generates a fused intermediate feature using Equations 4 and 5 below.

는 j번째 소스 영상의 특징에 대한 최적의 가중치 맵이다.

는 디코더부(130)로 보내질 융합된 중간 특징이다.Here, j denotes the jth input source image.

is an optimal weight map for the feature of the jth source image.

is the fused intermediate feature to be sent to the decoder unit 130.

2) Compensation Feature Fusion

In the case of the compensation feature, the encoder unit 110 may reconstruct missing details of the convolution process using this feature. Since each feature pixel value after compression represents the receptive field of the original image, the choose-max strategy is a better choice to merge in an element-by-element fashion.

The fusion unit 120 generates a fused compensation feature by using the compensation feature received from the encoder unit 110 .

The fusion unit 120 generates a fused first compensation feature with respect to the first convolution using Equation 6 below.

는 각각 첫 번째 컨볼루션의 적외선 특징들, 가시광선 특징들, 융합된 제1 보상 특징들을 나타낸다. m은 첫 번째 컨볼루션의 총 개수이다.here,

denotes infrared features, visible light features, and fused first compensation features of the first convolution, respectively. m is the total number of first convolutions.

은 요소별 방식에서 최대 선택 함수이다.(x,y) are the pixel coordinates of the feature,

is the maximum selection function in the element-wise method.

The fusion unit 120 generates a fused second compensation feature for the second convolution using Equation 7 below.

및

은 각각 두 번째 컨볼루션의 적외선 특징들, 가시광선 특징들, 융합된 제2 보상 특징들을 나타낸다.here,

and

denotes infrared features, visible light features, and fused second compensation features of the second convolution, respectively.

Equation 7 is expressed in the same way as Equation 6.

The decoder unit 130 calculates the output of the first deconvolution layer and the fused first compensation features received from the fusion unit 120 as elementwise summation by Equation 8 below, and inputs the first input. create a feature

은 제1 입력 특징이고,

은 첫 번째 디컨볼루션 레이어의 출력이다.here,

is the first input feature,

is the output of the first deconvolution layer.

The decoder unit 130 calculates the output of the second deconvolution layer and the fused second compensation features received from the fusion unit 120 as an elementwise summation according to Equation 9 below, and inputs the second input. create a feature

은 제2 입력 특징이고,

은 두 번째 디컨볼루션 레이어의 출력이다.here,

is the second input feature,

is the output of the second deconvolution layer.

및

]은 융합된 영상의 시각적 세부 사항을 보상할 수 있다. 그것들은 요소별 합계(Elementwise Summation)에 의해 해당 디컨볼루션 레이어(Deconvolutional Layer)에 전달되며, 수학식 8과 수학식 9와같이 나타낼 수 있다.As a result, the fused compensation features of the first convolutional layer and the second convolutional layer [

and

] can compensate for the visual details of the fused image. They are transferred to the corresponding deconvolution layer by elementwise summation and can be expressed as Equations 8 and 9.

는 각각 첫 번째 디컨볼루션 레이어 및 두 번째 디컨볼루션 레이어의 출력이고,

와

는 입력 특징이며, 이러한 입력 특징들은 디코더부(130)의 두 번째 디컨볼루션 레이어와 세 번째 디컨볼루션 레이어로 전달된다.

are the outputs of the first deconvolution layer and the second deconvolution layer, respectively,

and

is an input feature, and these input features are transferred to the second deconvolution layer and the third deconvolution layer of the decoder unit 130.

Experiment result

A. Data preparation

In the present invention, KAIST and FLIR data sets including pairs of infrared and visible images were used for training. The KAIST data set consists of 95,000 color column pairs taken from vehicles. The FLIR data set provides approximately 14,452 pairs of thermal and visible images for autonomous driving research.

Both data sets were recorded as a series of video clips at 20 and 30 Hz, respectively.

The data set covers a variety of scenarios including campus, road, downtown, street and highway captured day and night. To further diversify the training data, the frame interval was extended through resampling. The total number of infrared and visible ray images is 52,000, including 50,000 for training and 2000 for verification. Details of the data sets are shown in Table 1 (data sets used for training). All images in the experiment were resized to 256 × 256 pixels and converted to grayscale images within the [0, 1] range.

B. Experimental setup

1) Training Details

We trained the SEDRFuse network with the prepared training data set (Table 1). Batch size and Epoch number were set to 2 and 50, respectively.

The learning rate was set to 1 × 10 ^-4 . Pixel loss and SSIM loss were adopted as total loss functions. These two loss functions can limit both reconstructed pixel errors and edge errors. The total loss function can be expressed as Equation 10.

Here, Tloss, Ploss and SSIMloss represent total loss, pixel loss and SSIM-loss, respectively. SSIM is a structural similarity score, and SSIM-loss is defined by Equation 11.

은 구조 유사도 점수로 계산된 SSIM 함수를 나타낸다.Here, out and in refer to reconstructed data and input training data, respectively.

denotes the SSIM function calculated as a structural similarity score.

In Equation 12, M and N are image sizes, and (x, y) represent pixel positions.

The SEDRFuse network of the present invention was implemented on a computer equipped with NVIDIA GTX 1070 Ti (GPU), 32 GB RAM (memory), and Intel Core i5-8500 (CPU). The network architecture was programmed in Tensorflow.

4 is a diagram showing a training total loss curve in a training phase according to an embodiment of the present invention.

The total loss value is given for every 1000 iterations. Each epoch requires 25 × 103 iterations. From the curve, it can be seen that the total loss value tends to be stable around 50 epochs. This shows that the trained model has reached an optimal setting.

The test details were adapted from the experiment to test the trained model on the Nederlandse Organisatie voor Toegepast-Natuurwetenschappelijk Onderzoek (TNO) dataset4. Twenty pairs of infrared and visible light images containing various scenes were selected. All pairs of source images have been fully registered, and FIG. 5 shows sample images.

Figure 5 is the source image part of the TNO data set, the first and third rows are infrared images and the second and fourth rows are visible images.

As for the evaluation index, both subjective and objective evaluations were performed in the experiment to verify the fusion performance. Mean slope (AG), correlation coefficient (CC), entropy (EN), spatial frequency (SF), structural similarity convergence metric (SSIMfm), visual information fidelity for convergence (V IFF), and standard deviation (SD). The larger the metric mentioned above, the better the fusion method performs. For details, see the corresponding literature. In particular, the SSIMfm fusion metric used in this experiment is calculated as in Equation 13. Here, SSIM(·) represents the SSIM function. A and B are the two source images. F is a fused image that serves as a reference image.

Ablation Study

In the ablation study, we first optimized the parameters for our framework, including the number of residual blocks and training epochs.

The present invention performs three comparison experiments to validate the selected parameters. Selection of the number of residual blocks and training epochs can be made through trial and error.

For example, the number of residual blocks can be determined by observing the fusion performance for the number of blocks in terms of a particular metric. Similarly, the optimal number of epochs can be found in the plot of the fusion performance metric value against number. In experiments, we chose one residual block for our framework and confirmed that 50 epochs should be sufficient to train the model. The choice of these numbers also considers computational efficiency.

6 is a diagram illustrating a fusion strategy for intermediate features according to an embodiment of the present invention.

Fusion strategy for intermediate features

-norm 기반 및 어텐션 맵 기반 융합 전략을 조사했다. 도 6은 Trench와 House의 두 장면을 세 가지 전략으로 융합한 결과를 보여준다. 인간의 머리와 집 모서리(강조 표시 및 확대)의 관찰은 어텐션 기반 융합 전략이 중간 특징 융합에 대한

-norm 및 평균 기반 전략보다 더 많은 밝기 정보를 보존한다는 것을 보여주었다.Mean basis for intermediate feature fusion,

-We investigated norm-based and attention map-based convergence strategies. Figure 6 shows the results of fusion of two scenes of Trench and House with three strategies. Observations of human heads and house corners (highlighted and magnified) suggest that attention-based fusion strategies can be useful for intermediate feature fusions.

showed that it preserves more brightness information than -norm and average-based strategies.

-norm 기반 융합 전략 모두를 다시 능가함을 알 수 있다.Convergence outcomes on the TNO data set were also evaluated with nine selected convergence evaluation metrics. From the quantitative results reported in Table 2, the attention map-based fusion strategy averaged and

It can be seen that it again outperforms all of the -norm-based convergence strategies.

Therefore, attention-based strategies are more suitable for fusing intermediate features.

-norm 전략을 사용한 결과이고, 도 6의 (e)는 어텐션 맵 전략을 사용한 결과이다.Figure 6 (a) is an infrared image, Figure 6 (b) is a visible ray image, Figure 6 (c) is the result of using the averaging strategy, Figure 6 (d) is

This is the result using the -norm strategy, and FIG. 6(e) is the result using the attention map strategy.

-norm에 의한 결과의 확대된 영역은 어텐션 기반 융합 전략에 의한 영역보다 더 낮은 밝기 정보를 보여준다.averaging and

The enlarged region of the result by -norm shows lower brightness information than the region by the attention-based convergence strategy.

7 is a diagram illustrating a fusion strategy for compensation features according to an embodiment of the present invention.

The fusion strategy for reward features investigated pixel-by-pixel summation, concatenation, and select-max fusion strategies for reward feature fusion. Figure 7 shows the results of fusing two image pairs such as Bunker and Jeep with three strategies.

7 is a result of using (a) IR imaging, (b) visible ray imaging, and (c) linking strategy as fusion strategies for compensation features. (d) Results using pixel-by-pixel summation strategy. (e) This is the result using the maximum selection strategy. The enlarged area shows that the roof and window area fused with the selection maximal strategy is clearer than that of the other two strategies.

Visual evaluation shows that the pixel-by-pixel summation and concatenation strategy performs similarly to preserving brightness information in slightly darker IR images than the select max strategy. In particular, the fused roof and window areas (highlighted and enlarged) with the selection maximal strategy are clearer than the other two strategies.

We further tested three strategies on the TNO data set using selected convergence metrics.

The results in Table 3 show that the select maximal strategy achieves the best overall performance in terms of fusion metrics. Therefore, we adopt a select maximal strategy in the framework of the present invention.

8 is a diagram illustrating a study on whether or not a skip connection is present according to an embodiment of the present invention.

To determine the effect of skip linking, we conducted two experiments with and without skip linking.

The fusion results for Sandpath and Lake image pairs are shown in FIG. 8 . Detailed information in the visible light image can be seriously missed in fusion without skip connections. For example, fences and chairs are barely visible in Fig. 8(c).

8 is a study on the presence or absence of skip connection, which shows (a) an infrared image, (b) a visible ray image, (c) a result without skip connection, and (d) a result using skip connection.

Zoomed areas in results without skip connections suffer from missing detailed information. In contrast, the enlarged region resulting from the method of the present invention may well preserve the details of the source image.

In contrast, fusion with skip linking can preserve more details.

In addition, it can be seen from the quantitative results in Table 4 that fusion with skip linkage is generally superior. Therefore, in the framework of the present invention, skip connections are used to retain detailed information in the image reconstruction process.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements made by those skilled in the art using the basic concept of the present invention defined in the following claims are also included in the scope of the present invention. that fall within the scope of the right.

100: high-definition image fusion device
110: encoder unit
120: fusion part
130: decoder unit

Claims (9)

a feature extractor for extracting features from source images of an infrared image and a visible ray image, respectively;
An encoder unit composed of one or more convolutional layers functioning as a feature extractor and outputting intermediate features reflecting the main structure of the source image and compensation features representing details of the source image. ;
When the intermediate feature and the compensation feature are fused, an attention map of the source image for reflecting the feature of the source image is extracted from a feature map using the intermediate feature, and the generated attention map is a fusion unit generating a fused intermediate feature using a fused compensation feature and generating a fused compensation feature using the compensation feature; and
It is composed of one or more deconvolution layers, and includes a decoder unit that decodes the fused intermediate feature and the fused compensation feature to output a final fusion image,
The encoder unit adds one residual block after the last convolution layer, the residual block is composed of F (x) composed of an input feature x and one or more weight layers, and the F ( merging the output of x) and the input feature x in an element-by-element manner to generate a Rectified Linear Units (ReLUs) function;
The fusion unit calculates a probability weight map by a Softmax function according to Equations 1 and 2 below,
[Equation 1]

here,

is the probability weight map of each feature, (x, y) denotes the same position of all intermediate feature channels, i is the channel number, intermediate features generated by the residual block

, where k is the kth input source image, k = 1 is an infrared (IR) image, k = 2 is a visible ray image, i is the total number of intermediate features of the input image, and res is the residual layer output. is,
[Equation 2]

here,

is an element of the vector sequence,
The fusion unit generates an attention map for the source image by multiplying and summing corresponding probability weights for all intermediate features by Equation 3 below,
[Equation 3]

here,

is an attention map reflecting the activity level measurement of the source video,
The fusion part generates the fused intermediate feature using Equations 4 and 5 below,
[Equation 4]

[Equation 5]

Here, j is the jth input source image,

Is the optimal weight map for the feature of the jth source image,

is the fused intermediate feature to be sent to the decoder unit,
The fusion part generates a fused first compensation feature for the first convolution using Equation 6 below,
[Equation 6]

here,

denotes the infrared features, visible light features, and fused first compensation features of the first convolution, respectively, m is the total number of first convolutions, (x, y) are the pixel coordinates of the feature,

is the maximum selection function in the element-wise method,
The fusion part generates a fused second compensation feature for the second convolution using Equation 7 below,
[Equation 7]

here,

and

are infrared features, visible light features, and fused second compensation features of the second convolution, respectively. delete delete delete delete delete delete The method of claim 1,
The decoder unit calculates the output of the first deconvolution layer and the fused first compensation features received from the fusion unit as an elementwise summation by Equation 8 below to generate a first input feature High-definition image fusion device.
[Equation 8]

here,

is the first input feature,

is the output of the first deconvolution layer. The method of claim 1,
The decoder unit calculates the output of the second deconvolution layer and the fused second compensation features received from the fusion unit as an elementwise summation by Equation 9 below to generate a second input feature High-definition image fusion device.
[Equation 9]

here,

is the second input feature,

is the output of the second deconvolution layer.

Images