KR20230102134A - Real-time image fusion apparatus and method for remote sensing based on deep learning - Google Patents

Abstract

An apparatus for real-time image fusion for remote sensing based on deep learning according to an embodiment of the present invention includes a shallow feature extractor for extracting shallow features from an upsampled low-resolution multispectral image and a panchromatic image, respectively; An information pool for generating raw information features based on the panchromatic image, the upsampled low-resolution multispectral image, shallow features of the upsampled low-resolution multispectral image, and shallow features of the panchromatic image ; a first multi-scale feature extraction sub-network generating small-scale features, middle-scale features, and large-scale features of the up-sampled low-resolution multi-spectral image based on shallow features of the up-sampled low-resolution multi-spectral image; a second multi-scale feature extraction sub-network generating small-scale features, middle-scale features, and large-scale features of the panchromatic image based on shallow features of the panchromatic image; a multi-scale feature fusion unit fusing the raw information feature, the small-scale feature, middle-scale feature, and large-scale feature of the low-resolution multi-spectral image, and the small-scale feature, middle-scale feature, and large-scale feature of the panchromatic image; and a reconstruction unit that reconstructs a high-resolution multispectral image based on the fused feature and the upsampled low-resolution multispectral image.

Description

Real-time image fusion apparatus and method for remote sensing based on deep learning {REAL-TIME IMAGE FUSION APPARATUS AND METHOD FOR REMOTE SENSING BASED ON DEEP LEARNING}

The present invention relates to a real-time image fusion apparatus and method for remote sensing based on deep learning.

Remote sensing is a fundamental tool for understanding the Earth and supporting human-Earth communication. Multispectral (MS) imaging, an important field of remote sensing technology, is widely applied in real-time monitoring and surveillance. However, it is particularly difficult to obtain multispectral images with high spatial resolution using a single sensor due to sensor and bandwidth limitations.

In general, in order to solve this problem, a light-based survey satellite is equipped with a multi-sensor to obtain a multi-band low-resolution multi-spectral (LRMS) image and a single-band high resolution (HR) pan-chromatic image. obtain at the same time. The pan sharpening technique is a technique for reconstructing a high-resolution multispectral image by fusing complementary information of a low-resolution multispectral image and a high-resolution panchromatic image, and its spatial resolution is the same as that of the panchromatic image.

Various tasks such as image classification, image segmentation, and image detection are booming in the real-time remote sensing community. Typically, these tasks prefer to use high-resolution remote-sensing images as input. This is because the higher the resolution of the input image, the more information it will provide for subsequent processing and analysis. Conversely, low-resolution images tend to hinder the performance of these tasks because they are sensitive to the structure of objects and the relationship between pixels that are limited in low-resolution images. As shown in Figure 1, remote-sensing image fusion (pan sharpening) is very important for remote-sensing image processing, providing high-resolution images in both the spectral and spatial domains to increase the accuracy of subsequent operations.

Over the past decades, researchers have paid much attention to fan sharpening technology and proposed various meaningful methods. Existing fan sharpening methods can actually be divided into two categories: traditional algorithms and deep learning-based methods. Regarding existing algorithms, we can further classify them into three branches: component substitution (CS) approach, multi-resolution analysis (MRA) approach, and model-based optimization approach. can

For the CS approach, we usually transform the multispectral image into an appropriate region. Here, the spatial component of the multispectral image is replaced with a high-resolution panchromatic image. A fan-sharpened image is then achieved through the corresponding inverse transform. Intensity-hue-saturation (HIS), principal component analysis (PCA), Brovey transform (BT) and Gram-Schmidt transform are often incorporated into CS . These methods are simple, fast, and tend to obtain spatial information directly from panchromatic images, but suffer from spectral distortion. In the case of the MRA method, a discrete wavelet transform (DWT), a Laplacian pyramid (LP), and a trous wavelet transform (ATWT) are mainly integrated to decompose a panchromatic image, extract high-frequency information, and inject into LRMS. This method can obtain better spectral coherence, but may result in more severe spatial distortion.

With recent significant advances in deep learning, Convolutional Neural Networks (CNNs) have garnered a lot of attention from the fan sharpening community, achieving significant improvements in both performance and efficiency. Because both pan sharpening and single image super-resolution (SISR) aim to reconstruct an enhanced high-resolution (HR) image, pan sharpening is to some extent super-resolution related. The main difference is that the former primarily extracts spatial details from high-resolution panchromatic images, while the latter extracts details from corresponding low-resolution (LR) images. Therefore, many existing deep learning-based fan sharpening approaches have been enlightened by SISR methods.

In general, real-time and effective fan sharpening methods are preferred in practical applications. With the rapid development of graphics processing units (GPUs), most deep learning-based fan sharpening methods can meet real-time requirements. However, there is a need for a real-time image fusion device and method for remote sensing capable of obtaining clearer images and reducing spectral distortion under real-time application conditions.

KR 10-1132272 B1

The problem to be solved by the present invention is to make the most of the layered complementary features of the panchromatic image and the multispectral image extracted by the weight sharing encoder-decoder structure, and to enable a larger receptive field to obtain a multispectral image and corresponding It is to provide a real-time image fusion device and method for remote sensing based on deep learning, which can usefully acquire multi-scale complementary context information of panchromatic images.

Another problem to be solved by the present invention is based on deep learning, which can richly extract all of the features of various scales by acquiring multi-scale features step by step by integrating a coarse-to-fine strategy into the network. It is to provide a real-time image fusion device and method for remote sensing.

Another problem to be solved by the present invention is to build an information pool to preserve basic information for subsequent feature fusion, which can not only improve the expressiveness of the network but also help with gradient back-propagation. It is to provide a real-time image fusion device and method for remote sensing based on deep learning.

Another problem to be solved by the present invention is to provide a real-time image fusion device and method for remote sensing based on deep learning, which can significantly reduce spectral distortion and acquire clear, high-resolution images at the same time.

Real-time image fusion device for remote sensing based on deep learning according to an embodiment of the present invention for solving the above problems,

a shallow feature extraction unit extracting shallow features from the upsampled low-resolution multi-spectral image and the pan-chromatic image;

An information pool for generating raw information features based on the panchromatic image, the upsampled low-resolution multispectral image, shallow features of the upsampled low-resolution multispectral image, and shallow features of the panchromatic image ;

a first multi-scale feature extraction sub-network generating small-scale features, middle-scale features, and large-scale features of the up-sampled low-resolution multi-spectral image based on shallow features of the up-sampled low-resolution multi-spectral image;

a second multi-scale feature extraction sub-network generating small-scale features, middle-scale features, and large-scale features of the panchromatic image based on shallow features of the panchromatic image;

The raw information feature, the small-scale feature, middle-scale feature, and large-scale feature of the upsampled low-resolution multi-spectral image, and the multi-scale feature fusing the small-scale feature, middle-scale feature, and large-scale feature of the panchromatic image. fusion; and

and a reconstruction unit for reconstructing a high-resolution multispectral image based on the fused feature and the upsampled low-resolution multispectral image.

In the real-time image fusion device for remote sensing based on deep learning according to an embodiment of the present invention, the shallow feature extraction unit,

a first convolution layer that performs a convolution operation on the upsampled low-resolution multispectral image and outputs a shallow feature of the upsampled low-resolution multispectral image; and

and a second convolution layer that performs a convolution operation on the panchromatic image and outputs a shallow feature of the panchromatic image.

In addition, in the real-time image fusion device for remote sensing based on deep learning according to an embodiment of the present invention, the information pool,

a connection unit connecting the pan-chromatic image, the up-sampled low-resolution multi-spectral image, a shallow feature of the up-sampled low-resolution multi-spectral image, and a shallow feature of the pan-chromatic image; and

It may include a third convolution layer that performs a convolution operation on the output of the connection unit and outputs the original information feature.

In addition, in the real-time image fusion device for remote sensing based on deep learning according to an embodiment of the present invention, the first multi-scale feature extraction subnetwork,

a first encoder that lowers spatial resolution of shallow features of the upsampled low-resolution multispectral image and increases feature channels of feature maps; and

A first decoder for generating a small scale feature, a middle scale feature, and a large scale feature of the upsampled low-resolution multispectral image based on an output of the first encoder;

The second multi-scale feature extraction subnetwork,

a second encoder that lowers spatial resolution of shallow features of the panchromatic image and increases feature channels of feature maps; and

and a second decoder for generating a small scale feature, a middle scale feature, and a large scale feature of the panchromatic image based on an output of the second encoder.

In addition, in the real-time image fusion device for remote sensing based on deep learning according to an embodiment of the present invention, the first multi-scale feature extraction sub-network and the second multi-scale feature extraction sub-network are used in a convolution layer Weights can be shared.

In addition, in the real-time image fusion device for remote sensing based on deep learning according to an embodiment of the present invention, the multi-scale feature fusion unit,

a small-scale feature fusion unit fusing the small-scale feature of the upsampled low-resolution multi-spectral image with the small-scale feature of the panchromatic image and outputting a fused small-scale feature;

a middle scale feature fusion unit fusing middle scale features of the upsampled low-resolution multispectral image with middle scale features of the panchromatic image and outputting a fused middle scale feature; and

The fused small-scale feature, the fused middle-scale feature, the large-scale feature of the upsampled low-resolution multispectral image, the large-scale feature of the panchromatic image, and the raw information feature are fused to output a final fused feature. It may include a global feature fusion part that does.

A real-time image fusion method for remote sensing based on deep learning according to an embodiment of the present invention for solving the above problems is,

(A) extracting, by a shallow feature extraction unit, shallow features from the upsampled low-resolution multi-spectral image and the panchromatic image;

(B) an information pool based on the panchromatic image, the upsampled low resolution multispectral image, shallow features of the upsampled lowresolution multispectral image, and shallow features of the panchromatic image; generating an information feature;

(C) a first multi-scale feature extraction subnetwork generates small-scale features, middle-scale features, and large-scale features of the upsampled low-resolution multispectral image based on shallow features of the upsampled low-resolution multispectral image step;

(D) generating, by a second multi-scale feature extraction sub-network, small-scale features, middle-scale features, and large-scale features of the panchromatic image based on the shallow features of the panchromatic image;

(E) a multi-scale feature fusion unit, the raw information feature, the small-scale feature, middle-scale feature and large-scale feature of the upsampled low-resolution multi-spectral image, and the small-scale feature, middle-scale feature of the panchromatic image, and fusing large scale features; and

(F) reconstructing, by a reconstruction unit, a high-resolution multispectral image based on the fused feature and the upsampled low-resolution multispectral image.

In the real-time image fusion method for remote sensing based on deep learning according to an embodiment of the present invention, the step (A) is,

outputting shallow features of the upsampled low resolution multispectral image by performing a convolution operation on the upsampled low resolution multispectral image; and

and outputting a shallow feature of the panchromatic image by performing a convolution operation on the panchromatic image.

In addition, in the real-time image fusion method for remote sensing based on deep learning according to an embodiment of the present invention, the step (B) is,

(B-1) connecting the panchromatic image, the upsampled low-resolution multispectral image, shallow features of the upsampled lowresolution multispectral image, and shallow features of the panchromatic image; and

(B-2) performing a convolution operation on the output of step (B-1) to output the original information feature.

In addition, in the real-time image fusion method for remote sensing based on deep learning according to an embodiment of the present invention, the step (C),

(C-1) lowering spatial resolution of shallow features of the upsampled low-resolution multi-spectral image and increasing feature channels of feature maps; and

(C-2) generating a small-scale feature, a middle-scale feature, and a large-scale feature of the upsampled low-resolution multispectral image based on the output of step (C-1);

In the step (D),

(D-1) lowering spatial resolution of shallow features of the panchromatic image and increasing feature channels of feature maps; and

(D-2) generating a small-scale feature, a middle-scale feature, and a large-scale feature of the panchromatic image based on the output of step (D-1).

In addition, in the real-time image fusion method for remote sensing based on deep learning according to an embodiment of the present invention, the step (C) and the step (D) may share weights used in the convolutional layer.

In addition, in the real-time image fusion method for remote sensing based on deep learning according to an embodiment of the present invention, the step (E) is,

fusing small scale features of the upsampled low-resolution multispectral image with small scale features of the panchromatic image and outputting a fused small scale feature;

fusing middle-scale features of the upsampled low-resolution multi-spectral image with middle-scale features of the panchromatic image and outputting a fused middle-scale feature; and

The fused small-scale feature, the fused middle-scale feature, the large-scale feature of the upsampled low-resolution multispectral image, the large-scale feature of the panchromatic image, and the raw information feature are fused to output a final fused feature. steps may be included.

According to an apparatus and method for real-time image fusion for remote sensing based on deep learning according to an embodiment of the present invention, layered complementary features of a panchromatic image and a multispectral image extracted by a weight-sharing encoder-decoder structure are maximized. In addition, by enabling a larger receptive field, multi-scale complementary context information of a multispectral image and a corresponding panchromatic image can be usefully obtained.

According to the real-time image fusion apparatus and method for remote sensing based on deep learning according to an embodiment of the present invention, by integrating a coarse-to-fine strategy into a network to acquire multi-scale features step by step, various All features of the scale can be extracted abundantly.

According to the real-time image fusion apparatus and method for remote sensing based on deep learning according to an embodiment of the present invention, by constructing an information pool to preserve basic information for subsequent feature fusion, the expressive ability of the network is improved, and It can also help with gradient back-propagation.

According to the real-time image convergence apparatus and method for remote sensing based on deep learning according to an embodiment of the present invention, spectral distortion can be significantly reduced and a clear high-resolution image can be obtained at the same time, for remote sensing based on deep learning. It is to provide a real-time image fusion device and method.

1 illustrates the important role of fan-sharpening in the remote sensing community.
2 is a diagram illustrating a real-time image fusion device for remote sensing based on deep learning according to an embodiment of the present invention.
3 is a diagram showing the structure of a first multi-scale feature extraction sub-network of a multi-spectral path;
4 is a diagram showing the structure of a multi-scale feature fusion unit;
5 is a flowchart of a real-time image fusion method for remote sensing based on deep learning according to an embodiment of the present invention.
Figure 6a shows a loss curve of an ablation experiment corresponding to Table 4;
6B shows loss curves of single-scale fusion and multi-scale fusion corresponding to Table 5;
Figure 7 shows the visual results of different pan-sharpening methods on the QuickBird dataset.
Figure 8 shows the visual results of different pan-sharpening methods for the IKONOS dataset.
Figure 9 shows the visual results of different pan-sharpening methods on the Pleiades dataset.

Objects, specific advantages and novel features of the present invention will become more apparent from the following detailed description and preferred embodiments taken in conjunction with the accompanying drawings.

Prior to this, the terms or words used in this specification and claims should not be interpreted in a conventional and dictionary sense, and the inventor may appropriately define the concept of the term in order to explain his or her invention in the best way. It should be interpreted as a meaning and concept consistent with the technical spirit of the present invention based on the principles.

In adding reference numerals to components of each drawing in this specification, it should be noted that the same components have the same numbers as much as possible, even if they are displayed on different drawings.

In addition, terms such as “first”, “second”, “one side”, and “other side” are used to distinguish one component from another component, and the components are limited by the terms. It is not.

Hereinafter, in describing the present invention, detailed descriptions of related known technologies that may unnecessarily obscure the subject matter of the present invention will be omitted.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Real-time monitoring and surveillance plays an important role in remote sensing applications where multi-spectral (MS) images with high spatial resolution are widely required for better analysis. However, high-resolution multispectral images cannot be obtained directly due to sensor and bandwidth limitations. As an essential method to alleviate this problem, pan-sharpening aims to reconstruct a high-resolution multispectral image by fusing complementary information of a low-resolution multispectral image and a high-resolution panchromatic (PAN) image.

Most previous deep learning-based methods can meet real-time requirements with the help of a graphics processing unit (GPU). However, they do not fully utilize the advantageous layered information, leaving no significant room for performance improvement.

In the present invention, in order to achieve more effective performance while meeting the requirements of real-time implementation, we propose a multi-scale fusion network that makes the most of the layered complementary features of panchromatic images and multispectral images. do. Specifically, we introduce an encoder-decoder structure and a coarse-to-fine strategy to effectively extract multi-scale features of panchromatic images and multispectral images separately.

Meanwhile, information pools are being adopted to preserve raw information. Then a multi-scale feature fusion part is applied to fuse the multi-scale features of the decoder and information pool. Finally, a high-resolution multispectral image is reconstructed by utilizing the fused features. Extensive experiments show that the real-time image fusion device and method for remote sensing based on deep learning according to the present invention achieves advantageous performance compared to other methods in terms of quantitative metrics and visual quality. In addition, the results on the execution time indicate that the real-time image fusion apparatus and method for remote sensing based on deep learning according to the present invention can achieve real-time performance.

Specifically, multi-scale features of panchromatic images and multi-spectral images are effectively extracted by introducing a weight sharing encoder-decoder structure and a coarse-to-fine strategy. The encoder-decoder structure is useful for capturing the multi-spectral image and the multi-scale complementary contextual information of the corresponding panchromatic image by allowing the network to obtain a larger receptive field.

In the case of the decoder step, in the real-time image fusion apparatus and method for remote sensing based on deep learning according to an embodiment of the present invention, a coarse-to-fine strategy is used to gradually decode multi-scale features to obtain finer details step by step. integrate In this way, not only small-scale features, but also middle-scale and large-scale features are extracted richly, making for better subsequent fusion. It also employs information pools to preserve raw information.

In this information pool, the panchromatic image, which is an upsampled multispectral image, and shallow features (features obtained through a single convolutional layer) are concatenated and fed to the convolutional layer to reserve raw information. The intuitive motivation of the information pool is feature reuse and memory mechanisms, which means that shallow features containing raw spatial and spectral information help improve the performance of fan sharpening.

On the other hand, the skip connection of the information pool applied to subsequent feature fusion also has an advantage in gradient backpropagation. Then, a multi-scale feature fusion module is applied to fuse multi-scale features and raw information in the decoder and information pool, respectively. Finally, a high-resolution multispectral image is reconstructed through several residual blocks using the fused features. It is worth mentioning that global residual learning on upsampled multispectral images is employed to preserve spectral information. Experimental results show the superiority of the proposed multi-scale fusion network (MSFN).

In the following, the framework of the multi-scale fusion network (MSFN) proposed in the real-time image fusion device and method for remote sensing based on deep learning according to an embodiment of the present invention, including an overview architecture and mathematical expression, will be described. The two special modules are then described in detail. After that, real-time implementation will be described. Finally, a loss function is introduced.

network framework

As shown in FIG. 2, the real-time image fusion device 200 for remote sensing based on deep learning according to an embodiment of the present invention includes a shallow feature extraction unit 202 (SFE: Shallow Feature Extraction), a multi-scale feature extraction network ( 206) (MSFENet: Multiscale Feature Extraction Net), information pool 204 (IP: Information Pool), multi-scale feature fusion part 208 (MSFF: Multi-Scale Feature fusion) and reconstruction part 210 It consists of

The shallow feature extractor 202 extracts shallow features from the upsampled low-resolution multispectral image 212 and the panchromatic image, respectively, and an information pool 204 extracts the panchromatic image, the panchromatic image, and the panchromatic image. Generating raw information features based on the upsampled low-resolution multispectral image, shallow features of the upsampled low-resolution multispectral image, and shallow features of the panchromatic image, the first multi-scale feature extraction subnetwork 222 Small-scale features, middle-scale features, and large-scale features of the upsampled low-resolution multispectral image are generated based on shallow features of the upsampled low-resolution multispectral image, and the second multi-scale feature extraction subnetwork 224 Small-scale features, middle-scale features, and large-scale features of the panchromatic image are generated based on the shallow features of the panchromatic image, and the multi-scale feature fusing unit 226 includes the original information feature and the upsampled The small-scale feature, middle-scale feature, and large-scale feature of the low-resolution multi-spectral image and the small-scale feature, middle-scale feature, and large-scale feature of the panchromatic image are fused, and the reconstruction unit 210 fuses the fused feature and A high resolution multispectral image is reconstructed based on the upsampled low resolution multispectral image.

및

을 각각 저해상도 멀티 스펙트럼 이미지 및 팬크로마틱 이미지 입력으로 표시한다.

는 MSFN의 출력을 나타낸다. 특히 SFE 부분의 경우

및

이 별도로 처리된다. 대응하는

과 업샘플링된

로부터 각각 얕은 특징들

와

를 추출하기 위하여 단지 하나의 컨볼루션 레이어만 적용된다.

and

as low-resolution multispectral image and panchromatic image input, respectively.

represents the output of MSFN. Especially for the SFE part

and

This is handled separately. corresponding

and upsampled

Each shallow feature from

and

To extract , only one convolutional layer is applied.

와

는 각각 팬크로마틱 경로와 멀티 스케일 경로의 컨볼루션 연산을 나타낸다.

은 쌍삼차 업샘플링 연산을 나타낸다. 그런 다음 얕은 특징들

와

는 멀티 스케일 특징들을 추가로 추출하기 위해 멀티 스케일 특징 추출 네트워크(206)(MSFENet)에 공급되고 원시 정보 보존을 위해 정보 풀(204)(IP)에 공급된다.here

and

represents the convolution operation of the panchromatic path and the multi-scale path, respectively.

denotes a bicubic upsampling operation. Then the shallow features

and

is supplied to the multi-scale feature extraction network 206 (MSFENet) to further extract multi-scale features and to the information pool 204 (IP) for preservation of raw information.

In the case of the multi-scale feature extraction network 206 (MSFENet), a first multi-scale feature extraction network 222 and a second multi-scale feature extraction network 224, which are two sub-networks having the same architecture and shared weights, are used, respectively. Capture multi-scale information of panchromatic and multispectral images.

,

,

을 생성한다.Each sub-network incorporates an encoder-decoder structure and a coarse-fine strategy for effective multi-scale feature extraction. Details of the multi-scale feature extraction network 206 (MSFENet) will be described later. Each subnetwork takes as input the shallow features of the shallow feature extractor 202 (SFE) and has three outputs which can be formulated as follows:

,

,

generate

과

는 각각 팬크로마틱에 대한 서브 네트워크와 멀티 스케일에 대한 서브 네트워크의 합성 함수를 나타낸다.

,

,

은 각각 스몰 스케일(small scale), 미들 스케일(middle scale) 및 라지(large scale) 스케일 특징을 나타낸다. 본 발명에서 스케일은 특징 맵의 해상도를 나타낸다. 스몰 스케일 특징들은 거친(대략적인) 정보가 포함된 저해상도 특징 맵을 나타내고, 라지 스케일 특징들은 정제된 정보가 포함된 고해상도 특징 맵을 나타낸다. 이러한 멀티 스케일 특징들은 보완 정보 융합을 위해 멀티 스케일 특징 융합부(226)(MSFF)로 전달된다.here

class

denotes the composite function of a subnetwork for panchromatic and a subnetwork for multiscale, respectively.

,

,

denotes characteristics of a small scale, a middle scale, and a large scale, respectively. In the present invention, scale represents the resolution of a feature map. Small-scale features represent a low-resolution feature map with coarse (rough) information, and large-scale features represent a high-resolution feature map with refined information. These multi-scale features are transferred to the multi-scale feature fusion unit 226 (MSFF) for complementary information fusion.

,

를 입력으로 연결하고, 입력을 단일 컨볼루션 레이어로 처리하여 원시 정보를 갖는 출력 특징들을 생성할 것이다. 이 부분은 다음과 같이 나타낼 수 있다.Preserving raw information in relation to the information pool 204 (IP) is a simple but useful module. The intuitive motivation of the information pool 204 is a feature reuse and memory mechanism, meaning that shallow features containing raw spatial and spectral information help improve the performance of fan sharpening. As shown in FIG. 2, the information pool 204 includes a panchromatic image, an upsampled multispectral image, and shallow features.

,

to the input, and process the input with a single convolutional layer to generate output features with raw information. This part can be expressed as:

는 원시 정보를 나타낸다.

는 스킵 연결을 통해 후속 특징 융합을 위해 멀티 스케일 특징 융합부(226)(MSFF)로 전송된다. 따라서 그것은 기울기 역전파에도 이점이 있다.

와

는 각각 정보 풀(204)(IP)에서의 컨벌루션 레이어 및 연결(concatenation) 연산을 나타낸다.

은 쌍삼차 업샘플링 연산을 나타낸다.here

represents raw information.

is transmitted to the multi-scale feature fusion unit 226 (MSFF) for subsequent feature fusion through skip connection. So it also benefits from gradient backpropagation.

and

denotes the convolutional layer and concatenation operation in the information pool 204 (IP), respectively.

denotes a bicubic upsampling operation.

As a result, the multi-scale features of the multi-scale feature extraction network 206 (MSFENet) and raw information of the information pool 204 (IP) are utilized in the multi-scale feature fusion unit 226 (MSFF) for complementary information fusion, Generates fused features such as:

는 멀티 스케일 특징 융합부(226)(MSFF)의 특징을 나타낸다. 멀티 스케일 특징 융합부(226)(MSFF)에 대한 자세한 내용은 추후 설명될 것이다.here

denotes the characteristics of the multi-scale feature fusion unit 226 (MSFF). Details of the multi-scale feature fusing unit 226 (MSFF) will be described later.

는 재구성부(210)의 입력으로 사용된다. 재구성부(210)는 K개의 잔여 블록(228)과 꼬리에 있는 단일 컨볼루션 레이어(230) 및 제1 가산기(232)로 구성된다. 공정한 비교를 위해 매개변수의 수를 TFNet과 거의 같게 하기 위해 K를 12로 설정했다. 잔여 블록은 ResNet에서 처음 제안되었으며 다양한 컴퓨터 비전 작업에서 지배적인 역할을 했다. Then the information feature

is used as an input of the reconstruction unit 210. The reconstruction unit 210 is composed of K residual blocks 228, a single convolution layer 230 at the tail, and a first adder 232. For a fair comparison, we set K to 12 to make the number of parameters almost equal to TFNet. Residual blocks were first proposed in ResNet and have played a dominant role in various computer vision tasks.

For residual block adoption, Ledig et al. proposed SRResNet and SRGAN for SISR. However, batch normalization (BN) of the original residual block is not suitable for image reconstruction work because it changes the diversity of feature values. Therefore, Lim et al. obtained satisfactory performance by removing the BN layer from the remaining block. Therefore, we apply the residual block as in Lim et al. Details of the remaining blocks are shown in FIG. 2 . A residual image I _res is generated through the residual block 228 and a single convolutional layer 230 .

는 업샘플링된 멀티 스펙트럼 이미지의 전역 잔여 학습에 의해 얻을 수 있다.Here, H _re () denotes an overlapping function of the reconstruction unit 210. Therefore, the final fan-sharpened

can be obtained by global residual learning of upsampled multispectral images.

은 쌍삼차 업샘플링 연산을 나타낸다.here

denotes a bicubic upsampling operation.

Table 1 shows detailed parameter settings of a real-time image fusion device (MSFN) for remote sensing based on deep learning according to an embodiment of the present invention.

Multi-scale feature extraction network (206)

Regarding the multi-scale feature extraction network 206 (MSFENet: Multi-Scale Feature Extraction Net), the first multi-scale feature extraction sub-network 222 and the second multi-scale feature are two sub-networks having the same architecture and shared weights. The extraction sub-network 224 is used to capture multi-scale information of the multi-spectral image and the panchromatic image, respectively.

As in Fig. 3, each sub-network 222, 224 introduces an encoder-decoder structure. Since the two sub-networks are exactly the same, the first multi-scale feature extraction sub-network 222 of the multi-spectral path is taken as an example here.

이고, 반면에 대응하는 출력들은

,

,

이고, 각각 멀티 스펙트럼 이미지의 멀티 스케일 특징들을 지칭한다. 인코더 단에서, 초기에 컨벌루션 레이어를 사용하여 로우-레벨 특징을 추출한다.input is

, while the corresponding outputs are

,

,

, and denote multi-scale features of the multispectral image, respectively. At the encoder stage, we initially extract low-level features using convolutional layers.

는 스트라이드가 1인 컨벌루션 레이어를 나타낸다.

는 스킵 연결 및 추가적인 인코딩을 위해 사용될 것이다. 그 다음 스트라이드가 2인 컨벌루션 레이어를 적용하여 공간 해상도를 절반으로 낮추고 입력 특징 맵들의 특징 채널들을 2배로 늘린다.

denotes a convolutional layer with a stride of 1.

will be used for skip concatenation and additional encoding. Then, a convolutional layer with a stride of 2 is applied to reduce the spatial resolution by half and double the feature channels of the input feature maps.

는 방금 언급된 컨벌루션 연산을 나타낸다.

는 스킵 연결 및 추가 인코딩을 위해 사용될 것이다. 후속적으로, 스트라이드가 2인 컨벌루션 레이어를 적용하여 두번째 다운샘플링 프로세스를 수행한다.

denotes the convolution operation just mentioned.

will be used for skip concatenation and further encoding. Subsequently, a second downsampling process is performed by applying a convolutional layer with a stride of 2.

는

와 유사한 컨벌루션 연산을 나타낸다. 공간 크기를 절반으로 줄이고

의 특징 채널들을 2배로 늘리는,

는 스킵 연결 및 추가 인코딩을 위해 사용될 것이다.

Is

represents a convolution operation similar to cut the space in half

doubling the feature channels of

will be used for skip concatenation and further encoding.

The decoder mainly includes a small-scale feature extraction module 310 (SSFEM: small-scale feature extraction module), a middle-scale feature extraction module 314 (MSFEM: middle-scale feature extraction module) and a large-scale feature extraction module 318 ( It consists of three parts: large-scale feature extraction module (LSFEM).

의 특징 채널들과 같다. 또한 마지막 컨벌루션 레이어의 출력에

를 로컬 잔여 학습으로 추가하여, 메모리를 유지하고 정보 흐름에 이점을 줄 수 있다. 이 부분은 다음과 같이 설명할 수 있다.The small scale feature extractor 310 (SSFEM) is composed of four residual blocks 328 and a convolutional layer 330 at the tail. the number of filters

It is the same as the characteristic channels of Also in the output of the last convolutional layer

can be added as local residual learning to conserve memory and benefit information flow. This part can be explained as follows.

,

,

및

는 스몰 스케일 특징 추출부(310)(SSFEM)에서 4개의 잔여 블록들의 특징들을 나타낸다.

는 마지막 컨벌루션 레이어(330)의 연산을 나타낸다.

는 추출된 스몰 스케일 특징을 나타낸다.

,

,

and

denotes features of four residual blocks in the small scale feature extractor 310 (SSFEM).

represents the operation of the last convolutional layer 330.

denotes the extracted small-scale features.

는 먼저 스트라이드가 2인 디컨벌루션 레이어(332)에 의해 처리되어,

의 공간 크기가 두 배이고 특징 채널이 절반인 특징 맵을 생성한다. 그런 다음 꼬리에 두 개의 잔여 블록(324)과 컨볼루션 레이어(336)를 적용한다. 또한, 스킵 연결을 위해

가 미들 스케일 특징 추출부(312)(MSFEM)의 출력에 가산된다. 프로세스는 다음과 같이 공식화할 수 있다.In the middle scale feature extraction unit 312 (MSFEM), the input

is first processed by the deconvolution layer 332 with a stride of 2,

Create a feature map with twice the spatial size of and half the feature channels. Then, two residual blocks 324 and a convolution layer 336 are applied to the tail. Also, for skip connections

is added to the output of the middle scale feature extractor 312 (MSFEM). The process can be formulated as follows.

과

은 M미들 스케일 특징 추출부(312)(MSFEM)에서의 두 잔여 블록들(334)의 특징들을 나타낸다.

와

는 각각 첫번째 디컨벌루션 레이어(332)와 마지막 컨벌루션 레이어(334)의 연산을 나타낸다.

는 추출된 미들 스케일 특징을 나타낸다.

class

represents the features of the two residual blocks 334 in the M-middle-scale feature extractor 312 (MSFEM).

and

represents the operation of the first deconvolution layer 332 and the last convolution layer 334, respectively.

denotes the extracted middle scale features.

는 디컨벌루션 레이어(338)에서 처리된다. 이 레이어(338)를 통해

의 두 배의 공간 해상도와 절반의 채널을 가진 특징 맵을 얻는다.In the case of the large-scale feature extractor 318 (LSFEM), it is composed of a deconvolution layer 338 with a stride of 2 at the beginning, a residual block 340 in the middle, and a convolution layer 342 at the tail. Initially, the input of the large scale feature extraction unit 318 (LSFEM),

is processed in the deconvolution layer 338. through this layer 338

to obtain a feature map with twice the spatial resolution and half the channels of

를 가산한다. 이러한 방식으로, 라지 스케일 특징

가 획득된다. 이 부분은 다음과 같이 공식화할 수 있다.Subsequently, these feature maps are input to a residual block 340 and a convolution layer 342 to generate an output. In addition, to utilize the residual learning, the output for skip connection

add up In this way, large-scale features

is obtained This part can be formulated as follows.

은 라지 스케일 특징 추출부(318)(LSFEM)에서 초기 디컨벌루션 레이어(328)의 특징을 나타낸다.

및

은 각각 중간의 잔여 블록(340) 및 마지막 컨벌루션 레이어(342)의 연산을 나타낸다.

represents a feature of the initial deconvolution layer 328 in the large-scale feature extractor 318 (LSFEM).

and

denotes the operation of the middle residual block 340 and the last convolutional layer 342, respectively.

In the decoder stage, multi-scale features are progressively decoded, incorporating a coarse-to-fine strategy to obtain progressively finer details. Specifically, four residual blocks are used for small scale features, while two residual blocks are used for middle scale features and one residual block is used for middle scale features. Therefore, not only small-scale features but also middle-scale features and large-scale features are extracted richly to provide better subsequent fusion. It is worth mentioning that we keep residual blocks of 4, 2 and 1 for feature extraction at different scales.

,

및

가 획득된다. 마찬가지로 팬크로마틱 경로의 제2 멀티 스케일 특징 추출 서브 네트워크(224)는

,

및

을 생성한다. 후속적으로, 이들 6개의 출력은 추가 보완 정보 융합을 위해 멀티 스케일 특징 융합부(208, 226)(MSFF)에 제공될 것이다.In this way, the three multi-scale outputs of the first multi-scale feature extraction sub-network 222, namely

,

and

is obtained Similarly, the second multi-scale feature extraction subnetwork 224 of the panchromatic path

,

and

generate Subsequently, these six outputs will be provided to the multi-scale feature fusion section 208, 226 (MSFF) for further complementary information fusion.

Multi-scale feature fusion (208, 226)

The multi-scale feature of the multi-scale feature extraction network 206 (MSFENet) and the original information of the information pool 204 (IP) are multi-scale feature fusion units 206 and 208 (MSFF: multi-scale feature for complementary information fusion) fusion).

As can be seen in FIG. 4 , the multi-scale feature fusion blocks 206 and 208 (MSFF) mainly include the small-scale feature fusion block 400 (SSFFB) and the middle-scale feature fusion block 402 ( It consists of three parts: middle-scale features fusion block (MSFFB) and comprehensive features fusion block (CFFB).

와

가 연결되어 컨벌루션 레이어(408)의 입력으로 사용된다. 그런 다음 스몰 스케일 특징들이 저해상도(LR: low resolution) 공간에 융합되어, 계산 부담이 크게 줄어든다. 마지막으로 융합된 특징은 추가 융합을 위해 디컨벌루션 레이어(410)에 의해 스케일 인자 x4로 업샘플링된다. 이 프로세스는 다음과 같이 나타낼 수 있다.In the case of the small scale feature fusion unit 400 (SSFFB), a convolution layer 408 having a kernel size of 1 (kernel_size=1) and a deconvolution layer 410 having a stride of 4 are configured. by connection 406

and

is connected and used as an input of the convolutional layer 408. The small scale features are then fused into a low resolution (LR) space, greatly reducing the computational burden. Finally, the fused features are upsampled by a scale factor x4 by the deconvolution layer 410 for further fusion. This process can be represented as:

는 연결 동작을 나타낸다.

와

는 각각 연결된 특징을 압축하기 위한 1×1 컨벌루션 레이어(408)와 스트라이드가 4인 디컨벌루션 레이어(410)를 나타낸다.

는 스몰 스케일의 융합된 특징을 나타낸다.here

indicates a connection operation.

and

represents a 1×1 convolution layer 408 for compressing connected features and a deconvolution layer 410 having a stride of 4, respectively.

represents a small-scale fused feature.

와

의 연결을 입력으로 사용하고 1×1 컨볼루션 레이어(414)를 사용하여 중간 해상도(MR: medium-resolution) 공간에서 미들 스케일 특징들을 융합한다. 후속적으로, 융합된 특징은 추가 융합을 위해 스트라이드가 2인 디컨벌루션 레이어(416))를 사용하여 업샘플링된다. 이 부분은 다음과 같이 설명할 수 있다.Similarly, the middle scale feature fusion part 402 (MSFFB)

and

It takes as input the concatenation of , and fuses middle-scale features in medium-resolution (MR) space using a 1×1 convolutional layer 414 . Subsequently, the fused features are upsampled using a deconvolution layer 416 with a stride of 2 for further fusion. This part can be explained as follows.

는 연결 동작을 나타낸다.

과

은 각각 1×1 컨벌루션 레이어(414)와 스트라이드가 2인 디컨벌루션 레이어(416)를 나타낸다.

는 미들 스케일의 융합된 특징을 나타낸다.here

indicates a connection operation.

class

denotes a 1×1 convolution layer 414 and a deconvolution layer 416 having a stride of 2, respectively.

represents the fused character of the middle scale.

,

,

,

및

는 연결되어 종합적인 정보, 즉, 멀티 스케일 보완 정보 및 원시 정보를 융합하기 위하여 1×1 컨볼루션 레이어를 나타내는 종합 특징 융합부(404)(CFFB)에 제공된다. 이 마지막 융합된 특징은 다음과 같이 공식화될 수 있다.finally,

,

,

,

and

is concatenated and provided to the synthetic feature fusion unit 404 (CFFB) representing a 1×1 convolutional layer to fuse synthetic information, that is, multi-scale complementary information and original information. This last fused feature can be formulated as:

는 연결 동작을 나타낸다.

는 종합 특징 융합부(404)(CFFB)에서 1×1 컨볼루션 레이어(420)를 나타낸다.here

indicates a connection operation.

denotes the 1x1 convolutional layer 420 in the synthetic feature fusion section 404 (CFFB).

real-time implementation

Parameters and computational complexity are two essential factors for real-world applications and real-time processing. Therefore, in the present invention, there are two measures for real-time implementation. Meanwhile, the weights of the two encoder-decoders, that is, the first multi-scale feature extraction sub-network 222 and the second multi-scale feature extraction sub-network 224, which are two sub-networks of the multi-scale feature extraction network 206 (MSFENet) ) is shared to reduce parameters. On the other hand, for fewer computations, small scale features are fused to low resolution (LR) space and middle scale features are fused to medium resolution (MR) space.

Weight sharing encoder-decoder structure

The multi-scale feature extraction network 206 (MSFENet) uses two sub-networks with the same encoder-decoder architecture to capture the multi-spectral image and the multi-scale complementary context information of the corresponding panchromatic image. The encoder-decoder architecture enables a larger receptive field, conducive to effective multi-scale feature extraction. However, the encoder-decoder architecture also introduces a huge number of parameters. The parameters of the multi-scale feature extraction network 206 (MSFENet) are halved as both encoder-decoders share the same weights to utilize the encoder-decoder architecture for favorable performance and keep the network reasonably sized.

Fusion in low-resolution space for small-scale features and fusion in medium-resolution space for middle-scale features

와

) 또는 미들 스케일 특징(

와

)으로부터 융합된 특징을 획득하기 위한 주된 2가지 방법이 존재한다. 첫번째 방법은 도 3에 도시된 파이프라인, 연결부(Concat), 컨벌루션 레이어(Conv) 및 디컨벌루션 레이어(Deconv)를 채택하는 것이고, 반면에 두 번째 방법은 디컨벌루션 레이어(Deconv), 연결부(Concat) 및 컨벌루션 레이어(Conv)의 또 다른 파이프라인을 채택한다.Complementary small scale features (

and

) or middle scale characteristics (

and

) There are two main methods for obtaining fused features. The first method is to adopt the pipeline shown in Fig. 3, the convolution (Concat), the convolution layer (Conv) and the deconvolution layer (Deconv), while the second method adopts the deconvolution layer (Deconv), the convolution (Concat) and another pipeline of convolutional layers (Conv).

The former fuses complementary features in low-resolution (LR) or medium-resolution (MR) space, while the latter fuses them in high-resolution (HR) space. Obviously, the former has a smaller computational complexity because the computational cost increases as the spatial resolution increases and the spatial resolution of feature maps in low-resolution (LR) or medium-resolution (MR) space is lower than that in high-resolution (HR) space.

loss function

Besides the network architecture, the loss function is another important factor affecting the performance of fan sharpening. The L ₂ loss function is commonly applied in conventional SISR and fan sharpening tasks. Nevertheless, it is more likely that the L ₂ loss function falls into a local minimum, leading to unexpected artifacts. Therefore, in the present invention, by adopting the L ₁ loss function to optimize the network, better luminance and color can be preserved, and it converges faster than the L ₂ loss function.

와 같이 표시되는데,

와

은 각각 저해상도 멀티 스펙트럼 이미지와 팬크로마틱 이미지를 나타낸다.

는 대응하는 그라운드-트루스(ground truth) 고해상도 멀티 스펙트럼 이미지를 나타낸다. 따라서, 본 발명에 의한 멀티 스펙트럼 융합 네트워크의 손실 함수는 다음과 같이 나타낼 수 있다.training set is

displayed as

and

denotes a low-resolution multispectral image and a panchromatic image, respectively.

denotes the corresponding ground-truth high-resolution multispectral image. Therefore, the loss function of the multispectral convergence network according to the present invention can be expressed as follows.

및

는 각각 본 발명의 일 실시예에 의한 딥러닝에 기반한 원격 감지를 위한 실시간 이미지 융합 장치(MSFN)의 훈련 샘플의 수(크기), 함수 및 매개변수들을 나타낸다.N,

and

Respectively represents the number (size) of training samples, functions and parameters of the real-time image fusion device (MSFN) for remote sensing based on deep learning according to an embodiment of the present invention.

Experiment result

dataset

In order to demonstrate the superiority of the real-time image fusion device and method for remote sensing based on deep learning according to an embodiment of the present invention, experiments on four data sets of QuickBird, IKONOS, Spot-6, and Pleiades are implemented. The spectral wavelength and spatial resolution characteristics of these satellites are summarized in Table 2. We follow Wald's protocol for experimental simulations, taking into account that high-resolution multispectral images do not actually exist in the fan-sharpening dataset. In particular, both the multispectral image and the panchromatic image are downsampled by a scale factor of 4. In this way, downsampled multispectral images and panchromatic images can be utilized as network inputs, and original multispectral images can be used as references for network training and method evaluation.

implementation details

The four previously mentioned data sets are separately used to implement the experiments. For each data set, upsampled low-resolution multispectral (LRMS)/panchromatic/high-resolution multispectral (HRMS) image pairs are initially extracted with a size of 256 × 256. It is then split 90/10% for training/testing. In addition, data augmentation is employed for each training set including random cropping, rotation and flipping. In the training phase, the mini-batch size is set to 8 and each patch pair is randomly cropped to 64×64 from the training image pair. The training process of each experiment takes 1000 epochs. Apply Adam to optimize the network. The initial learning rate is set to 0.0001 and reduced by a factor of 0.5 every 200 epochs. All deep learning-based experiments are implemented in Pytorch on a desktop with a GTX1080Ti GPU. Experiments of the existing method are performed in MATLAB R2018a with a 4.2GHz Intel i7-7700K CPU.

Ablation studies

To verify the efficiency of information pool (IP), coarse-to-fine (CTF) strategy (CTF), and multi-scale convergence (MSF), we analyze the performance of five networks: baseline (without IP, CTF, and MSF), MSFN without IP, MSFN without CTF, MSFN without MSF, and Suggested MSFN. All ablation experiments are performed on the QuickBird data set. The results of the ablation experiments and the corresponding loss curves are shown in Table 3 and FIG. 6A, respectively.

Information Pool 204 (IP).

The information pool 204 (IP) is a module that is simple but helps to preserve basic information, while useful for gradient back-propagation. Intuitive motives are feature reuse and memory mechanisms, implying that shallow features containing raw spatial and spectral information have the potential to improve the performance of fan sharpening. As shown in Table 3, the results in the second and last rows show the efficiency of the information pool 204 (IP).

Coarse-to-fine (CTF) strategy

For the decoder stage of MSFENet, the present invention incorporates a coarse-fine strategy that progressively decodes multi-scale features to obtain finer details step by step. Specifically, four residual blocks are used for small scale features, while two residual blocks are used for middle scale features and one residual block is used for large scale features. In this way, not only small-scale features, but also middle-scale features and large-scale features are extracted richly, making for better subsequent fusion. To demonstrate the efficiency of CTF, we remove residual blocks from MSFEM and LSFEM and merge all 7 residual blocks into SSFEM. As shown in Table 3, the performance of the network without CTF is inferior to that of MSFN, indicating the advantage of CTF.

Multi-scale fusion (MSF)

와

을 융합하여 보완 정보를 획득하고, 표 3의 네 번째 행에 결과를 생성한다. 단일 스케일 융합과 멀티 스케일 융합 사이의 추가 비교를 위해, 스몰 스케일 융합과 미들 스케일 융합 각각을 가지고 2개의 단일 스케일 유합 실험을 수행하였다. 스몰 스케일 융합은 스몰 스케일 특징인

와

만을 융합하는 것을 의미하고, 미들 스케일 융합은 미들 스케일 특징인

와

만을 융합하는 것을 의미한다. 비교 결과는 표 4에 표시되고, 해당 손실 곡선은 도 6b에 표시된다. 멀티 스케일 융합이 실제로 단일 스케일 융합보다 성능이 우수하다는 것은 분명하며, 이는 MSF의 효율성을 보여준다. 전반적으로 모든 IP, CTF 및 MSF는 팬 샤프닝의 더 나은 성능에 유리하다.MSF is an essential part to fully exploit the layered complementary properties of panchromatic and multispectral images. For the ablation study of MSF, single-scale fusion is applied instead of MSF. In particular, large scale features

and

to obtain complementary information, and generate the result in the fourth row of Table 3. For further comparison between single-scale fusion and multi-scale fusion, two single-scale fusion experiments were performed with small-scale fusion and middle-scale fusion, respectively. Small scale fusion is a small scale characteristic.

and

Means to fuse only, and middle scale fusion is a characteristic of middle scale

and

means merging only The comparison results are shown in Table 4, and the corresponding loss curves are shown in Fig. 6b. It is clear that multi-scale fusion actually outperforms single-scale fusion, demonstrating the efficiency of MSF. Overall, all IP, CTF and MSF are in favor of better performance of fan sharpening.

Comparison with other methods

Quantitative evaluation and visual performance

In order to confirm the effect of MSFN according to the present invention, four traditional methods and several deep learning-based methods, including Indusion, MMP, AWLP, GS, PNN, DiCNN1, PanNet, ECNN, DRPNN, BIEDN and TFNet, were used according to the present invention. Compare with method. For quantitative evaluation, six metrics widely used in this experiment were adopted: correlation coefficient (CC), relative global dimension synthesis error (ERGAS), four-band extension of universal image quality index (Q ₄ ), and spectral angle mapper (SAM). ), relative mean spectral error (RASE) and root mean square error (RMSE). The objective quantitative results for the four previously mentioned data sets are listed in Tables 5, 6, 7 and 8, respectively. For each data set, the final result is calculated by averaging the results for the corresponding test image.

As can be seen from these tables, it can be seen that the MSFN according to the present invention achieves the best performance among all comparative fan sharpening methods for all objective evaluation indicators. These quantitative results demonstrate the superiority of the MSFN of the present invention, not only succeeding in recovering spatial details but also preserving spectral information.

In addition to objective results, it provides several visual comparison results. Figures 7 to 9 show these subjective results of QuickBird, IKONOS and Pleiades, respectively. The first three bands of the multispectral image are selected for visual comparison. Corresponding residual images are also provided to highlight the differences for better visual inspection. The residual image represents the difference between the pan sharpened image and the reference image. Clearly, the smaller the difference, the better the reconstruction. In the enlarged region of Fig. 7, it is easy to see that most deep learning-based methods tend to obtain smoother spatial details, whereas traditional methods tend to obtain clear spatial information but suffer from spectral distortion. It is clear that only the method according to the present invention achieves a satisfactory preservation in both the spectral and spatial domains, especially in the enlarged domain three black dots are noted. The residual image further demonstrates that the method according to the present invention has the least spatial and spectral distortion compared to the reference image. As can be seen in Fig. 8, traditional methods such as Indusion, AWLP and GS noticeably improve spatial detail at the cost of introducing serious spectral distortion, especially for viewing green fields and canyons. The same phenomenon is seen in some deep learning-based methods such as PNN, DiCNN1, ECNN and DRPNN. Conversely, MMP and PanNet have less spectral distortion but recover smoother spatial detail.

In contrast, TFNet and our method achieve satisfactory spectral and spatial conservation. For further observation, the method according to the present invention reduces spectral distortion while obtaining sharper shapes of canyons and green ground, as can be seen in the magnified region of Fig. 8 and the corresponding residual images.

As can be seen in Fig. 9, the traditional method improves spatial detail impressively, but at the same time spectral distortion is noticeable, especially for AWLP and GS. Deep learning-based methods are more likely to achieve more satisfactory reconstructions. However, PNN, DiCNN1, PanNet, and ECNN tend to have artifacts in complex texture regions depending on the residual image. In contrast, DRPNN, TFNet and our method produce better results. For further observation, the method according to the present invention produces the clearest straight contours of the drainage channel without spectral distortion and more similar to the reference image. The comparison mentioned above shows the superiority of the MSFN according to the present invention. In summary, the method according to the present invention achieves good performance in terms of quantitative metrics and visual quality.

nonparametric test

In addition to quantitative metrics and visual quality, we perform some non-parametric statistical tests that have proven powerful for comparing different methods. Three widely applied nonparametric tests including Friedman, Aligned Friedman and Quade are used in this experiment for testing statistical significance. For each test, the average rank of all methods is calculated according to their performance on different datasets in terms of CC metrics.

The results of the Friedman, Aligned Friedman and Quade nonparametric tests are listed in Tables 9, 10 and 11, respectively. The lower the rank number, the better the method. As can be seen from this table, the MSFN according to the present invention achieves the best performance for all comparative fan sharpening methods. All three non-parametric tests show the superiority of the method according to the present invention.

real-time evaluation

For temporal evaluation, pairs of upsampled LRMS and panchromatic images of size 1024×1024 are directly fed into this trained model to generate pan-sharpened images. The results of each method for different datasets are shown in Table 12. Results indicate that the method according to the present invention can achieve real-time performance by generating one image in 0.0068 seconds. It is worth noting that the method according to the present invention takes more time to reconstruct the pan sharpened image than other deep learning based methods except BIEDN. This is reasonable since the purpose of the method according to the present invention is to improve performance as much as possible in the conditions of real-time applications. As mentioned earlier, MSFN achieves the best performance of all comparative fan sharpening methods. Therefore, the method according to the present invention can achieve more effective performance while meeting the requirements of real-time implementation.

Conclusion and Future Challenges

In order to meet the requirements of real-time implementation and achieve more effective performance at the same time, the present invention proposes a multi-scale convergence network (MSFN) for pan sharpening task that can make full use of the hierarchical complementary features of panchromatic and multispectral images. do. In the proposed MSFN, an encoder-decoder structure and a coarse-fine strategy are introduced to effectively extract multi-scale features of panchromatic images and multispectral images separately. On the other hand, by adopting an information pool to preserve raw information, it also provides advantages for gradient backpropagation.

In addition, the multi-scale fusion module is applied to fuse the multi-scale features from the decoder and information pool, making full use of the hierarchical complementary spatial and spectral information. Extensive experiments show the superiority of the method of the present invention for a fan sharpening method.

Although the present invention has been described in detail through specific examples, this is for explaining the present invention in detail, the present invention is not limited thereto, and within the technical spirit of the present invention, those skilled in the art It will be clear that the modification or improvement is possible by

All simple modifications or changes of the present invention fall within the scope of the present invention, and the specific protection scope of the present invention will be clarified by the appended claims.

200: Real-time image fusion device for remote sensing based on deep learning
202: shallow feature extraction unit 204: information pool
206: multi-scale feature extraction network 208: multi-scale feature fusion unit
210: reconstruction unit
212: upsampled low resolution multispectral image
214, 216, 220, 230, 234, 238: convolutional layer
218: connection part
222: first multi-scale feature extraction subnetwork
224: second multi-scale feature extraction subnetwork
226: multi-scale feature convergence unit 228: remaining block
232, 240: Adder 236: ReLU
300: encoder 302: decoder
304, 306, 308, 330, 336, 342: convolutional layer
310, 322: small scale feature extraction unit 312, 316, 320: adder
314, 324: middle scale feature extraction unit
318, 326: large scale feature extraction unit 328, 334, 340: residual block
332, 338: deconvolution layer 400: small scale feature convergence
402: Middle scale feature fusion part 404: Comprehensive feature fusion part
406, 412, 418: connection part
408, 414, 420: convolution layer
410, 416: deconvolution layer

Claims (12)

a shallow feature extraction unit extracting shallow features from the upsampled low-resolution multi-spectral image and the pan-chromatic image;
An information pool for generating raw information features based on the panchromatic image, the upsampled low-resolution multispectral image, shallow features of the upsampled low-resolution multispectral image, and shallow features of the panchromatic image ;
a first multi-scale feature extraction sub-network generating small-scale features, middle-scale features, and large-scale features of the up-sampled low-resolution multi-spectral image based on shallow features of the up-sampled low-resolution multi-spectral image;
a second multi-scale feature extraction sub-network generating small-scale features, middle-scale features, and large-scale features of the panchromatic image based on shallow features of the panchromatic image;
The raw information feature, the small-scale feature, middle-scale feature, and large-scale feature of the upsampled low-resolution multi-spectral image, and the multi-scale feature fusing the small-scale feature, middle-scale feature, and large-scale feature of the panchromatic image. fusion; and
A real-time image fusion device for remote sensing based on deep learning comprising a reconstruction unit for reconstructing a high-resolution multi-spectral image based on the fused feature and the upsampled low-resolution multi-spectral image. The method of claim 1,
The shallow feature extraction unit,
a first convolution layer that performs a convolution operation on the upsampled low-resolution multispectral image and outputs a shallow feature of the upsampled low-resolution multispectral image; and
A real-time image fusion device for remote sensing based on deep learning, comprising a second convolution layer that performs a convolution operation on the panchromatic image and outputs a shallow feature of the panchromatic image. The method of claim 1,
The information pool,
a connection unit connecting the pan-chromatic image, the up-sampled low-resolution multi-spectral image, a shallow feature of the up-sampled low-resolution multi-spectral image, and a shallow feature of the pan-chromatic image; and
A real-time image fusion device for remote sensing based on deep learning, comprising a third convolution layer for performing a convolution operation on the output of the connection unit and outputting the original information feature. The method of claim 1,
The first multi-scale feature extraction subnetwork,
a first encoder that lowers spatial resolution of shallow features of the upsampled low-resolution multispectral image and increases feature channels of feature maps; and
A first decoder for generating a small scale feature, a middle scale feature, and a large scale feature of the upsampled low-resolution multispectral image based on an output of the first encoder;
The second multi-scale feature extraction subnetwork,
a second encoder that lowers spatial resolution of shallow features of the panchromatic image and increases feature channels of feature maps; and
A real-time image fusion device for remote sensing based on deep learning comprising a second decoder for generating small scale features, middle scale features, and large scale features of the panchromatic image based on the output of the second encoder. The method of claim 4,
The real-time image fusion device for remote sensing based on deep learning, wherein the first multi-scale feature extraction sub-network and the second multi-scale feature extraction sub-network share weights used in a convolution layer. The method of claim 1,
The multi-scale feature fusion part,
a small-scale feature fusion unit fusing the small-scale feature of the upsampled low-resolution multi-spectral image with the small-scale feature of the panchromatic image and outputting a fused small-scale feature;
a middle scale feature fusion unit fusing middle scale features of the upsampled low-resolution multispectral image with middle scale features of the panchromatic image and outputting a fused middle scale feature; and
The fused small-scale feature, the fused middle-scale feature, the large-scale feature of the upsampled low-resolution multispectral image, the large-scale feature of the panchromatic image, and the raw information feature are fused to output a final fused feature. A real-time image fusion device for remote sensing based on deep learning, including a comprehensive feature fusion unit that (A) extracting, by a shallow feature extraction unit, shallow features from the upsampled low-resolution multi-spectral image and the panchromatic image;
(B) an information pool based on the panchromatic image, the upsampled low resolution multispectral image, shallow features of the upsampled lowresolution multispectral image, and shallow features of the panchromatic image; generating an information feature;
(C) a first multi-scale feature extraction subnetwork generates small-scale features, middle-scale features, and large-scale features of the upsampled low-resolution multispectral image based on shallow features of the upsampled low-resolution multispectral image step;
(D) generating, by a second multi-scale feature extraction sub-network, small-scale features, middle-scale features, and large-scale features of the panchromatic image based on the shallow features of the panchromatic image;
(E) a multi-scale feature fusion unit, the raw information feature, the small-scale feature, middle-scale feature and large-scale feature of the upsampled low-resolution multi-spectral image, and the small-scale feature, middle-scale feature of the panchromatic image, and fusing large scale features; and
(F) a real-time image fusion method for remote sensing based on deep learning comprising the step of reconstructing, by a reconstruction unit, a high-resolution multispectral image based on the fused feature and the upsampled low-resolution multispectral image. The method of claim 7,
In the step (A),
outputting shallow features of the upsampled low resolution multispectral image by performing a convolution operation on the upsampled low resolution multispectral image; and
A real-time image fusion method for remote sensing based on deep learning comprising the step of performing a convolution operation on the panchromatic image and outputting a shallow feature of the panchromatic image. The method of claim 7,
In the step (B),
(B-1) connecting the panchromatic image, the upsampled low-resolution multispectral image, shallow features of the upsampled lowresolution multispectral image, and shallow features of the panchromatic image; and
(B-2) performing a convolution operation on the output of step (B-1) to output the original information feature; The method of claim 7,
In the step (C),
(C-1) lowering spatial resolution of shallow features of the upsampled low-resolution multi-spectral image and increasing feature channels of feature maps; and
(C-2) generating a small-scale feature, a middle-scale feature, and a large-scale feature of the upsampled low-resolution multispectral image based on the output of step (C-1);
In the step (D),
(D-1) lowering spatial resolution of shallow features of the panchromatic image and increasing feature channels of feature maps; and
(D-2) generating small-scale features, middle-scale features, and large-scale features of the panchromatic image based on the output of step (D-1), for remote sensing based on deep learning. Real-time image fusion method. The method of claim 10,
The step (C) and the step (D) share the weight used in the convolution layer, real-time image fusion device for remote sensing based on deep learning. The method of claim 7,
In the step (E),
fusing small scale features of the upsampled low-resolution multispectral image with small scale features of the panchromatic image and outputting a fused small scale feature;
fusing middle-scale features of the upsampled low-resolution multi-spectral image with middle-scale features of the panchromatic image and outputting a fused middle-scale feature; and
The fused small-scale feature, the fused middle-scale feature, the large-scale feature of the upsampled low-resolution multispectral image, the large-scale feature of the panchromatic image, and the raw information feature are fused to output a final fused feature. A real-time image fusion method for remote sensing based on deep learning, comprising the step of doing.

Images