CN113191993A

CN113191993A - Panchromatic and multispectral image fusion method based on deep learning

Info

Publication number: CN113191993A
Application number: CN202110425489.7A
Authority: CN
Inventors: 张凯; 盛志; 张风; 王安飞; 刁文秀; 李卓林
Original assignee: Shandong Normal University
Current assignee: Shandong Normal University
Priority date: 2021-04-20
Filing date: 2021-04-20
Publication date: 2021-07-30
Anticipated expiration: 2041-04-20
Also published as: CN113191993B

Abstract

The invention provides a panchromatic and multispectral image fusion method and system based on deep learning, which comprises the following steps: acquiring a full-color image and a multispectral image to be fused; inputting the full-color image and the multispectral image into a pre-trained image fusion model to obtain a fused remote sensing image; the scheme can fully utilize the deep neural network, highlight the spatial structure information of the image and adjust the image style by utilizing the constructed branch spectrum adjusting network, so that the fused remote sensing image not only has high spatial resolution, but also can well reserve the specific style information of a specific satellite.

Description

Panchromatic and multispectral image fusion method based on deep learning

Technical Field

The disclosure belongs to the technical field of remote sensing image processing, and particularly relates to a panchromatic and multispectral image fusion method based on deep learning.

Background

The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art.

Earth remote sensing satellites typically provide two different types of images, namely high-spatial and low-spectral resolution panchromatic images and low-spatial and high-spectral resolution multispectral images. At present, due to technical limitation, a satellite sensor acquires full-color images and multispectral images, and cannot directly acquire multispectral images with high space and high spectral resolution.

The inventor finds that a remote sensing image fusion method through a deep neural network exists in the existing method, but the existing traditional full-color image and multispectral image fusion based on the deep neural network usually trains a model for each satellite, a large amount of training data is needed, separate training data is needed for images of different styles shot by satellites of different sensors, different network models need to be trained for fusing the images of different styles of the different sensors, so that the model training efficiency is low, and meanwhile, the trained fusion model cannot ensure the fusion quality on the premise of no large amount of training data.

Disclosure of Invention

In order to solve the above problems, the present disclosure provides a deep learning-based fusion method for panchromatic and multispectral images, which can make full use of a depth neural network, highlight spatial structure information of an image, and adjust the style of the image by using a constructed branch spectrum adjusting network, so that the fused remote sensing image not only has high spatial resolution, but also can well retain specific style information of a specific satellite.

According to a first aspect of the embodiments of the present disclosure, there is provided a panchromatic and multispectral image fusion method based on deep learning, including:

acquiring a full-color image and a multispectral image to be fused;

inputting the full-color image and the multispectral image into a pre-trained image fusion model to obtain a fused remote sensing image;

the image fusion model comprises a spatial structure enhancement network and a spectral information adjustment network, wherein the spatial structure enhancement network is based on a convolutional neural network and is trained by using remote sensing image training sets of different types of satellites; the spectral information adjusting network comprises a plurality of branch networks, each branch network is trained by using a remote sensing image of a specific satellite, and a fused image is obtained by multiplying output results of the spatial structure enhancing network and the spectral information adjusting network.

Further, the input of the spatial structure enhancement network is a panchromatic image and an initial multispectral image, the panchromatic image and the initial multispectral image are firstly stacked to obtain an image M, 4 convolution modules are adopted in the middle of the image M to extract spatial information, the convolution modules are densely connected, the M is input into a first convolution module, and the output is ms 2; m is stacked with the output ms2 of the first convolution module, and input into the second convolution module, and the output is ms 3; m is stacked with ms2 and ms3, input into a third convolution module and output as ms 4; m is stacked with ms2, ms3, ms4 as input to a fourth convolution module with an output of ms 5; the full color image itself was stacked four times, outputting pan 2; pan2 is then added to ms5, outputting HRMS 1.

Further, the spectral information adjusting network inputs and outputs the spatial structure enhancement network to specific branches for processing, and each branch network comprises 32 convolutional layers of 3 × 3 filters, a global average pooling layer, and two fully-connected layers, namely 4 convolutional layers of 3 × 3 filters; and the spectral information adjusting network adjusts each channel of the spatial structure enhancing network, and the output result is Mask.

Further, performing dot multiplication on the output result HRMS1 of the spatial structure enhancement network and the output result Mask of the spectral information adjustment network, wherein the dot multiplication result is a final high-resolution multispectral image;

further, the acquired panchromatic image and the multispectral image to be fused need to be preprocessed, and the multispectral image with low spatial resolution is subjected to four-time up-sampling operation to obtain an initial multispectral image with the same size as the panchromatic image.

According to a second aspect of the embodiments of the present disclosure, there is provided a panchromatic and multispectral image fusion system based on deep learning, including:

an image acquisition unit for acquiring a full-color image and a multispectral image to be fused;

the image fusion unit is used for inputting the full-color image and the multispectral image into a pre-trained image fusion model to obtain a fused remote sensing image;

According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the memory, wherein the processor implements the deep learning-based panchromatic and multispectral image fusion method when executing the program.

According to a fourth aspect of the embodiments of the present disclosure, there is provided a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a deep learning-based panchromatic and multispectral image fusion method as described.

Compared with the prior art, the beneficial effect of this disclosure is:

(1) the image fusion model disclosed by the invention can fully utilize the deep neural network, highlight the spatial structure information of the image and adjust the image style by utilizing the constructed branch spectrum adjusting network, so that the fused remote sensing image not only has high spatial resolution, but also can well reserve the specific style information of a specific satellite.

(2) According to the scheme, only one common model is required to be trained for various satellite images, and the problem that training data for a certain satellite is less is solved by using common training of various satellite data, so that the training effect of the fusion model is improved, and the quality of the fusion image is improved.

(3) The scheme of the invention improves the utilization rate of data, and is more robust and the fusion result is more stable compared with the prior art.

Advantages of additional aspects of the disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the disclosure.

Drawings

The accompanying drawings, which are included to provide a further understanding of the disclosure, illustrate embodiments of the disclosure and together with the description serve to explain the disclosure and are not to limit the disclosure.

Fig. 1 is a flowchart of a spatial structure enhancement network according to a first embodiment of the disclosure;

fig. 2(a) is a schematic diagram of a branched network structure of a spectral information adjusting network according to a first embodiment of the disclosure.

Fig. 2(b) is a flowchart of a specific branch structure in the spectral information adjusting network according to the first embodiment of the disclosure;

fig. 2(c) is a schematic diagram of dot product of output results of the spatial structure enhancement network and the spectral information adjustment network according to the first embodiment of the disclosure;

fig. 3(a) -3 (d) are graphs comparing the fusion results of the image fusion method according to an embodiment of the present disclosure on a low spatial resolution multispectral image and a high spatial resolution panchromatic image (where fig. 3(a) is the multispectral image with low spatial resolution, fig. 3(b) is the panchromatic image with high spatial resolution, fig. 3(c) is a reference image, and fig. 3(d) is the multispectral image fused according to the method of the present disclosure.

Detailed Description

The present disclosure is further described with reference to the following drawings and examples.

It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the disclosure. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments according to the present disclosure. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.

The embodiments and features of the embodiments in the present disclosure may be combined with each other without conflict.

The first embodiment is as follows:

the embodiment aims to provide a panchromatic and multispectral image fusion method based on deep learning.

A panchromatic and multispectral image fusion method based on deep learning comprises the following steps:

acquiring a full-color image and a multispectral image to be fused;

For the sake of understanding, the following detailed description of the embodiments of the present disclosure is made with reference to the accompanying drawings:

the scheme of the disclosure provides a two-stage deep neural network model for a method for fusing a full-color image and a multispectral image of a satellite, wherein the first stage is a spatial structure enhancement network shared by multiple satellites based on a convolutional neural network, the second stage is a spectral information adjustment network of multiple branches, and the specific branch adjusts the result of the first stage into the style of a specific satellite. The spatial structure enhancement network uses images of various different styles as training data, the images with better spatial structure are trained through the convolutional neural network, the image style is further adjusted through the spectral information adjustment network, in order to make the purpose, technical scheme and advantages of the disclosure clearer, the technical scheme of the disclosure is further explained in detail below by combining with the attached drawings and implementation, and the specific steps are as follows.

The method is different from a general fusion method based on a deep convolutional network in that images of various different sensors can be used as training images, training data are expanded, a network structure needs to be designed for each type of image and a model needs to be trained in the traditional fusion network, images of different styles are trained by using a common network model, and then spectrum adjustment is carried out on the images by using a branch spectrum information adjustment network to obtain images of the needed styles.

The method is based on the deep learning network, the fusion network is divided into a space structure enhancing network and a branch spectrum information adjusting network, the space structure enhancing network trains the space structure adjusting network by using satellite data of various sensors, training samples are effectively expanded, images with good space structure information are obtained, and the color style of the images is not similar to that of one of the satellites. And then extracting the style of the remote sensing image of the specific sensor by using the spectrum information adjusting network, and performing spectrum adjustment on the image with better spatial structure information obtained in the first step to obtain the image with the specific style.

Specifically, the steps of the method of the present disclosure are explained from data input, model training to fusion result output:

step 1: inputting an image

(1a) Inputting a training image: multispectral images of low spatial resolution and panchromatic images of high spatial resolution;

(1b) the low-spatial-resolution multispectral image ms and the high-resolution panchromatic image pan used for training comprise corresponding real images;

(1c) performing four-time upsampling operation on the multispectral image with low spatial resolution through deconvolution to obtain an initial multispectral image ms1 which is the same as the size of the full-color image p and belongs to R256 4;

step 2: extracting effective characteristic diagram through a spatial structure enhancement network to perform fusion of full-color image and multispectral image

(2a) The full-color image p and the initial multispectral image ms1 are first stacked to obtain an image M e R256 x 4. In the stacking operation, the images are stitched in a third dimension.

(2b) M is the same as R256 and 4, and the output is ms2 is the same as R256 and 4; m e R256 4 is stacked with the output ms2 e R256 4 of the first convolution module as input to the second convolution module with the output ms3 e R256 4; m is left at R256 4 and the output ms2 is left at R256 4 of the first convolution module, the output ms3 is left at R256 4 of the second convolution module is stacked and used as input, the input is the third convolution module, and the output is ms4 is left at R256 4; m e R256 4 is stacked with the first convolution module output ms2 e R256 4, the second convolution module output ms3 e R256 4, the third convolution module output ms4 e R256 4 as inputs to the fourth convolution module with the output ms5 e R256 4. The full-color image p itself was stacked four times with the output pan2 e R256 e 4, highlighting the spatial structure information, and then added to the output ms5 e R256 e 4 of the fourth convolution module. The output result is HRMS1 ∈ R256 × 4.

And step 3: spectral information adjustment network extraction style feature

(3a) Establishing a spectrum information adjusting network: the fused HRMS1 for different kinds of satellites inputs different spectral tuning branches. The spectrum adjustment network comprises a plurality of branch networks, each branch network processes one kind of remote sensing image, and the HRMS1 inputs the corresponding branch processing.

(3b) For one of the branches, the input is HRMS1 ∈ R256 × 4. HRMS1 ∈ R256 ∈ 4 is first passed through a convolution layer of 32 3 × 3 filters, output H1 ∈ R256 ∈ R32, H1 ∈ R256 ∈ 32 is input into a global averaging pooling layer, global flattening pooling performs feature compression on two-dimensional information, each two-dimensional feature channel is changed into a real number, 32 channels correspond to 32 real numbers, the real number has global perception to some extent, output H2 ∈ R32, H7 ∈ R32 is input into two fully-connected layers, the first fully-connected layer inputs 32 nodes, outputs 16 nodes, comprises an activation function Rectified LinearUnit (ReLU), the second fully-connected layer inputs 16 nodes, outputs 32 nodes, comprises an activation function moid, the output results of two fully-connected layers are H3 ∈ R32, H3 ∈ R73742 is adjusted to H1, H6332 is multiplied by H63R 256 and MS 6332 is adjusted to generate H6332 points, h3 was adjusted channel by channel for H1. The dot product result is processed through one convolution layer (4 3 × 3 filters), and the output result is Mask ∈ R256 × 4.

(3c) Mask ∈ R256 × 4 and HRMS1 ∈ R256 × 4. Obtaining a final high-resolution multispectral image;

and 4, step 4: training network

The scheme of the disclosure trains the network by adopting a stochastic gradient descent algorithm, and the loss function is an L2 norm of the distance between the generated image and the reference image. The learning rate is set to 0.001. The iteration number is set to 8000 turns, the batch _ size is set to 8, and the trained training model is output after the iteration is finished.

And 5: image fusion

After the two-stage deep neural network training is completed, the tested multispectral image with low spatial resolution and panchromatic image with high spatial resolution are input into a spatial structure enhancement network to obtain an intermediate result C, then the intermediate result C is input into a spectral information adjustment network to obtain a D, and the D and the C are subjected to dot multiplication to obtain a final result, namely a fusion image.

The effect of the scheme of the present disclosure is illustrated by the following simulations:

1. simulation environment:

PyCharm Community Edition 2020.1x64，Window 10。

2. simulation content:

in the embodiment of the present disclosure, five satellite images are adopted: 216 pairs of fastside satellite images, the images including a captured green field and a light brown terrace. 248 pairs of world view satellite images, the image content is dense houses under the feet of the mountain, which is a small city under the feet of the mountain. 696 for the geoeye satellite image, the image content is a bald mountain area, which contains the town. 104 pairs of Ikonos satellite images, the content is mountains in Sichuan area. 416 GaoFen-2 satellite images, the content is urban area of the city. The multispectral image size of low spatial resolution is 64 × 64 × 4, the spatial resolution is 8.0m, the panchromatic image size of high spatial resolution is 256 × 256, the spatial resolution is 2.0m, and the reference image size is 256 × 256 × 4;

fig. 3(a) is a low spatial resolution multi-spectral image, 64 x 4 in size,

fig. 3(b) is a high spatial resolution full color image, 256 x 256 in size,

fig. 3(c) is a reference picture, 256 x 4 in size,

fig. 3(d) is a high spatial resolution multispectral image obtained after fusing fig. 3(a) and fig. 3(b) using the present disclosure, with a size of 256 × 256 × 4.

With reference to the drawings, it can be seen that the resolution information of fig. 3(d) is significantly improved compared to fig. 3(a), especially in the detailed parts of tree roads, houses and the like, the edges are clearer, and the spectral information of fig. 3(d) is richer compared to fig. 3(b), so that the disclosure can better merge fig. 3(a) and fig. 3 (b).

In order to verify the effect of the method, the existing method is respectively used for comparing with the method. The technology comprises the following steps: the BDSD transformation method, the AWLP transformation method, the Indusion method, the SVT algorithm, the VPLMC, the PNN and the PanNet fuse the images to be fused in the images (a) and (b) in the images (2) and perform objective index evaluation on the fusion result, wherein the evaluation indexes are as follows:

1) and the correlation coefficient CC represents the retention degree of the spectral information, the result is in the interval [0, 1], and the closer the correlation coefficient is to 1, the more similar the fusion result is to the reference image.

2) The RMSE root mean square error represents the square root of the ratio of the square of the deviation between the predicted value and the true value to the observation frequency n, and the smaller the numerical value, the better the fusion result.

3) The closer the global composite error index ERG is to 1, the better.

4) And the spectrum radian SAM represents the distortion degree of the spectrum, the spectrum of each pixel in the image is regarded as a high-dimensional vector, the similarity between the spectra is measured by calculating the included angle between the two vectors, and the closer to 0, the better the fusion result is.

5) And the global quality evaluation indexQ represents the overall similarity of the images in space and spectrum, the result range is in an interval [0, 1], and the larger the global quality evaluation index is, the more similar the fused image is to the reference image.

6) And the overall image quality index UIQI represents the closeness degree of the fused image and the reference image, and the closer to 1, the better the fusion result.

The fused results of the present disclosure and the prior art were evaluated from the objective evaluation indexes according to the above evaluation indexes, and the results are shown in table 1.

TABLE 1 Objective evaluation of fusion results of various methods

As can be seen from table 1, the above evaluation values are superior to the evaluation values of the prior art, and thus it can be seen that most of the objective evaluation indexes of the present disclosure are superior to the objective evaluation indexes of the prior art.

The fusion image obtained by the method disclosed by the invention is rich in spatial information and better reserves the spectral information of the multispectral image.

The subjective evaluation according to the simulation result of the simulation content is that the spatial resolution of the original multispectral image is improved by the fused image through visual analysis, and the training data amount is increased by using various images as training data.

Example two:

the embodiment aims to provide a panchromatic and multispectral image fusion system based on deep learning.

In further embodiments, there is also provided:

an electronic device comprising a memory and a processor, and computer instructions stored on the memory and executed on the processor, the computer instructions when executed by the processor performing the method of embodiment one. For brevity, no further description is provided herein.

It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general purpose processors, digital signal processors DSP, application specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and so on. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory may include both read-only memory and random access memory, and may provide instructions and data to the processor, and a portion of the memory may also include non-volatile random access memory. For example, the memory may also store device type information.

A computer readable storage medium storing computer instructions which, when executed by a processor, perform the method of embodiment one.

The method in the first embodiment may be directly implemented by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor. The software modules may be located in ram, flash, rom, prom, or eprom, registers, among other storage media as is well known in the art. The storage medium is located in a memory, and a processor reads information in the memory and completes the steps of the method in combination with hardware of the processor. To avoid repetition, it is not described in detail here.

Those of ordinary skill in the art will appreciate that the various illustrative elements, i.e., algorithm steps, described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

The panchromatic and multispectral image fusion method and system based on deep learning can be realized, and have wide application prospects.

The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure, and various modifications and changes may be made to the present disclosure by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present disclosure should be included in the protection scope of the present disclosure.

Although the present disclosure has been described with reference to specific embodiments, it should be understood that the scope of the present disclosure is not limited thereto, and those skilled in the art will appreciate that various modifications and changes can be made without departing from the spirit and scope of the present disclosure.

Claims

1. A panchromatic and multispectral image fusion method based on deep learning is characterized by comprising the following steps:

acquiring a full-color image and a multispectral image to be fused;

2. The panchromatic and multispectral image fusion method based on deep learning as claimed in claim 1, wherein the input of the spatial structure enhancement network is panchromatic image and initial multispectral image, the panchromatic image and the initial multispectral image are firstly stacked to obtain an image M, 4 convolution modules are adopted in the middle for extracting spatial information, the convolution modules are densely connected, M is input into the first convolution module, and the output is ms 2; m is stacked with the output ms2 of the first convolution module, and input into the second convolution module, and the output is ms 3; m is stacked with ms2 and ms3, input into a third convolution module and output as ms 4; m is stacked with ms2, ms3, ms4 as input to a fourth convolution module with an output of ms 5; the full color image itself was stacked four times, outputting pan 2; pan2 is then added to ms5, outputting HRMS 1.

3. The method for fusion of panchromatic and multispectral images based on deep learning of claim 1, wherein the spectral information adjustment network inputs the output of the spatial structure enhancement network into a specific branch for processing, each branch network comprising 32 convolutional layers of 3 x 3 filters, a global average pooling layer, and two fully connected layers, 4 convolutional layers of 3 x 3 filters; and the spectral information adjusting network adjusts each channel of the spatial structure enhancing network, and the output result is Mask.

4. The method as claimed in claim 1, wherein the spatial structure enhancement network outputs HRMS1 and the spectral information adjustment network outputs Mask, which are dot-multiplied to obtain the final high-resolution multispectral image HRMS.

5. The method of claim 1, wherein the panchromatic image and multispectral image to be fused are pre-processed, and the multispectral image with low spatial resolution is up-sampled four times to obtain an initial multispectral image with the same size as the panchromatic image.

6. A panchromatic and multispectral image fusion system based on deep learning is characterized by comprising the following components:

7. The panchromatic and multispectral image fusion system based on deep learning of claim 6, wherein the input of the spatial structure enhancement network is panchromatic image and initial multispectral image, the panchromatic image and the initial multispectral image are firstly stacked to obtain an image M, 4 convolution modules are adopted in the middle for extracting spatial information, the convolution modules are densely connected, M is input into the first convolution module, and the output is ms 2; m is stacked with the output ms2 of the first convolution module, and input into the second convolution module, and the output is ms 3; m is stacked with ms2 and ms3, input into a third convolution module and output as ms 4; m is stacked with ms2, ms3, ms4 as input to a fourth convolution module with an output of ms 5; the full color image itself was stacked four times, outputting pan 2; pan2 is then added to ms5, outputting HRMS 1.

8. The deep learning-based panchromatic and multispectral image fusion system of claim 6, wherein the spectral information adjustment network inputs the output of the spatial structure enhancement network into a specific branch for processing, each branch network comprising 32 convolutional layers of 3 x 3 filters, a global average pooling layer, and two fully-connected layers, 4 convolutional layers of 3 x 3 filters; and the spectral information adjusting network adjusts each channel of the spatial structure enhancing network, and the output result is Mask.

9. An electronic device comprising a memory, a processor, and a computer program stored and executed on the memory, wherein the processor implements a deep learning based panchromatic and multispectral image fusion method as claimed in any one of claims 1-4 when executing the program.

10. A non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a deep learning based panchromatic and multispectral image fusion method according to any one of claims 1-4.