CN113191325A - Image fusion method, system and application thereof - Google Patents

Image fusion method, system and application thereof Download PDF

Info

Publication number
CN113191325A
CN113191325A CN202110567685.8A CN202110567685A CN113191325A CN 113191325 A CN113191325 A CN 113191325A CN 202110567685 A CN202110567685 A CN 202110567685A CN 113191325 A CN113191325 A CN 113191325A
Authority
CN
China
Prior art keywords
image
information
spatial
extracting
multispectral
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110567685.8A
Other languages
Chinese (zh)
Other versions
CN113191325B (en
Inventor
钟锡武
钱静
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Institute of Advanced Technology of CAS
Original Assignee
Shenzhen Institute of Advanced Technology of CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Institute of Advanced Technology of CAS filed Critical Shenzhen Institute of Advanced Technology of CAS
Priority to CN202110567685.8A priority Critical patent/CN113191325B/en
Publication of CN113191325A publication Critical patent/CN113191325A/en
Application granted granted Critical
Publication of CN113191325B publication Critical patent/CN113191325B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G06V20/13—Satellite images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
    • G06F18/2135—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods based on approximation criteria, e.g. principal component analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G06V20/194—Terrestrial scenes using hyperspectral data, i.e. more or other wavelengths than RGB
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
    • Y02A40/00—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production
    • Y02A40/10—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production in agriculture

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Biomedical Technology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Molecular Biology (AREA)
  • Health & Medical Sciences (AREA)
  • Evolutionary Biology (AREA)
  • Multimedia (AREA)
  • Remote Sensing (AREA)
  • Astronomy & Astrophysics (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

The present application belongs to the field of image processing technology, and in particular, relates to an image fusion method, system and application thereof. The application provides an image fusion method, which comprises the steps of extracting first high-pass information of a multispectral image to obtain a first multispectral image, and extracting second high-pass information of a full-color image to obtain a first full-color image; extracting first spatial information of the first multispectral image and extracting second spatial information of the first panchromatic image; fusing the first spatial information and the second spatial information to obtain spatial features; and reconstructing the spatial features to obtain a high spatial resolution image, and simultaneously directly transmitting the multispectral image and the panchromatic image to the high resolution image after the spatial features are reconstructed, thereby improving the spectral resolution of the fused image.

Description

Image fusion method, system and application thereof
Technical Field
The present application belongs to the field of image processing technology, and in particular, relates to an image fusion method, system and application thereof.
Background
With current remote sensing system designs, both spectral and spatial resolution often cannot be maintained at high levels simultaneously. The images acquired by different sensors differ in geometrical characteristics, spectral resolution and spatial resolution. Some sensors acquire rich spectral information of a scene, but lack sufficient spatial information, such as multispectral images (MS). On the other hand, some sensors are good at capturing spatial information, but cannot capture reliable spectral information, such as a panchromatic image (PAN). High spatial resolution images provide subtle geometric features, while high spectral resolution images provide rich spectral information that can be used to identify and analyze targets. In order to take full advantage of the information provided by multispectral images and panchromatic images, it is common to fuse a low-resolution multispectral image with a high-resolution panchromatic image of the same scene to produce an image with a more detailed spatial and spectral structure, i.e., pancharapening.
Remote sensing image pancharapening is developed to date, various technologies and algorithms exist, and the pancharapening is often used as a basis for other applications (such as semantic segmentation, classification and the like of remote sensing images) of the remote sensing images, and is particularly important in remote sensing image processing. The current widely used technical methods include: principal component analysis based, wavelet transform based, convolutional neural network based, and generative countermeasure network based methods. Although there are many methods that have been developed, none of these methods are optimal because they tend to make inefficient use of the spatial and spectral information of the MS image and the PAN image. In the existing fusion algorithm, it is often considered that spatial information exists in the PAN image and spectral information exists in the MS image, but this often ignores the spatial information existing in the MS image and the spectral information possibly existing in the PAN image, which results in the loss of spectral and spatial information to different degrees. Meanwhile, the existing deep learning method uses a simple stack of feature maps in feature fusion, and such an operation merely provides a fixed linear aggregation of feature maps, and it is not known at all whether such a combination is suitable for a specific object.
Disclosure of Invention
1. Technical problem to be solved
Based on the existing deep learning method, a simple feature map stack is used in feature fusion, such an operation only provides a fixed linear aggregation of feature maps, and the problem of whether the group is suitable for a specific object is completely unknown, meanwhile, the existing method often ignores spatial information existing in an MS image and spectral information possibly existing in a PAN image, and causes loss of fused image spectrum and spatial information to a certain extent, and the application provides an image fusion method, a system and application thereof.
2. Technical scheme
In order to achieve the above object, the present application provides an image fusion method, including the steps of: step 1: extracting first high-pass information of the multispectral image to obtain a first multispectral image, and extracting second high-pass information of the full-color image to obtain a first full-color image; step 2: extracting first spatial information of the first multispectral image and extracting second spatial information of the first panchromatic image; and step 3: fusing the first spatial information and the second spatial information to obtain spatial features; and 4, step 4: and reconstructing the spatial features to obtain a high spatial resolution image, and simultaneously directly transmitting the multispectral image and the panchromatic image to the high resolution image after the spatial features are reconstructed, thereby improving the spectral resolution of the fused image.
Another embodiment provided by the present application is: the extracting of the first high-pass information of the multispectral image comprises up-sampling the input multispectral image to make the multispectral image and the panchromatic image have the same size, and then extracting the first high-pass information of the up-sampled multispectral image by adopting high-pass filtering; the extracting the second high-pass information of the full-color image includes extracting the second high-pass information of the full-color image using high-pass filtering.
Another embodiment provided by the present application is: the first high-pass information extracting first low-pass information of the up-sampled multispectral image by using mean filtering, and then subtracting the first low-pass information from the up-sampled multispectral image; the second high-pass information extracts second low-pass information of the panchromatic image by employing mean filtering, and then subtracts the second low-pass information from the up-sampled multispectral image.
Another embodiment provided by the present application is: the first spatial information is extracted by adopting a convolutional neural network, and the second spatial information is extracted by adopting the convolutional neural network.
Another embodiment provided by the present application is: the reconstructing the spatial feature comprises reconstructing the spatial feature by adopting a U-Net network; and transmitting the up-sampling multispectral image and the panchromatic image to a spatial reconstruction image through spectral mapping by adopting long jump connection to obtain an image with high spatial resolution and high spectral resolution.
The application also provides an image fusion system, which comprises a feature extraction module, an attention feature fusion module and an image reconstruction module which are sequentially connected; the characteristic extraction module is used for acquiring high-pass information of an original image and then extracting image characteristics to obtain a characteristic diagram; the attention feature fusion module is used for fusing the feature map; and the image reconstruction module is used for reconstructing a high-spatial resolution image from the fused image.
Another embodiment provided by the present application is: the image reconstruction module comprises a long jump connection submodule, and the long jump connection submodule is used for transmitting the image spectrum information to the space for reconstruction and then fusing the image spectrum information with the image with the reconstructed space information.
Another embodiment provided by the present application is: said system adopts
Figure BDA0003081355730000022
Training as a loss function, said
Figure BDA0003081355730000023
The loss function is:
Figure BDA0003081355730000021
where N is the number of training samples in the small batch,
Figure BDA0003081355730000031
and
Figure BDA0003081355730000032
is a PAN image and a low-resolution MS image, Y(i)Is the corresponding high resolution MS image and θ is a parameter of the Attention _ FPNet network.
Another embodiment provided by the present application is: the attention feature fusion module is:
Figure BDA0003081355730000033
wherein X1, X2 represent two input features, Z ∈ RC×H×WThe fused features are represented as a result of the fusion,
Figure BDA0003081355730000034
representing the weights derived by the channel attention module M, consisting of real numbers between 0 and 1,
Figure BDA0003081355730000035
corresponding to the dashed line in fig. 2, consists of real numbers between 0 and 1,
Figure BDA0003081355730000036
it means that the broadcast addition method is performed,
Figure BDA0003081355730000037
representing element-by-element multiplication.
The application also provides an application of the image fusion method, and the image fusion method is applied to the remote sensing image super-resolution reconstruction problem.
3. Advantageous effects
Compared with the prior art, the image fusion method, the image fusion system and the application thereof have the beneficial effects that:
the image fusion method provided by the application adopts a double-branch fusion network based on Attention feature fusion to solve the pancharapening problem, and is named as Attention _ FPNet.
The image fusion method provided by the application reconstructs the spatial information of the image in the high-pass filtering domain, and more fully considers the spatial information in the multispectral and panchromatic images. Meanwhile, the input panchromatic image and the up-sampled multispectral image are directly transmitted to the image after the space is reconstructed through a long jump connection, the spectral information of the panchromatic image and the multispectral image is considered, the spectral resolution of the fused image is improved, and the possible loss of spatial information caused by the deepening of a network is supplemented. Meanwhile, the attention feature fusion method is used, the relation among different feature graphs is fully considered, and the fusion quality is improved.
According to the image fusion method, the convolution neural network is utilized to depend on the high-efficiency fusion performance of the feature extraction capability and the attention mechanism of the powerful convolution network with little frequency spectrum distortion, the double-branch fusion network based on attention feature fusion is used, in order to more fully utilize spatial information in an MS image and a PAN image, the spatial information of the MS and the PAN after high-pass filtering is fused, then the spatial information of the fused image is reconstructed, meanwhile, an attention feature fusion module is used for replacing a common channel stacking method, the relation among different channels is considered, and therefore the quality of feature fusion is improved.
According to the image fusion method, in order to obtain a fusion image with higher spectral resolution, the spectral information in the MS image and the spectral information in the PAN image are considered at the same time, a long jump connection is used, the input PAN image and the MS image after up-sampling are directly transmitted to the fusion image after spatial reconstruction, and therefore the loss of the spectral information is reduced.
The image fusion method provided by the application inevitably causes the loss of the spatial information along with the deepening of the network depth, and the long jump connection plays a role in supplementing the spatial information. By the method and the device, the multispectral image with higher resolution can be obtained.
According to the image fusion method, the attention feature fusion method is used for replacing a simple channel stacking method used in the past to fuse the feature map, the weight among different channels is considered, and the quality of feature fusion is improved.
According to the image fusion method, the spatial resolution of the image is reconstructed in a high-pass filtering domain instead of an image domain, the spatial information in the MS image and the spatial information in the PAN image are more fully considered, the spatial resolution of the fused image can be improved, and meanwhile, the spectral information of the MS image and the spectral information of the PAN image are more fully utilized by using a long jump connection.
The image fusion system provided by the application uses
Figure BDA0003081355730000041
Loss function rather than widespread use
Figure BDA0003081355730000042
A loss function to optimize the network.
Drawings
FIG. 1 is a schematic diagram of an Attention _ FPNet of the present application;
FIG. 2 is a schematic view of an attention feature fusion module of the present application;
FIG. 3 is a detailed structural diagram of Attention _ FPNet of the present application;
fig. 4 is a schematic view of a first effect of the present application;
FIG. 5 is a schematic diagram of a second effect of the present application;
fig. 6 is a schematic diagram of a third effect of the present application.
Detailed Description
Hereinafter, specific embodiments of the present application will be described in detail with reference to the accompanying drawings, and it will be apparent to those skilled in the art from this detailed description that the present application can be practiced. Features from different embodiments may be combined to yield new embodiments, or certain features may be substituted for certain embodiments to yield yet further preferred embodiments, without departing from the principles of the present application.
The remote sensing image panchromatic sharpening method is a pancharapening method, the multispectral remote sensing image is enhanced by utilizing panchromatic wave bands, the observation process of the panchromatic wave bands and the multiband image is simulated by combining the characteristics of sensors, and the expected value of the multispectral image with high resolution is estimated by utilizing priori knowledge. The method enables the panchromatic waveband data and the multi-spectral waveband data to be automatically aligned, successfully reserves the spectral information, increases the spatial resolution and enriches the ground information.
In recent years, many different pansharpening methods have been proposed. These methods can be broadly classified into the following four categories: component replacement (CS), multiresolution analysis (MRA), hybrid approaches (combining CS and MAR), model-based approaches, and deep learning-based approaches.
(1) The component replacement method comprises the following steps: the CS method transforms the MS image to another color space based on a reversible transformation, which separates spatial information and spectral information of the MS image, and replaces the spatial information of the MS image separated with spatial information of the PAN image after histogram matching. And finally, converting the MS image after replacing the spatial information into the original color space by using inverse conversion. IHS (Intensity-Hue-preservation), Principal Component Analysis (PCA), Brovey Transform (BT) and Gram-Schmidt (GS) -based transformations are the best known CS methods.
(2) Multi-resolution analysis: the MRA method decomposes each original data into a series of images with different resolutions by using a plurality of multidimensional methods such as laplacian pyramid decomposition, wavelet transformation, contourlet transformation, curvelet transformation, and the like, performs fusion on the images with different resolutions, and finally performs inverse transformation to obtain a fused image.
(3) The mixing method comprises the following steps: hybrid methods combine the advantages of CS and MRA methods
(4) Model-based methods: a model-based method adopts a reverse thinking mode, firstly assumes a degradation process from a high-resolution MS image to a low-resolution MS and a high-resolution PAN, describes the degradation process by an optimal model, and carries out recovery from the degradation process in the reverse direction.
(5) The method based on deep learning comprises the following steps: the deep learning-based method can obtain ideal fusion performance by relying on the feature extraction capability of a strong convolution network with little frequency spectrum distortion. In 2016, Giuseppe and the like are improved on the basis of a single image super-resolution reconstruction algorithm SRCNN, and a first three-layer network structure based on deep learning is provided for solving the pancharapening problem. Firstly, an input MS image and an input PAN image are stacked on a channel dimension, and then the MS image and the PAN image are sent into a three-layer network structure to reconstruct the images, so that a multispectral image with high spatial resolution is generated. The idea of the method is applied later, and a plurality of pancharapening network structures based on deep learning are generated to help the subsequent development.
Referring to fig. 1 to 6, the present application provides an image fusion method, including the steps of: step 1: extracting first high-pass information of the multispectral image to obtain a first multispectral image, and extracting second high-pass information of the full-color image to obtain a first full-color image; step 2: extracting first spatial information of the first multispectral image and extracting second spatial information of the first panchromatic image; and step 3: fusing the first spatial information and the second spatial information to obtain spatial features; and 4, step 4: and reconstructing the spatial features to obtain a high spatial resolution image, and simultaneously directly transmitting the multispectral image and the panchromatic image to the high resolution image after the spatial features are reconstructed, thereby improving the spectral resolution of the fused image.
Further, the extracting the first high-pass information of the multispectral image comprises up-sampling the input multispectral image to make the multispectral image and the panchromatic image have the same size, and then extracting the first high-pass information of the up-sampled multispectral image by adopting high-pass filtering; the extracting the second high-pass information of the full-color image includes extracting the second high-pass information of the full-color image using high-pass filtering.
Further, the first high-pass information extracts first low-pass information of the up-sampled multispectral image by using mean filtering, and then subtracts the first low-pass information from the up-sampled multispectral image; the second high-pass information extracts second low-pass information of the panchromatic image by employing mean filtering, and then subtracts the second low-pass information from the up-sampled multispectral image.
Further, the first spatial information is extracted by using a convolutional neural network, and the second spatial information is extracted by using a convolutional neural network.
Further, the reconstructing the spatial feature comprises reconstructing the spatial feature by using a U-Net network; and transmitting the up-sampling multispectral image and the panchromatic image to a spatial reconstruction image through spectral mapping by adopting long jump connection to obtain an image with high spatial resolution and high spectral resolution.
The application also provides an image fusion system, which comprises a feature extraction module, an attention feature fusion module and an image reconstruction module which are sequentially connected; the characteristic extraction module is used for acquiring high-pass information of an original image and then extracting image characteristics to obtain a characteristic diagram; the attention feature fusion module is used for fusing the feature map; and the image reconstruction module is used for reconstructing a high-spatial resolution image from the fused image.
Feature extraction module
The MS image is first up-sampled to have the same image size as the PAN. In order to obtain the high-pass information of the image, the application subtracts the low-pass information found by using the averaging filter from the original image, thereby obtaining the high-pass information of the original image. Thereafter, features are extracted from the MS image and the PAN image, respectively, from which the high-pass filtering has been extracted, using two sub-networks. The two sub-networks have similar structures and different weights, one sub-network takes the image of 4 wave bands as input, and the other sub-network takes the image of a single wave band as input. Each sub-network contains three successive convolutional layers, each followed by a modified Linear Unit (ReLU).
Attention feature fusion module
After passing through the feature extraction module, two feature maps which respectively and definitely represent the spatial information of the MS image and the PAN image are obtained. In order to fully utilize the spatial information of the MS image and the PAN image, the extracted feature maps must be fused. However, in the conventional deep learning method, when feature maps are fused, only two feature maps are directly stacked, only one fixed linear aggregation of feature maps can be provided, the relationship existing between different feature maps is not considered, and it is completely unknown whether such a combination is suitable for a specific object. To this end, the present application replaces the channel stacking method used in the prior art method with an Attention Feature Fusion (AFF) [14], which is structured as shown in FIG. 2. The AFF can be expressed as:
Figure BDA0003081355730000061
wherein X1, X2 represent two input features, Z ∈ RC×H×WThe fused features are represented as a result of the fusion,
Figure BDA0003081355730000062
representing the weights derived by the channel attention module M, which consists of real numbers between 0 and 1,
Figure BDA0003081355730000063
corresponding to the dashed line in fig. 2, is also made up of real numbers between 0 and 1.
Figure BDA0003081355730000064
It means that the broadcast addition method is performed,
Figure BDA0003081355730000065
representing element-by-element multiplication.
Image reconstruction module
By the implementation of the two modules, the fusion of the spatial information of the MS image and the PAN image is completed, and an image with high spatial resolution needs to be reconstructed from the fused image. The present application first downsamples the image. The present application does not use the maximum pooling and average pooling used by most convolutional neural networks to obtain features with scale and rotation invariance, because the detail information of the image is very important in the pancharapening fusion, and therefore, in the whole network, a convolution kernel with the step size of 2 is used for downsampling, rather than a simple pooling operation. After two times of downsampling, feature maps of two different scales are obtained, and the feature maps only account for 1/2 × 1/2 and 1/4 × 1/4 of the input feature proportion respectively. Then, two deconvolution are used for up-sampling, and two characteristic graphs with the scales of 1/2 × 1/2 and 1 × 1 of the input characteristics are generated step by step.
Since the features extracted by the convolutional neural network map the semantics and abstract information of the image in the deep convolutional layer, it is difficult to recover the detail texture of the image from the semantics and abstract information of the image. To restore realistic details, inspired by U-Net [39], the generated feature map is copied to the position after the first up-sampling before the second down-sampling and connected with the corresponding feature map to inject the detail information lost in the down-sampling process. The last layer outputs the required high resolution MS image. The detailed structure of the deep learning model used in the present application is shown in fig. 3.
The pancharapening task is to obtain a multispectral image with high spatial resolution and high spectral resolution, and the conventional method usually utilizes a certain feature extraction method to extract spectral information of an MS image, but such operation can cause the loss of the spectral information in the MS image and ignore spectral information possibly existing in a PAN image, so that the application uses a long jump connection to directly transmit the input MS and PAN images to a space for reconstruction and then fuse the MS and PAN images with the image with the reconstructed spatial information.
Furthermore, the image reconstruction module comprises a long jump connection submodule, and the long jump connection submodule is used for transmitting the image spectrum information to the space reconstruction module and then fusing the image spectrum information with the image with the reconstructed space information.
Further, the system employs
Figure BDA0003081355730000074
Training as a loss function, said
Figure BDA0003081355730000075
The loss function is:
Figure BDA0003081355730000071
where N is the number of training samples in the small batch,
Figure BDA0003081355730000072
and
Figure BDA0003081355730000073
is a PAN image and a low-resolution MS image, Y(i)Is the corresponding high resolution MS image and θ is a parameter of the Attention _ FPNet network.
Loss function
In addition to the network structure, the loss function is another important factor affecting the quality of the reconstructed image. Use of previous image reconstruction tasks
Figure BDA0003081355730000076
The norm is taken as a loss function, but the generated image has a fuzzy phenomenon. Therefore, this application uses
Figure BDA0003081355730000077
The network of the present application is trained as a loss function.
Further, the attention feature fusion module is:
Figure BDA0003081355730000081
wherein X1, X2 represent two input features, Z ∈ RC×H×WThe fused features are represented as a result of the fusion,
Figure BDA0003081355730000082
representing the weights derived by the channel attention module M, consisting of real numbers between 0 and 1,
Figure BDA0003081355730000083
corresponding to the dashed line in fig. 2, consists of real numbers between 0 and 1,
Figure BDA0003081355730000084
it means that the broadcast addition method is performed,
Figure BDA0003081355730000085
representing element-by-element multiplication.
The application also provides an application of the image fusion method, and the image fusion method is applied to the remote sensing image pancharapening problem.
Precision inspection and evaluation
The present application compares the method used with several widely used techniques, including: PCA, IHS, wavelet, MTF _ GLP _ HPM, GSA, CNMF, PNN, PanNet, ResTFNet.
Tables 1-3 show quantitative indices on the three satellite datasets Pleiades, SPOT-6 and Gaofen-2. Fig. 4-6 show qualitative results over three satellite datasets. From tables 1-3, it can be seen that the Attention _ FPNet achieves the best performance in most indicators. Particularly on the Pleiades dataset, all the indexes achieved the best performance. The SPOT-6 and Gaofen-2 data sets achieved the best performance for all the other criteria, except that QNR achieved the 4 th and 2 nd performance, respectively.
As can be seen from the present application in fig. 4, all methods except the Wavelet and PNN algorithms produce visually pleasing pansharpening images. Images generated by the Wavelet method have serious blurring and artifact phenomena. The PNN method also exhibits a blurring effect. The IHS method, while having good visual quality, also has significant spectral distortion. In fig. 5, except for Wavelet and CNMF, other methods can achieve better visual effect. Wavelet still has serious blurring and artifact phenomena, while CNMF has some serious information loss of spatial details. On the Pleiades dataset, as in fig. 6, the Wavelet and PNN methods again appear blurred. The Attention _ FPNet algorithm of the present application does better in terms of spectral preservation and also produces richer spatial details.
Table 1 quantitative evaluation on SPOT-6 dataset. And (4) sorting according to results, wherein the first four names are respectively marked as (1), (2), (3) and (4).
Figure BDA0003081355730000086
Table 2 quantitative evaluation on Pleiades dataset. And (4) sorting according to results, wherein the first four names are respectively marked as (1), (2), (3) and (4).
Figure BDA0003081355730000087
Figure BDA0003081355730000091
Table 3 quantitative evaluation on Gaofen-2 dataset. And (4) sorting according to results, wherein the first four names are respectively marked as (1), (2), (3) and (4).
Figure BDA0003081355730000092
And (3) accuracy inspection conclusion: based on the above experimental analysis, it is found that the method of the present application is superior to other commonly used methods in both spectral and spatial indicators and visual effects on three satellite datasets. This shows that the method of the present application is effective for solving the remote sensing image pancharapening problem.
Experiments are carried out on three satellite data sets of Pleiades, SPOT-6 and Gaofen-2, and the experimental results show that the Attention _ FPNet used in the method is superior to other existing common technologies in spectral and spatial information reconstruction. Experiments prove that the double-branch fusion network based on attention feature fusion is feasible for the pancharapening task.
Eight widely used indicators were used to quantitatively evaluate the performance of the proposed method and the comparative method.
The peak signal-to-noise ratio (PSNR) reflects the quality of the fused reconstructed image by calculating the ratio of the maximum peak value of the reconstructed image to the mean square error of the two images on the basis of the Mean Square Error (MSE). PSNR is defined as:
Figure BDA0003081355730000093
wherein, MAXIIs the maximum value representing the color of the image point. The higher the PSNR value between two images, the less the reconstructed image is distorted with respect to the high resolution image. MSE is defined as:
Figure BDA0003081355730000094
where I and K are two images of size m n, one of which is a noise approximation of the other.
Structural Similarity (SSIM) measures the overall fusion quality by calculating the mean, variance and covariance of the fused image and the reference image. The SSIM measurement consists of three types of contrast modules, brightness, contrast, and structure. Suppose that two images X, Y of size M N are given, where the mean, variance and covariance of X and Y are ux、uy、
Figure BDA0003081355730000101
Figure BDA0003081355730000102
δxyAnd (4) showing. The comparison functions defining brightness, contrast and structure are respectively
Figure BDA0003081355730000103
Figure BDA0003081355730000104
Figure BDA0003081355730000105
The three component factors are combined to form an SSIM index which is defined as
SSIM(X,Y)＝[I(X,y)]a[c(X,y)]p[s(x.Y)]γ #(6)
The closer the SSIM value is to 1, the higher the similarity between the two images.
The global integrated error index (ERGAS) mainly evaluates the spectral quality of all fused bands within the spectral range, taking into account the overall situation of spectral variations. It is defined as
Figure BDA0003081355730000106
Wherein h is the resolution of the high-resolution image, l is the resolution of the low-resolution image, N is the number of wavebands, Bi is the multispectral image, and Mi is the average of the radiance values of the multispectral image. The smaller its value, the better the spectral quality of the fused image in the spectral range.
Spectral Angle Mapping (SAM) evaluates spectral quality by calculating the angle between the corresponding pixel of the fused image and the reference image. It is defined as
Figure BDA0003081355730000107
Wherein, Ia，JaIs the pixel vector of the fused image and the reference image at the distance point a. For an ideal fused image, the value of the SAM should be 0.
The Spatial Correlation Coefficient (SCC) is to estimate the similarity of spatial details of the fused image and the reference image, extract high frequency information of the reference image using a high pass filter, and calculate a Correlation Coefficient (CC) between the high frequency information [48 ].
Using a high-laplacian filter herein
Figure BDA0003081355730000111
A high frequency is obtained. sCC, the higher the number of points, the more spatial information that indicates the PAN image is injected during the fusion process. sCC between the fused image and the reference image is calculated. The final sCC is averaged over all bands of the MS image.
The correlation coefficient is calculated as
Figure BDA0003081355730000112
Where X is the fused image, Y is the reference image, w and h are the width and height of the image, and μ represents the average of the images.
Index Q combines three factors to calculate image distortion, correlation loss, brightness distortion and contrast distortion. It is defined as
Figure BDA0003081355730000113
Wherein Z is1And Z2Representing the b-th band of the fused image and the reference image. When Q is 1, it indicates that the best fidelity is available for reference.
QNR is a non-reference image quality evaluation method. It is composed of the spectral distortion index DλAnd spatial distortion index DSAnd (4) forming. Wherein, L wave bands are represented as ILRMSThe generated HRMS picture is IHRMSOnly one band is IPANOf the PAN image of (1), which degenerates the corresponding ILPANImage, then
Figure BDA0003081355730000114
Figure BDA0003081355730000115
Figure BDA0003081355730000116
The ideal value of QNR is 1, indicating better quality of the fused image.
Because the existing deep learning method is only simple channel stacking and does not know whether the combination is suitable for a specific object at all when feature fusion is carried out, the attention feature fusion method is used, the relation among different feature graphs is fully considered, and the fusion quality is improved.
Although the present application has been described above with reference to specific embodiments, those skilled in the art will recognize that many changes may be made in the configuration and details of the present application within the principles and scope of the present application. The scope of protection of the application is determined by the appended claims, and all changes that come within the meaning and range of equivalency of the technical features are intended to be embraced therein.

Claims (10)

1. An image fusion method, characterized by: the method comprises the following steps:
step 1: extracting first high-pass information of the multispectral image to obtain a first multispectral image, and extracting second high-pass information of the full-color image to obtain a first full-color image;
step 2: extracting first spatial information of the first multispectral image and extracting second spatial information of the first panchromatic image;
and step 3: fusing the first spatial information and the second spatial information to obtain spatial features;
and 4, step 4: and reconstructing the spatial features to obtain a high spatial resolution image, and simultaneously directly transmitting the multispectral image and the panchromatic image to the high resolution image after the spatial features are reconstructed, thereby improving the spectral resolution of the fused image.
2. The image fusion method of claim 1, characterized in that: the extracting of the first high-pass information of the multispectral image comprises up-sampling the input multispectral image to make the multispectral image and the panchromatic image have the same size, and then extracting the first high-pass information of the up-sampled multispectral image by adopting high-pass filtering; the extracting the second high-pass information of the full-color image includes extracting the second high-pass information of the full-color image using high-pass filtering.
3. The image fusion method of claim 2, characterized in that: the first high-pass information extracting first low-pass information of the up-sampled multispectral image by using mean filtering, and then subtracting the first low-pass information from the up-sampled multispectral image; the second high-pass information extracts second low-pass information of the panchromatic image by employing mean filtering, and then subtracts the second low-pass information from the up-sampled multispectral image.
4. The image fusion method of claim 1, characterized in that: the first spatial information is extracted by adopting a convolutional neural network, and the second spatial information is extracted by adopting the convolutional neural network.
5. The image fusion method of claim 1, characterized in that: the reconstructing the spatial feature comprises reconstructing the spatial feature by adopting a U-Net network; and transmitting the up-sampling multispectral image and the panchromatic image to a spatial reconstruction image through spectral mapping by adopting long jump connection to obtain an image with high spatial resolution and high spectral resolution.
6. An image fusion system, characterized by: the system comprises a feature extraction module, an attention feature fusion module and an image reconstruction module which are sequentially connected;
the characteristic extraction module is used for acquiring high-pass information of an original image and then extracting image characteristics to obtain a characteristic diagram;
the attention feature fusion module is used for fusing the feature map;
and the image reconstruction module is used for reconstructing a high-spatial resolution image from the fused image.
7. The image fusion system of claim 6, wherein: the image reconstruction module comprises a long jump connection submodule, and the long jump connection submodule is used for transmitting the image spectrum information to the space for reconstruction and then fusing the image spectrum information with the image with the reconstructed space information.
8. The image fusion system of claim 6, wherein: the system is trained with l1 as the loss function, the l1 loss function being:
Figure FDA0003081355720000021
where N is the number of training samples in the small batch,
Figure FDA0003081355720000022
and
Figure FDA0003081355720000023
is a PAN image and a low-resolution MS image, Y(i)Is the corresponding high resolution MS image and θ is a parameter of the Attention _ FPNet network.
9. The image fusion system of claim 6, wherein: the attention feature fusion module is:
Figure FDA0003081355720000024
wherein X1, X2 represent two input features, Z ∈ RC×H×WThe fused features are represented as a result of the fusion,
Figure FDA0003081355720000025
representing the weights derived by the channel attention module M, consisting of real numbers between 0 and 1,
Figure FDA0003081355720000026
corresponding to the dashed line in fig. 2, consists of real numbers between 0 and 1,
Figure FDA0003081355720000027
it means that the broadcast addition method is performed,
Figure FDA0003081355720000028
representing element-by-element multiplication.
10. An application of an image fusion method is characterized in that: the image fusion method of any one of claims 1-5 is applied to the remote sensing image pancharapening problem.
CN202110567685.8A 2021-05-24 2021-05-24 Image fusion method, system and application thereof Active CN113191325B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110567685.8A CN113191325B (en) 2021-05-24 2021-05-24 Image fusion method, system and application thereof

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110567685.8A CN113191325B (en) 2021-05-24 2021-05-24 Image fusion method, system and application thereof

Publications (2)

Publication Number Publication Date
CN113191325A true CN113191325A (en) 2021-07-30
CN113191325B CN113191325B (en) 2023-12-12

Family

ID=76985682

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110567685.8A Active CN113191325B (en) 2021-05-24 2021-05-24 Image fusion method, system and application thereof

Country Status (1)

Country Link
CN (1) CN113191325B (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113537247A (en) * 2021-08-13 2021-10-22 重庆大学 Data enhancement method for converter transformer vibration signal
CN114429424A (en) * 2022-04-01 2022-05-03 中国石油大学(华东) Remote sensing image super-resolution reconstruction method applicable to uncertain degradation mode
CN114511470A (en) * 2022-04-06 2022-05-17 中国科学院深圳先进技术研究院 Attention mechanism-based double-branch panchromatic sharpening method

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106508048B (en) * 2011-12-05 2014-08-27 中国科学院自动化研究所 A kind of similar scale image interfusion method based on multiple dimensioned primitive form
CN109886870A (en) * 2018-12-29 2019-06-14 西北大学 Remote sensing image fusion method based on binary channels neural network
US20190287216A1 (en) * 2018-03-19 2019-09-19 Mitsubishi Electric Research Laboratories, Inc. Systems and Methods for Multi-Spectral Image Super-Resolution
CN111539900A (en) * 2020-04-24 2020-08-14 河南大学 IHS remote sensing image fusion method based on guided filtering
KR102160687B1 (en) * 2019-05-21 2020-09-29 인천대학교 산학협력단 Aviation image fusion method
CN112465733A (en) * 2020-08-31 2021-03-09 长沙理工大学 Remote sensing image fusion method, device, medium and equipment based on semi-supervised learning
CN112488978A (en) * 2021-02-05 2021-03-12 湖南大学 Multi-spectral image fusion imaging method and system based on fuzzy kernel estimation
CN112819737A (en) * 2021-01-13 2021-05-18 西北大学 Remote sensing image fusion method of multi-scale attention depth convolution network based on 3D convolution
CN114511470A (en) * 2022-04-06 2022-05-17 中国科学院深圳先进技术研究院 Attention mechanism-based double-branch panchromatic sharpening method

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106508048B (en) * 2011-12-05 2014-08-27 中国科学院自动化研究所 A kind of similar scale image interfusion method based on multiple dimensioned primitive form
US20190287216A1 (en) * 2018-03-19 2019-09-19 Mitsubishi Electric Research Laboratories, Inc. Systems and Methods for Multi-Spectral Image Super-Resolution
CN109886870A (en) * 2018-12-29 2019-06-14 西北大学 Remote sensing image fusion method based on binary channels neural network
KR102160687B1 (en) * 2019-05-21 2020-09-29 인천대학교 산학협력단 Aviation image fusion method
CN111539900A (en) * 2020-04-24 2020-08-14 河南大学 IHS remote sensing image fusion method based on guided filtering
CN112465733A (en) * 2020-08-31 2021-03-09 长沙理工大学 Remote sensing image fusion method, device, medium and equipment based on semi-supervised learning
CN112819737A (en) * 2021-01-13 2021-05-18 西北大学 Remote sensing image fusion method of multi-scale attention depth convolution network based on 3D convolution
CN112488978A (en) * 2021-02-05 2021-03-12 湖南大学 Multi-spectral image fusion imaging method and system based on fuzzy kernel estimation
CN114511470A (en) * 2022-04-06 2022-05-17 中国科学院深圳先进技术研究院 Attention mechanism-based double-branch panchromatic sharpening method

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
JUNFENG YANG 等: "PanNet: A deep network architecture for pan-sharpening", 《THE IEEE INTERNATIONAL CONFERENCE ON COMPUTER VISION》, pages 5449 - 5457 *
XIWU ZHONG 等: "Attention_FPNet: Two-Branch Remote Sensing Image Pansharpening Network Based on Attention Feature Fusion", 《IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING》, vol. 14, pages 11879 - 11891, XP011891498, DOI: 10.1109/JSTARS.2021.3126645 *
朱超: "多光谱与全色遥感图像融合算法的研究", 《中国优秀硕士论文学位论文全文数据库 信息科技辑库》, no. 11, pages 028 - 158 *
肖亮;刘鹏飞;李恒;: "多源空――谱遥感图像融合方法进展与挑战", 中国图象图形学报, no. 05, pages 5 - 17 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113537247A (en) * 2021-08-13 2021-10-22 重庆大学 Data enhancement method for converter transformer vibration signal
CN114429424A (en) * 2022-04-01 2022-05-03 中国石油大学(华东) Remote sensing image super-resolution reconstruction method applicable to uncertain degradation mode
CN114511470A (en) * 2022-04-06 2022-05-17 中国科学院深圳先进技术研究院 Attention mechanism-based double-branch panchromatic sharpening method
CN114511470B (en) * 2022-04-06 2022-07-08 中国科学院深圳先进技术研究院 Attention mechanism-based double-branch panchromatic sharpening method

Also Published As

Publication number Publication date
CN113191325B (en) 2023-12-12

Similar Documents

Publication Publication Date Title
Blum et al. An Overview of lmage Fusion
CN110533620B (en) Hyperspectral and full-color image fusion method based on AAE extraction spatial features
CN112507997B (en) Face super-resolution system based on multi-scale convolution and receptive field feature fusion
Xie et al. Hyperspectral image super-resolution using deep feature matrix factorization
CN113191325B (en) Image fusion method, system and application thereof
CN111127374B (en) Pan-sharing method based on multi-scale dense network
Zhang et al. One-two-one networks for compression artifacts reduction in remote sensing
CN110415199B (en) Multispectral remote sensing image fusion method and device based on residual learning
CN110544212B (en) Convolutional neural network hyperspectral image sharpening method based on hierarchical feature fusion
Patel et al. Super-resolution of hyperspectral images: Use of optimum wavelet filter coefficients and sparsity regularization
CN111696043A (en) Hyperspectral image super-resolution reconstruction algorithm of three-dimensional FSRCNN
CN112785480B (en) Image splicing tampering detection method based on frequency domain transformation and residual error feedback module
CN113763299A (en) Panchromatic and multispectral image fusion method and device and application thereof
CN114266957A (en) Hyperspectral image super-resolution restoration method based on multi-degradation mode data augmentation
Fan et al. Global sensing and measurements reuse for image compressed sensing
CN108335265B (en) Rapid image super-resolution reconstruction method and device based on sample learning
CN114511470B (en) Attention mechanism-based double-branch panchromatic sharpening method
CN115100075A (en) Hyperspectral panchromatic sharpening method based on spectral constraint and residual error attention network
CN115861749A (en) Remote sensing image fusion method based on window cross attention
Daithankar et al. Analysis of the wavelet domain filtering approach for video super-resolution
CN109785253B (en) Panchromatic sharpening post-processing method based on enhanced back projection
CN114638761A (en) Hyperspectral image panchromatic sharpening method, device and medium
CN115131258A (en) Hyperspectral, multispectral and panchromatic image fusion method based on sparse tensor prior
Li et al. Pansharpening via subpixel convolutional residual network
CN113284067A (en) Hyperspectral panchromatic sharpening method based on depth detail injection network

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant