CN114638762A - Modularized hyperspectral image scene self-adaptive panchromatic sharpening method - Google Patents

Modularized hyperspectral image scene self-adaptive panchromatic sharpening method Download PDF

Info

Publication number
CN114638762A
CN114638762A CN202210293380.7A CN202210293380A CN114638762A CN 114638762 A CN114638762 A CN 114638762A CN 202210293380 A CN202210293380 A CN 202210293380A CN 114638762 A CN114638762 A CN 114638762A
Authority
CN
China
Prior art keywords
adaptive
scene
image
hyperspectral image
convolution
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210293380.7A
Other languages
Chinese (zh)
Inventor
贺霖
奚达涵
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
South China University of Technology SCUT
Original Assignee
South China University of Technology SCUT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by South China University of Technology SCUT filed Critical South China University of Technology SCUT
Priority to CN202210293380.7A priority Critical patent/CN114638762A/en
Publication of CN114638762A publication Critical patent/CN114638762A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G06T5/73—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Abstract

The invention discloses a modularized hyperspectral image scene self-adaptive panchromatic sharpening method, which comprises the following steps of: reading the hyperspectral image and the full-color image matched with the hyperspectral image, and preprocessing the hyperspectral image and the full-color image; constructing a training data set and a test data set; constructing a multi-scale feature extraction module, further obtaining a scene self-adaptive sharpening module, and constructing a modular scene self-adaptive convolution neural network model; initializing weights and biases of all convolution layers of the modularized scene self-adaptive convolution neural network model, inputting a training data set into the modularized scene self-adaptive convolution neural network model, and obtaining a prediction image through network forward propagation to finish training; and inputting the test data set into the trained modularized scene self-adaptive convolutional neural network model to obtain a hyperspectral image with high spatial resolution. The invention effectively reduces the local distortion in the sharpening result and enhances the panchromatic sharpening effect.

Description

Modularized hyperspectral image scene self-adaptive panchromatic sharpening method
Technical Field
The invention relates to the field of remote sensing image processing, in particular to a modularized hyperspectral image scene self-adaptive panchromatic sharpening method.
Background
In order to ensure that an imaging result has an acceptable signal to noise ratio, in general, a spectral imaging system needs to comprehensively consider the mutual restriction factors between the spatial resolution and the spectral resolution of a remote sensing image, a hyperspectral image with high spatial resolution is difficult to directly acquire through a single sensor, a hyperspectral image with relatively low spatial resolution and a single-band full-color image with high spatial resolution are acquired, and then the spectral information and the spatial information of the two images are fused through an image processing technology, so that the hyperspectral image with high spatial resolution is generated. This process is called hyperspectral image panchromatic sharpening.
Traditional hyperspectral image panchromatic sharpening methods can be divided into three major categories, including component substitution methods, multiresolution analysis methods and methods based on variational optimization. The component substitution method replaces the spatial components of the low-spatial-resolution hyperspectral image with the full-color image through a domain transformation technology so as to realize a sharpening effect, and the method mainly comprises a principal component analysis transformation method, a Schmidt orthogonal transformation method and the like. The multiresolution analysis method relates to the extraction and reinjection of spatial detail information under multiple scales, and the method mainly comprises a wavelet transform method, a Laplace pyramid transform method and the like. The method based on the variation optimization reconstructs the high-spatial-resolution hyperspectral image by using a variation theory and an iterative optimization algorithm to solve an image degradation inverse problem, and the method mainly comprises a nonnegative matrix decomposition algorithm, a Bayesian algorithm and the like.
In recent years, due to the mining of large-scale data and the improvement of hardware computing power, a plurality of hyperspectral image panchromatic sharpening methods based on a neural network are proposed in succession, and the performance effect far exceeding that of the traditional panchromatic sharpening method is achieved. These methods generally utilize convolutional neural networks to obtain high spatial resolution hyperspectral images from low spatial resolution hyperspectral images as well as full-color images in an end-to-end manner. However, due to the weight sharing characteristic of the convolution kernel and the solidification of kernel parameters in the test process, the existing convolution neural network-based method tends to pay attention to the global sharpening effect, neglects the fact that the local scene characteristics have differences, and lacks a means for performing adaptive adjustment according to the scene characteristics of different spatial positions in the input image, so that the sharpening result is easy to generate local distortion.
Disclosure of Invention
In order to overcome the defects and shortcomings of the prior art, the invention provides a modularized hyperspectral image scene self-adaptive panchromatic sharpening method.
The method follows the idea of modularization, and designs a multi-scale feature extraction module and a scene self-adaptive sharpening module. The multi-scale feature extraction module can provide a receptive field with a plurality of scales by combining the hole convolution with different hole rates, thereby fully capturing targets with different sizes in the input feature map and effectively extracting the spatial information of the input feature map. The scene self-adaptive sharpening module can generate a group of modulation convolution kernels in a self-adaptive mode according to scene characteristics of different spatial positions in the input feature map, so that the scenes of the different spatial positions of the input feature map are finely adjusted through the group of convolution kernels, and local distortion in a sharpening result is reduced. On the basis, the invention uses the designed multi-scale feature extraction module and the scene self-adaptive sharpening module to construct a modularized scene self-adaptive convolution neural network, and the network can carry out gradual self-adaptive adjustment on scenes of all spatial positions according to the difference of scene characteristics of different spatial positions in an input image through the series connection of the plurality of scene self-adaptive sharpening modules, thereby effectively reducing the local distortion in a sharpening result and enhancing the panchromatic sharpening effect.
The invention adopts the following technical scheme:
a modularized hyperspectral image scene self-adaptive panchromatic sharpening method comprises the following steps:
reading the hyperspectral image and the full-color image matched with the hyperspectral image, and preprocessing the hyperspectral image and the full-color image;
constructing a training data set and a testing data set based on the preprocessed hyperspectral image and full-color image;
constructing a multi-scale feature extraction module, further obtaining a scene self-adaptive sharpening module, and constructing a modular scene self-adaptive convolution neural network model according to the scene self-adaptive sharpening module;
the method specifically comprises the following steps: the multi-scale feature extraction module comprises three convolutional layers, wherein one polymer convolutional layer is input into the three convolutional layers, and each convolutional layer comprises a first cavity convolutional layer, a second cavity convolutional layer and a third cavity convolutional layer;
the scene self-adaptive sharpening module comprises a multi-scale feature extraction module, an expansion convolution layer, a space spectrum conversion layer, two scene fine-tuning convolution layers and a scene self-adaptive convolution layer;
initializing weights and biases of all convolution layers of the modularized scene self-adaptive convolution neural network model, inputting a training data set into the modularized scene self-adaptive convolution neural network model, and obtaining a prediction image through network forward propagation to finish training;
and inputting the test data set into the trained modularized scene self-adaptive convolutional neural network model to obtain a hyperspectral image with high spatial resolution.
Further, the spatial dimensions of the hyperspectral image and the panchromatic image satisfy the following relationship:
r＝h2/h1＝w2/w1
where r denotes the ratio of the spatial resolution of the panchromatic image and the hyperspectral image, h1And w1Respectively representing the height and width, h, of the hyperspectral image2And w2Respectively representing the height and width of a full color image.
Further, the pre-processing comprises:
performing low-pass filtering on the hyperspectral image and the panchromatic image by using a smoothing filter adaptive to frequency response to obtain a first hyperspectral image and a first panchromatic image;
down-sampling the first hyperspectral image and the first panchromatic image to obtain a second hyperspectral image and a second panchromatic image with degraded spatial resolution by r times;
and upsampling the second hyperspectral image by using a polynomial interpolation method to obtain a third hyperspectral image of which the spatial resolution is improved by r times.
Further, in the multi-scale feature extraction module, the void rates of the three first void convolution layers are different, the void rates of the three second void convolution layers are different, and the void rates of the three third void convolution layers are different.
Further, the scene adaptive sharpening module comprises:
multi-scale feature extraction module for inputting feature map
Figure BDA0003562348980000031
Output 64 characteristic maps
Figure BDA0003562348980000032
Extracting multi-scale spatial feature information;
a dilated convolution layer comprising 576 standard convolution kernels with a 3 × 3 receptive field, input feature maps
Figure BDA0003562348980000033
Output 576 feature maps E(i)；
Space spectrum conversion layer, input feature map E(i)Output 64 × h2×w2Scene self-adaptive convolution kernel K with 3 x 3 group receptive fields(i)；
The first scene trim convolution layer, consists of 64 standard convolution kernels with a 3 × 3 receptive field. Input feature map
Figure BDA0003562348980000034
Output 64 characteristic maps
Figure BDA0003562348980000035
The second scene trim convolution layer, consists of 64 standard convolution kernels with a 3 × 3 receptive field. Input feature map
Figure BDA0003562348980000036
Output 64 characteristic maps
Figure BDA0003562348980000037
Scene adaptive convolutional layer, input feature map
Figure BDA0003562348980000038
And scene adaptive convolution kernel K(i)Outputting 64 characteristic maps
Figure BDA0003562348980000039
Further, a modular scene adaptive convolutional neural network model is constructed according to the scene adaptive sharpening module, and the method specifically comprises the following steps:
the first spectrum compression convolution layer comprises 64 standard convolution kernels with 1 multiplied by 1 receptive field and is input into a hyperspectral image training sample
Figure BDA00035623489800000310
Output 64 feature maps
Figure BDA00035623489800000311
A second spectrum compression convolution layer containing 64 standard convolution kernels with 1 × 1 receptive field and input characteristic diagram
Figure BDA00035623489800000312
Output 64 characteristic maps
Figure BDA00035623489800000313
Splice layer, input panchromatic image training sample
Figure BDA00035623489800000314
And characteristic diagrams
Figure BDA00035623489800000315
Output 65 characteristic maps C(i)；
A first scene adaptive sharpening module, input feature map C(i)Realizing the first-level scene adaptive modulation;
the second scene self-adaptive sharpening module inputs the output of the first-level scene self-adaptive modulation to realize the second-level scene self-adaptive modulation;
the third scene self-adaptive sharpening module inputs the output of the second-level scene self-adaptive modulation and outputs 64 characteristic maps
Figure BDA0003562348980000041
Realizing third-level scene adaptive modulation;
a first spectral reconstruction convolution layer containing 64 standard convolution kernels with a 1 × 1 reception field, and an input feature map
Figure BDA0003562348980000042
Output 64 characteristic maps
Figure BDA0003562348980000043
The second spectral reconstruction convolution layer contains 64 standard convolution kernels with a 1 × 1 receptive field. Input feature map
Figure BDA0003562348980000044
Outputting b characteristic graphs
Figure BDA0003562348980000045
A spectrum compensation layer for inputting a hyperspectral image training sample
Figure BDA0003562348980000046
And characteristic diagrams
Figure BDA0003562348980000047
Outputting a predicted image O(i)。
Further, initializing weights and biases of each convolution layer of the modularized scene self-adaptive convolution neural network model, inputting a training data set into the modularized scene self-adaptive convolution neural network model, obtaining a prediction image through network forward propagation, and completing training, specifically:
presetting fixed value parameters including a learning rate, iteration times and the number of input samples, and initializing the weight and the bias of each convolution layer of the modularized scene self-adaptive convolution neural network model;
obtaining a training sample with low spatial resolution from a training data set, inputting the training sample into a modular scene self-adaptive convolutional neural network model, and obtaining a predicted image through network forward propagation;
selecting the average absolute error as a loss function, calculating an error value between the predicted image and the high-spatial-resolution reference image, minimizing the error value by using a gradient-based optimization algorithm, and iteratively updating the weight and the bias of the modular scene self-adaptive convolutional neural network;
and when the error value converges to the minimum value, obtaining the optimal weight and bias of the modular scene self-adaptive convolutional neural network, and storing the optimal weight and bias to obtain the trained modular scene self-adaptive convolutional neural network model.
Further, the expression of the mean absolute error as a loss function is:
Figure BDA0003562348980000048
where phi represents the input-output mapping relation of the modular scene adaptive convolutional neural network, theta represents the weight and bias of the network, and NpRepresenting the number of training samples input during each round of iterative optimization, | · | | non-calculationFRepresenting the Frobenius norm.
A storage medium having stored thereon a computer program which, when executed by a processor, implements the method of adaptive panchromatic sharpening of hyperspectral image scenes.
An apparatus comprising a memory, a processor, and a method stored on the memory and executable on the processor for adaptive panchromatic sharpening of hyperspectral image scenes.
The invention has the beneficial effects that:
(1) the modularized hyperspectral image scene self-adaptive panchromatic sharpening method provided by the invention designs a multi-scale feature extraction module which can provide a receptive field with a plurality of scales by combining the hole convolution with different hole rates, thereby fully capturing targets with different sizes in an input feature map and effectively extracting the spatial information of the input feature map.
(2) The modularized hyperspectral image scene self-adaptive panchromatic sharpening method provided by the invention designs a scene self-adaptive sharpening module which can generate a group of modulation convolution kernels in a self-adaptive manner according to scene characteristics of different spatial positions in an input feature map, so that the scenes of the different spatial positions of the input feature map are finely adjusted through the group of convolution kernels, and the local distortion in a sharpening result is reduced.
(3) The modularized hyperspectral image scene self-adaptive panchromatic sharpening method provided by the invention uses the designed multi-scale feature extraction module and the scene self-adaptive sharpening module to construct a modularized scene self-adaptive convolutional neural network, and the network can gradually and self-adaptively adjust scenes of various spatial positions according to the difference of scene characteristics of different spatial positions in an input image through the series connection of the plurality of scene self-adaptive sharpening modules, thereby effectively reducing local distortion in a sharpening result and enhancing the panchromatic sharpening effect.
Drawings
FIG. 1 is a flow chart of the operation of the present invention;
FIG. 2 is a block diagram of the multi-scale feature extraction module of the present invention;
FIG. 3 is a block diagram of a scene adaptive sharpening module of the present invention;
FIG. 4 is a block diagram of a scene adaptive convolutional neural network model of the present invention;
fig. 5(a) is a Houston hyperspectral reference image, fig. 5(b) is a third hyperspectral image after up-sampling processing by using a bicubic interpolation method, fig. 5(c) is an image after processing by using a nonnegative matrix factorization algorithm, fig. 5(d) is an image after processing by using a bayesian algorithm, and fig. 5(e) is an image after processing by using the method of the embodiment.
Detailed Description
The present invention will be described in further detail with reference to examples and drawings, but the present invention is not limited to these examples.
Example 1
As shown in fig. 1 to 4, a modularized hyperspectral image scene adaptive panchromatic sharpening method includes the following steps:
s1, reading the original hyperspectral image and the matched panchromatic image and preprocessing the hyperspectral image and the matched panchromatic image;
read hyperspectral image
Figure BDA0003562348980000061
And paired full-color images
Figure BDA0003562348980000062
The space size of (a) satisfies the following quantitative relationship:
r＝h2/h1＝w2/w1
where r denotes the ratio of the spatial resolution of the panchromatic image and the hyperspectral image, h1And w1Respectively representing the height and width, h, of the hyperspectral image2And w2Respectively representing the height and width of a full color image. And b represents the number of spectral channels of the hyperspectral image.
Further, the specific process of the pretreatment is as follows: using a filter with a specific frequency response to the original hyperspectral image
Figure BDA0003562348980000063
And full color image
Figure BDA0003562348980000064
Carrying out smooth filtering to obtain a first hyperspectral image and a first panchromatic image, and then carrying out down-sampling processing of reducing the spatial resolution by r times on the first hyperspectral image and the first panchromatic image to obtain a second hyperspectral image
Figure BDA0003562348980000065
And a second full-color image
Figure BDA0003562348980000066
Then, the second hyperspectral image is subjected to upsampling processing for improving the spatial resolution by r times by using a polynomial interpolation method to obtain a third hyperspectral image
Figure BDA0003562348980000067
S2, constructing a training data set and a testing data set based on the preprocessed hyperspectral image and full-color image;
further, according to the principle of non-overlapping each other, from the second to the thirdThree high spectral images
Figure BDA0003562348980000068
Specific region of and a second full-color image
Figure BDA0003562348980000069
At a fixed sampling interval
Figure BDA00035623489800000610
And
Figure BDA00035623489800000611
as a training sample, the training sample is,
Figure BDA00035623489800000612
and
Figure BDA00035623489800000613
randomly ordering to form a training data set; from the third hyperspectral image
Figure BDA00035623489800000614
And the second full-color image
Figure BDA00035623489800000615
And intercepting part of the sub-images as test samples at the corresponding positions, and forming a test data set.
It is generally considered that the training set is required to include the characteristics of all the surface features as much as possible, and is representative, so that the partial region must be an image region with rich surface feature types. The specific region in the present embodiment is the relatively rich image region.
S3 constructs a multi-scale feature extraction module.
Further, the multi-scale feature extraction module comprises three convolutional layers, wherein one polymer convolutional layer is input into each of the three convolutional layers, and each convolutional layer comprises a first cavity convolutional layer, a second cavity convolutional layer and a third cavity convolutional layer.
The method specifically comprises the following steps:
the first hole convolution layer D-Conv1_1 includes 64 hole convolution kernels having a reception field of 3 × 3 and a hole rate D of 1. Input feature map
Figure BDA00035623489800000616
Output 64 characteristic maps
Figure BDA00035623489800000617
The operation process can be expressed as
Figure BDA00035623489800000618
Figure BDA00035623489800000619
Wherein
Figure BDA00035623489800000620
And
Figure BDA00035623489800000621
a weight matrix and a bias matrix respectively representing the layer of hole convolution kernels,
Figure BDA00035623489800000622
represents the Leaky Relu activation function;
the first hole convolution layer D-Conv1_2 includes 64 hole convolution kernels having a reception field of 3 × 3 and a hole rate D of 2. Input feature map
Figure BDA0003562348980000071
Output 64 characteristic maps
Figure BDA0003562348980000072
The operation process can be expressed as
Figure BDA0003562348980000073
Figure BDA0003562348980000074
Wherein
Figure BDA0003562348980000075
And
Figure BDA0003562348980000076
a weight matrix and a bias matrix respectively representing the layer of hole convolution kernels,
Figure BDA0003562348980000077
also the Leaky Relu activation function;
the first hole convolution layer D-Conv1_3 includes 64 hole convolution kernels having a reception field of 3 × 3 and a hole rate D of 3. Input feature map
Figure BDA0003562348980000078
Output 64 characteristic maps
Figure BDA0003562348980000079
The operation process can be expressed as
Figure BDA00035623489800000710
Figure BDA00035623489800000711
Wherein
Figure BDA00035623489800000712
And
Figure BDA00035623489800000713
respectively representing the weight matrix and the bias matrix of the layer of hole convolution kernels,
Figure BDA00035623489800000714
also the Leaky Relu activation function;
the second hole convolution layer D-Conv2_1 includes 64 hole convolution kernels having a reception field of 3 × 3 and a hole rate D of 1. Input feature map
Figure BDA00035623489800000715
Output 64 characteristic maps
Figure BDA00035623489800000716
The operation process can be expressed as
Figure BDA00035623489800000717
Figure BDA00035623489800000718
Wherein
Figure BDA00035623489800000719
And
Figure BDA00035623489800000720
respectively representing the weight matrix and the bias matrix of the layer of hole convolution kernels,
Figure BDA00035623489800000721
also the Leaky Relu activation function;
the second hole convolution layer D-Conv2_2 includes 64 hole convolution kernels having a reception field of 3 × 3 and a hole rate D of 2. Input feature map
Figure BDA00035623489800000722
And
Figure BDA00035623489800000723
output 64 feature maps
Figure BDA00035623489800000724
The operation process can be expressed as
Figure BDA00035623489800000725
Wherein
Figure BDA00035623489800000726
And
Figure BDA00035623489800000727
a weight matrix and a bias matrix respectively representing the layer of hole convolution kernels,
Figure BDA00035623489800000728
to representWill be provided with
Figure BDA00035623489800000729
And
Figure BDA00035623489800000730
the splicing is performed in the channel dimension and,
Figure BDA00035623489800000731
also the Leaky Relu activation function;
the second hole convolution layer D-Conv2_3 includes 64 hole convolution kernels having a reception field of 3 × 3 and a hole rate D of 3. Input feature map
Figure BDA00035623489800000732
And
Figure BDA00035623489800000733
output 64 characteristic maps
Figure BDA00035623489800000734
The operation process can be expressed as
Figure BDA00035623489800000735
Wherein
Figure BDA00035623489800000736
And
Figure BDA00035623489800000737
a weight matrix and a bias matrix respectively representing the layer of hole convolution kernels,
Figure BDA00035623489800000738
show that
Figure BDA00035623489800000739
And
Figure BDA00035623489800000740
the splicing is performed in the channel dimension and,
Figure BDA00035623489800000741
also the Leaky Relu activation function;
the third hole convolution layer D-Conv3_1 includes 64 hole convolution kernels having a reception field of 3 × 3 and a hole rate D of 1. Input feature map
Figure BDA00035623489800000742
Output 64 characteristic maps
Figure BDA00035623489800000743
The operation process can be expressed as
Figure BDA00035623489800000744
Figure BDA00035623489800000745
Wherein
Figure BDA00035623489800000746
And
Figure BDA00035623489800000747
a weight matrix and a bias matrix respectively representing the layer of hole convolution kernels,
Figure BDA00035623489800000748
also the Leaky Relu activation function;
the third hole convolution layer D-Conv3_2 includes 64 hole convolution kernels having a reception field of 3 × 3 and a hole rate D of 2. Input feature map
Figure BDA00035623489800000749
And
Figure BDA00035623489800000750
output 64 characteristic maps
Figure BDA00035623489800000751
The operation process can be expressed as
Figure BDA00035623489800000752
Wherein
Figure BDA00035623489800000753
And
Figure BDA00035623489800000754
a weight matrix and a bias matrix respectively representing the layer of hole convolution kernels,
Figure BDA00035623489800000755
show that
Figure BDA00035623489800000756
And
Figure BDA00035623489800000757
the splicing is performed in the channel dimension and,
Figure BDA0003562348980000081
the Leaky Relu activation function is also denoted;
the third hole convolution layer D-Conv3_3 includes 64 hole convolution kernels having a reception field of 3 × 3 and a hole rate D of 3. Input feature map
Figure BDA0003562348980000082
And
Figure BDA0003562348980000083
output 64 characteristic maps
Figure BDA0003562348980000084
The operation process can be expressed as
Figure BDA0003562348980000085
Wherein
Figure BDA0003562348980000086
And
Figure BDA0003562348980000087
weight matrix and bias matrix respectively representing convolution kernel of the layer of holes，
Figure BDA0003562348980000088
Show that
Figure BDA0003562348980000089
And
Figure BDA00035623489800000810
the splicing is performed in the channel dimension and,
Figure BDA00035623489800000811
also the Leaky Relu activation function;
aggregate convolution layer A-Conv, contains 64 standard convolution kernels with a 3 × 3 receptive field. Input feature map
Figure BDA00035623489800000812
And
Figure BDA00035623489800000813
output 64 characteristic maps
Figure BDA00035623489800000814
The operation process can be expressed as
Figure BDA00035623489800000815
Figure BDA00035623489800000816
Wherein WAAnd BAA weight matrix and an offset matrix respectively representing the layer of standard convolution kernels,
Figure BDA00035623489800000817
show that
Figure BDA00035623489800000818
And
Figure BDA00035623489800000819
the splicing is performed in the channel dimension and,
Figure BDA00035623489800000820
the Leaky Relu activation function is also indicated.
S4 constructs a scene adaptive sharpening module.
The scene self-adaptive sharpening module comprises a multi-scale feature extraction module, an expansion convolution layer, a space spectrum conversion layer, two scene fine-tuning convolution layers and a scene self-adaptive convolution layer.
Multi-scale feature extraction module for inputting feature map
Figure BDA00035623489800000821
Output 64 characteristic maps
Figure BDA00035623489800000822
Extracting multi-scale spatial feature information;
dilated convolution layer E-Conv, contains 576 standard convolution kernels with a 3 × 3 receptive field. Input feature map
Figure BDA00035623489800000823
Output 576 feature maps E(i). The operation process can be expressed as
Figure BDA00035623489800000824
Wherein WEAnd BEA weight matrix and a bias matrix representing the standard convolution kernel of the layer, respectively, the layer not using the activation function.
Space spectrum conversion layer C-to-S, input feature map E(i)Output 64 × h2×w2Scene self-adaptive convolution kernel K with 3 x 3 group receptive fields(i). This layer will input a feature map E(i)The channel dimensionality is rearranged and converted into space dimensionality to generate a 3 x 3 scene self-adaptive convolution kernel K(i)；
The first scene trim convolutional layer S-Conv1, contains 64 standard convolution kernels with a 3 × 3 receptive field. Input feature map
Figure BDA00035623489800000825
Output 64 feature maps
Figure BDA00035623489800000826
The operation process can be expressed as
Figure BDA00035623489800000827
Wherein
Figure BDA00035623489800000828
And
Figure BDA00035623489800000829
a weight matrix and an offset matrix respectively representing the layer of standard convolution kernels,
Figure BDA00035623489800000830
Figure BDA00035623489800000831
indicating the Relu activation function.
The second scene trim convolutional layer S-Conv2, contains 64 standard convolution kernels with a 3 × 3 receptive field. Input feature map
Figure BDA00035623489800000832
Output 64 characteristic maps
Figure BDA00035623489800000833
The operation process can be expressed as
Figure BDA00035623489800000834
Wherein
Figure BDA00035623489800000835
And
Figure BDA00035623489800000836
a weight matrix and an offset matrix respectively representing the layer of standard convolution kernels,
Figure BDA00035623489800000837
the Relu activation function is also indicated.
Scene adaptive convolutional layer SA-Conv, outputCharacteristic diagram
Figure BDA0003562348980000091
And scene adaptive convolution kernel K(i)Output 64 feature maps
Figure BDA0003562348980000092
The operation process can be expressed as
Figure BDA0003562348980000093
Wherein
Figure BDA0003562348980000094
Which represents a pixel-by-pixel convolution operation,
Figure BDA0003562348980000095
the Relu activation function is also indicated.
S5, constructing a modular scene self-adaptive convolutional neural network model according to the scene self-adaptive sharpening module;
further, the method comprises the following steps:
the first spectrally compressed convolutional layer Conv1, contains 64 standard convolutional kernels with a 1 × 1 reception field. Input hyperspectral image training samples
Figure BDA0003562348980000096
Output 64 characteristic maps
Figure BDA0003562348980000097
The operation process can be expressed as
Figure BDA0003562348980000098
Figure BDA0003562348980000099
Wherein W1And B1A weight matrix and an offset matrix respectively representing the layer of standard convolution kernels,
Figure BDA00035623489800000910
represents the Relu activation function;
second spectral compressionConvolutional layer Conv2, contains 64 standard convolution kernels with a 1 × 1 receptive field. Input feature map
Figure BDA00035623489800000911
Output 64 characteristic maps
Figure BDA00035623489800000912
The operation process can be expressed as
Figure BDA00035623489800000913
Wherein W2And B2A weight matrix and an offset matrix respectively representing the layer of standard convolution kernels,
Figure BDA00035623489800000914
also denoted Relu activation function;
splicing layer Concat. Input panchromatic image training sample
Figure BDA00035623489800000915
And characteristic diagrams
Figure BDA00035623489800000916
Output 65 characteristic maps C(i). The layer will be full color image training sample
Figure BDA00035623489800000917
And characteristic diagrams
Figure BDA00035623489800000918
Splicing in channel dimension to obtain spliced characteristic diagram C(i)；
A first scene adaptive sharpening module for inputting a feature map C(i). Realizing the first-level scene self-adaptive modulation;
and the second scene self-adaptive sharpening module inputs the output of the first-level scene self-adaptive modulation. Realizing the second-level scene adaptive modulation;
the third scene self-adaptive sharpening module inputs the output of the second-level scene self-adaptive modulation and outputs 64 characteristic maps
Figure BDA00035623489800000919
Realizing third-level scene adaptive modulation;
the first spectrally reconstructed convolutional layer Conv3, which contains 64 standard convolutional kernels with a 1 × 1 reception field. Input feature map
Figure BDA00035623489800000920
Output 64 characteristic maps
Figure BDA00035623489800000921
The operation process can be expressed as
Figure BDA00035623489800000922
Wherein W3And B3A weight matrix and an offset matrix respectively representing the layer of standard convolution kernels,
Figure BDA00035623489800000923
also denoted Relu activation function;
the second spectrally reconstructed convolutional layer Conv4, which contains 64 standard convolutional kernels with a 1 × 1 reception field. Input feature map
Figure BDA00035623489800000924
Outputting b characteristic graphs
Figure BDA00035623489800000925
The operation process can be expressed as
Figure BDA00035623489800000926
Wherein W4And B4A weight matrix and a bias matrix respectively representing the standard convolution kernel of the layer, the layer not using the activation function;
the spectral compensation layer compact. Input hyperspectral image training samples
Figure BDA00035623489800000927
And characteristic diagrams
Figure BDA00035623489800000928
Outputting a predicted image O(i). The operation process can be expressed as
Figure BDA00035623489800000929
Wherein
Figure BDA00035623489800000930
Indicating a pixel-by-pixel addition operation.
S6, setting hyper-parameters and initializing the weight and bias of each convolution layer of the modularized scene self-adaptive convolution neural network.
The hyper-parameters are preset fixed value parameters, and the fixed value parameters comprise a learning rate, iteration times, the number of input samples and the like.
S7, based on the training data set, obtaining a training sample with low spatial resolution, inputting the training sample into a modular scene self-adaptive convolutional neural network, and obtaining a prediction image through network forward propagation;
s8, selecting the average absolute error as a loss function, calculating an error value between the predicted image and the high spatial resolution reference image, minimizing the error value by using a gradient-based optimization algorithm, and iteratively updating the weight and the bias of the modularized scene self-adaptive convolution neural network. Repeating S7-S8 when the error value does not converge to a minimum value;
the expression of the mean absolute error loss function selected in step 8 is as follows:
Figure BDA0003562348980000101
where phi represents the input-output mapping relation of the modular scene adaptive convolutional neural network, theta represents the weight and bias of the network, and NpRepresenting the number of training samples input during each round of iterative optimization, | · | | non-wovenFRepresenting the Frobenius norm.
S9, when the error value converges to the minimum value, obtaining the optimal weight and bias of the modular scene self-adaptive convolutional neural network and storing the optimal weight and bias;
s10 multiplexing the modularized scene self-adaptive convolutional neural network structure, loading the optimal network weight and biasing to the network structure;
and S11, based on the test data set, obtaining a test sample with low spatial resolution, inputting the test sample into the modularized scene self-adaptive convolutional neural network loaded with the optimal weight and the bias, and outputting a hyperspectral image with high spatial resolution.
The present example was validated using Houston hyperspectral and full-color images from the ITRES CASI-1500 sensor. The hyperspectral image contains 144 spectral channels and has a spatial resolution of 30 x 30, the full-color image has a spatial resolution of 150 x 150, and the ratio of the spatial resolutions of the two is 1: 5.
Fig. 5(a) is a Houston hyperspectral reference image, fig. 5(b) is a third hyperspectral image after upsampling processing by using a bicubic interpolation method, fig. 5(c) is an image after processing by using a non-negative matrix factorization algorithm, fig. 5(d) is an image after processing by using a bayesian algorithm, and fig. 5(e) is an image after processing by using the method described in the embodiment. As can be seen from the figure, compared with a reference image, the image subjected to up-sampling processing by using a bicubic interpolation method loses a large amount of spatial detail information, and has a serious spatial distortion problem; the image processed by the nonnegative matrix factorization algorithm has a good space detail reconstruction effect, but a certain degree of spectral distortion occurs in a partial region; the image processed by the Bayesian algorithm has higher spectral fidelity, but still has a more obvious spatial blurring phenomenon in a partial region; the image processed by the method of the embodiment not only realizes the best global sharpening effect, but also effectively reduces the local distortion phenomenon, and achieves remarkable improvement in both the spatial detail reconstruction and the spectral fidelity.
Example 2
A storage medium having stored thereon a computer program which, when executed by a processor, implements the method of adaptive panchromatic sharpening of hyperspectral image scenes.
Reading a hyperspectral image and a full-color image matched with the hyperspectral image, and preprocessing the hyperspectral image and the full-color image;
constructing a training data set and a testing data set based on the preprocessed hyperspectral image and full-color image;
constructing a multi-scale feature extraction module, further obtaining a scene self-adaptive sharpening module, and constructing a modular scene self-adaptive convolution neural network model according to the scene self-adaptive sharpening module;
the method specifically comprises the following steps: the multi-scale feature extraction module comprises three convolutional layers, wherein one polymer convolutional layer is input into the three convolutional layers, and each convolutional layer comprises a first cavity convolutional layer, a second cavity convolutional layer and a third cavity convolutional layer;
the scene self-adaptive sharpening module comprises a multi-scale feature extraction module, an expansion convolution layer, a space spectrum conversion layer, two scene fine-tuning convolution layers and a scene self-adaptive convolution layer;
initializing weights and biases of all convolution layers of the modularized scene self-adaptive convolution neural network model, inputting a training data set into the modularized scene self-adaptive convolution neural network model, and obtaining a prediction image through network forward propagation to finish training;
and inputting the test data set into the trained modularized scene self-adaptive convolutional neural network model to obtain a hyperspectral image with high spatial resolution.
Example 3
An apparatus comprising a memory, a processor, and a method stored on the memory and executable on the processor for adaptive panchromatic sharpening of hyperspectral image scenes.
The above embodiments are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments, and any other changes, modifications, substitutions, combinations, and simplifications which do not depart from the spirit and principle of the present invention should be construed as equivalents thereof, and all such changes, modifications, substitutions, combinations, and simplifications are intended to be included in the scope of the present invention.

Claims (10)

1. A modularized hyperspectral image scene self-adaptive panchromatic sharpening method is characterized by comprising the following steps:
reading the hyperspectral image and the full-color image matched with the hyperspectral image, and preprocessing the hyperspectral image and the full-color image;
constructing a training data set and a testing data set based on the preprocessed hyperspectral image and the preprocessed panchromatic image;
constructing a multi-scale feature extraction module, further obtaining a scene self-adaptive sharpening module, and constructing a modular scene self-adaptive convolution neural network model according to the scene self-adaptive sharpening module;
the method specifically comprises the following steps: the multi-scale feature extraction module comprises three convolutional layers, wherein one polymer convolutional layer is input into the three convolutional layers, and each convolutional layer comprises a first cavity convolutional layer, a second cavity convolutional layer and a third cavity convolutional layer;
the scene self-adaptive sharpening module comprises a multi-scale feature extraction module, an expansion convolution layer, a space spectrum conversion layer, two scene fine-tuning convolution layers and a scene self-adaptive convolution layer;
initializing weights and biases of all convolution layers of the modularized scene self-adaptive convolution neural network model, inputting a training data set into the modularized scene self-adaptive convolution neural network model, and obtaining a prediction image through network forward propagation to finish training;
and inputting the test data set into the trained modularized scene self-adaptive convolutional neural network model to obtain a hyperspectral image with high spatial resolution.
2. The hyperspectral image scene adaptive panchromatic sharpening method according to claim 1, wherein the spatial dimensions of the hyperspectral image and the panchromatic image satisfy the following relationship:
r＝h2/h1＝w2/w1
where r represents the ratio of the spatial resolution of the panchromatic image to the hyperspectral image, h1And w1Respectively representing the height and width, h, of the hyperspectral image2And w2Respectively representing the height and width of a full color image.
3. The hyperspectral image scene adaptive panchromatic sharpening method according to claim 1, wherein the preprocessing comprises:
performing low-pass filtering on the hyperspectral image and the panchromatic image by using a smoothing filter adaptive to frequency response to obtain a first hyperspectral image and a first panchromatic image;
down-sampling the first hyperspectral image and the first panchromatic image to obtain a second hyperspectral image and a second panchromatic image with degraded spatial resolution by r times;
and upsampling the second hyperspectral image by using a polynomial interpolation method to obtain a third hyperspectral image of which the spatial resolution is improved by r times.
4. The hyperspectral image scene adaptive panchromatic sharpening method according to claim 1, wherein in the multi-scale feature extraction module, the voidage of three first hole convolution layers is different, the voidage of three second hole convolution layers is different, and the voidage of three third hole convolution layers is different.
5. The hyperspectral image scene adaptive panchromatic sharpening method according to claim 1, wherein the scene adaptive sharpening module comprises:
multi-scale feature extraction module, input feature map
Figure FDA0003562348970000021
Output 64 characteristic maps
Figure FDA0003562348970000022
Extracting multi-scale spatial feature information;
a dilated convolution layer comprising 576 standard convolution kernels with a 3 × 3 receptive field, input feature maps
Figure FDA0003562348970000023
Output 576 feature maps E(i)；
Space spectrum conversion layer, input feature map E(i)Output 64 × h2×w2Scene self-adaptive convolution kernel K with 3 x 3 group receptive fields(i)；
First fieldThe scene fine tuning convolution layer comprises 64 standard convolution kernels with the reception fields of 3 multiplied by 3 and an input feature map
Figure FDA0003562348970000024
Output 64 characteristic maps
Figure FDA0003562348970000025
A second scene fine tuning convolution layer including 64 standard convolution kernels with 3 × 3 receptive fields, and input feature map
Figure FDA0003562348970000026
Output 64 characteristic maps
Figure FDA0003562348970000027
Scene adaptive convolutional layer, input feature map
Figure FDA0003562348970000028
And scene adaptive convolution kernel K(i)Outputting 64 characteristic maps
Figure FDA0003562348970000029
6. The hyperspectral image scene adaptive panchromatic sharpening method according to claim 1, wherein a modular scene adaptive convolutional neural network model is constructed according to a scene adaptive sharpening module, and specifically comprises the following steps:
the first spectrum compression convolution layer comprises 64 standard convolution kernels with 1 multiplied by 1 receptive field and is input into a hyperspectral image training sample
Figure FDA00035623489700000210
Output 64 feature maps
Figure FDA00035623489700000211
The second lightThe spectrum compressed convolution layer comprises 64 standard convolution kernels with 1 × 1 receptive field and an input feature map
Figure FDA00035623489700000212
Output 64 characteristic maps
Figure FDA00035623489700000213
Splice layer, input panchromatic image training sample
Figure FDA00035623489700000214
And characteristic diagrams
Figure FDA00035623489700000215
Output 65 characteristic maps C(i)；
A first scene adaptive sharpening module for inputting a feature map C(i)Realizing the first-level scene adaptive modulation;
the second scene self-adaptive sharpening module inputs the output of the first-level scene self-adaptive modulation to realize the second-level scene self-adaptive modulation;
the third scene self-adaptive sharpening module inputs the output of the second-level scene self-adaptive modulation and outputs 64 characteristic maps
Figure FDA00035623489700000216
Realizing third-level scene adaptive modulation;
a first spectral reconstruction convolution layer containing 64 standard convolution kernels with a 1 × 1 reception field, and an input feature map
Figure FDA00035623489700000217
Output 64 feature maps
Figure FDA00035623489700000218
A second spectral reconstruction convolution layer containing 64 standard convolution kernels with a 1 × 1 reception field, and input feature map
Figure FDA0003562348970000031
Outputting b characteristic graphs
Figure FDA0003562348970000032
A spectrum compensation layer for inputting a hyperspectral image training sample
Figure FDA0003562348970000033
And characteristic diagrams
Figure FDA0003562348970000034
Outputting a predicted image O(i)。
7. The hyperspectral image scene adaptive panchromatic sharpening method according to claim 1, wherein the weight and bias of each convolution layer of the modularized scene adaptive convolution neural network model are initialized, a training data set is input into the modularized scene adaptive convolution neural network model, a predicted image is obtained through network forward propagation, and training is completed, specifically:
presetting fixed value parameters, wherein the fixed value parameters comprise a learning rate, iteration times and the number of input samples, and initializing the weight and the bias of each convolution layer of the modularized scene self-adaptive convolution neural network model;
obtaining a training sample with low spatial resolution from a training data set, inputting the training sample into a modular scene self-adaptive convolutional neural network model, and obtaining a predicted image through network forward propagation;
selecting the average absolute error as a loss function, calculating an error value between the predicted image and the high-spatial-resolution reference image, minimizing the error value by using a gradient-based optimization algorithm, and iteratively updating the weight and the bias of the modular scene self-adaptive convolutional neural network;
and when the error value converges to the minimum value, obtaining the optimal weight and bias of the modular scene self-adaptive convolutional neural network, and storing the optimal weight and bias to obtain the trained modular scene self-adaptive convolutional neural network model.
8. The hyperspectral image scene adaptive panchromatic sharpening method according to claim 7, wherein the expression of the mean absolute error as a loss function is:
Figure FDA0003562348970000035
where phi represents the input-output mapping relation of the modular scene adaptive convolutional neural network, theta represents the weight and bias of the network, and NpRepresenting the number of training samples input during each round of iterative optimization, | · | | non-wovenFRepresenting the Frobenius norm.
9. A storage medium having stored thereon a computer program, wherein the computer program, when being executed by a processor, is adapted to carry out the method for adaptive panchromatic sharpening of hyperspectral image scenes according to any of the claims 1 to 8.
10. A device comprising a memory, a processor and a method for adaptive panchromatic sharpening of hyperspectral image scenes according to any of claims 1 to 8 stored on the memory and executable on the processor.
CN202210293380.7A 2022-03-24 2022-03-24 Modularized hyperspectral image scene self-adaptive panchromatic sharpening method Pending CN114638762A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210293380.7A CN114638762A (en) 2022-03-24 2022-03-24 Modularized hyperspectral image scene self-adaptive panchromatic sharpening method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210293380.7A CN114638762A (en) 2022-03-24 2022-03-24 Modularized hyperspectral image scene self-adaptive panchromatic sharpening method

Publications (1)

Publication Number Publication Date
CN114638762A true CN114638762A (en) 2022-06-17

Family

ID=81948897

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210293380.7A Pending CN114638762A (en) 2022-03-24 2022-03-24 Modularized hyperspectral image scene self-adaptive panchromatic sharpening method

Country Status (1)

Country Link
CN (1) CN114638762A (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100288910A1 (en) * 2009-05-14 2010-11-18 Raytheon Company Adaptive spatial-spectral processing (assp)
CN109727207A (en) * 2018-12-06 2019-05-07 华南理工大学 High spectrum image sharpening method based on Forecast of Spectra residual error convolutional neural networks
CN109859110A (en) * 2018-11-19 2019-06-07 华南理工大学 The panchromatic sharpening method of high spectrum image of control convolutional neural networks is tieed up based on spectrum
CN110533077A (en) * 2019-08-01 2019-12-03 南京理工大学 Form adaptive convolution deep neural network method for classification hyperspectral imagery
CN111008936A (en) * 2019-11-18 2020-04-14 华南理工大学 Multispectral image panchromatic sharpening method
CN113362223A (en) * 2021-05-25 2021-09-07 重庆邮电大学 Image super-resolution reconstruction method based on attention mechanism and two-channel network

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100288910A1 (en) * 2009-05-14 2010-11-18 Raytheon Company Adaptive spatial-spectral processing (assp)
CN109859110A (en) * 2018-11-19 2019-06-07 华南理工大学 The panchromatic sharpening method of high spectrum image of control convolutional neural networks is tieed up based on spectrum
CN109727207A (en) * 2018-12-06 2019-05-07 华南理工大学 High spectrum image sharpening method based on Forecast of Spectra residual error convolutional neural networks
CN110533077A (en) * 2019-08-01 2019-12-03 南京理工大学 Form adaptive convolution deep neural network method for classification hyperspectral imagery
CN111008936A (en) * 2019-11-18 2020-04-14 华南理工大学 Multispectral image panchromatic sharpening method
CN113362223A (en) * 2021-05-25 2021-09-07 重庆邮电大学 Image super-resolution reconstruction method based on attention mechanism and two-channel network

Similar Documents

Publication Publication Date Title
CN109886871B (en) Image super-resolution method based on channel attention mechanism and multi-layer feature fusion
CN111028177B (en) Edge-based deep learning image motion blur removing method
Ng et al. Solving constrained total-variation image restoration and reconstruction problems via alternating direction methods
CN110415199B (en) Multispectral remote sensing image fusion method and device based on residual learning
CN110276726B (en) Image deblurring method based on multichannel network prior information guidance
CN109636769A (en) EO-1 hyperion and Multispectral Image Fusion Methods based on the intensive residual error network of two-way
CN111008936B (en) Multispectral image panchromatic sharpening method
CN110288524B (en) Deep learning super-resolution method based on enhanced upsampling and discrimination fusion mechanism
CN111028153A (en) Image processing and neural network training method and device and computer equipment
CN109920013B (en) Image reconstruction method and device based on progressive convolution measurement network
CN111353939B (en) Image super-resolution method based on multi-scale feature representation and weight sharing convolution layer
CN112669248A (en) Hyperspectral and panchromatic image fusion method based on CNN and Laplacian pyramid
CN113744136A (en) Image super-resolution reconstruction method and system based on channel constraint multi-feature fusion
CN115760814A (en) Remote sensing image fusion method and system based on double-coupling deep neural network
CN115984110A (en) Swin-transform-based second-order spectral attention hyperspectral image super-resolution method
CN113538246A (en) Remote sensing image super-resolution reconstruction method based on unsupervised multi-stage fusion network
Mikaeli et al. Single-image super-resolution via patch-based and group-based local smoothness modeling
CN115526779A (en) Infrared image super-resolution reconstruction method based on dynamic attention mechanism
CN113962882B (en) JPEG image compression artifact eliminating method based on controllable pyramid wavelet network
Deng et al. Multiple frame splicing and degradation learning for hyperspectral imagery super-resolution
CN113744134A (en) Hyperspectral image super-resolution method based on spectrum unmixing convolution neural network
CN113222812A (en) Image reconstruction method based on information flow reinforced deep expansion network
CN116883799A (en) Hyperspectral image depth space spectrum fusion method guided by component replacement model
Mandal et al. Employing structural and statistical information to learn dictionary (s) for single image super-resolution in sparse domain
CN116612009A (en) Multi-scale connection generation countermeasure network medical image super-resolution reconstruction method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination