CN114820358A - Method and system for panchromatic sharpening of hyperspectral image at any resolution - Google Patents

Method and system for panchromatic sharpening of hyperspectral image at any resolution Download PDF

Info

Publication number
CN114820358A
CN114820358A CN202210396467.7A CN202210396467A CN114820358A CN 114820358 A CN114820358 A CN 114820358A CN 202210396467 A CN202210396467 A CN 202210396467A CN 114820358 A CN114820358 A CN 114820358A
Authority
CN
China
Prior art keywords
resolution
image
scale
panchromatic
hyperspectral image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210396467.7A
Other languages
Chinese (zh)
Inventor
贺霖
谢锦桦
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
South China University of Technology SCUT
Original Assignee
South China University of Technology SCUT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by South China University of Technology SCUT filed Critical South China University of Technology SCUT
Priority to CN202210396467.7A priority Critical patent/CN114820358A/en
Publication of CN114820358A publication Critical patent/CN114820358A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G06T5/73—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformation in the plane of the image
    • G06T3/40—Scaling the whole image or part thereof
    • G06T3/4007—Interpolation-based scaling, e.g. bilinear interpolation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning

Abstract

The invention discloses a method and a system for panchromatic sharpening of a hyperspectral image at any resolution, which comprises the steps of reading an original hyperspectral image, synthesizing a corresponding panchromatic image, preprocessing the original hyperspectral image and the corresponding panchromatic image to obtain a multi-scale image pair, and obtaining a training data set, a verification data set and a test sample; constructing an arbitrary resolution panchromatic sharpening network based on extended sub-pixel convolution; initializing network parameters, randomly selecting any scale data in a training data set for training, obtaining the minimum value when the network loss is stable, inputting a test sample to obtain a sharpened target resolution hyperspectral image, wherein the corresponding network is a trained panchromatic sharpening network with any resolution of extended sub-pixel convolution. According to the method, the importance of different spatial position information is adjusted by introducing scales as prior information, so that the panchromatic sharpening performance of the hyperspectral image with any resolution is further improved.

Description

Method and system for panchromatic sharpening of hyperspectral image at any resolution
Technical Field
The invention relates to the field of remote sensing images, in particular to a method and a system for panchromatic sharpening of a hyperspectral image at any resolution.
Background
The hyperspectral image can reflect the inherent spectral characteristics of general observed ground objects due to the fine spectral resolution (within 10 nm), and is widely applied to land coverage classification, agricultural management, target detection and the like. However, the actual hyperspectral imaging process is physically limited by the imaging sensor, the spatial resolution is often poor, and the subsequent application range is limited. In order to obtain a hyperspectral image with high spatial resolution, a feasible scheme is to carry out hyperspectral image panchromatic sharpening, fuse a single-band panchromatic image with high spatial resolution with the hyperspectral image, and enhance the spatial quality of the hyperspectral image by using the spatial details of the panchromatic image. Traditional panchromatic sharpening methods can be classified into component replacement, multi-scale analysis, bayesian estimation, matrix decomposition, and the like. With the wide application of deep learning, a panchromatic sharpening method based on a deep convolutional neural network gradually becomes a research hotspot, a mapping relation of input and output hyperspectral data is fitted through end-to-end learning, and a given observation image is estimated in a prediction stage by the mapping relation to obtain a final sharpening result.
Panchromatic sharpening of hyperspectral images is also often used as a preprocessing step for remote sensing thematic map enhancement, change detection, and the like. However, in these applications, the desired target spatial resolution may not be exactly that of a full color image. Most of the existing panchromatic sharpening methods only pay attention to the fact that the resolution of a hyperspectral image is improved to the corresponding full-color image spatial resolution, and the requirements of the applications are difficult to meet. Therefore, the hyperspectral panchromatic sharpening with any resolution provides a new scheme for sharpening the hyperspectral image to any required spatial resolution, and the requirement of any possible user-customized resolution in practical application can be flexibly met. The panchromatic sharpening of the hyperspectral image with any resolution aims at obtaining the hyperspectral image under any target spatial resolution by fusing and sharpening the low-spatial-resolution hyperspectral image and the high-spatial-resolution panchromatic image with the user-defined spatial resolution as the target. The panchromatic sharpening of the hyperspectral image with any resolution is taken as a novel panchromatic sharpening task, and the prominent problems in general hyperspectral image sharpening are solved, such as obvious difference of spectral coverage ranges of the hyperspectral image and a panchromatic image, restoration of image details on a large number of continuous wave bands of the low-spatial-resolution hyperspectral image and the like. One major challenge to be faced is how a single model generalizes the panchromatic sharpening process to arbitrary scales under training of limited scales.
Disclosure of Invention
In order to overcome the defects and shortcomings of the prior art, the invention provides a method and a system for panchromatic sharpening of a hyperspectral image with any resolution.
The method is a hyperspectral image arbitrary resolution panchromatic sharpening method based on extended sub-pixel convolution, scale information is introduced through a scale attention residual error module to serve as a priori extraction feature, then the extended sub-pixel convolution module is adopted to conduct sampling on an arbitrary scale, and a hyperspectral image under a target spatial resolution is obtained. After training is carried out under a limited scale, the consistency of the sub-pixel convolution space mode is expanded, the sub-pixel convolution space mode can be generalized to hyperspectral image panchromatic sharpening with any resolution, and the single model can be generalized to any scale through the training of the limited scale.
The invention adopts the following technical scheme:
an arbitrary resolution panchromatic sharpening method for hyperspectral images comprises the following steps:
reading an original hyperspectral image H epsilon R M×N×C And synthesizing a corresponding full-color image P e R M×N Wherein M and N represent the height and width of the hyperspectral image, C represents the number of wave bands, and R represents the whole real number set;
preprocessing the original hyperspectral image and the corresponding panchromatic image, and collecting S ═ S according to the selected training scale 1 ,s 2 ,…,s k Generating a multi-scale image pair to obtain a training data set, a verification data set and a test sample;
constructing an arbitrary resolution panchromatic sharpening network based on extended sub-pixel convolution;
initializing network parameters, randomly selecting any scale data in a training data set for training, and acquiring the minimum value of network loss stability, wherein the corresponding network is a trained panchromatic sharpening network with any resolution of extended sub-pixel convolution;
and inputting a test sample, and acquiring a sharpened target resolution hyperspectral image.
Further, reading the original hyperspectral image H epsilon R M×N×C And synthesizing a corresponding full-color image P e R M×N The method specifically comprises the following steps: the full-color image corresponding to the acquired hyperspectral image is obtained by averaging all visible light wave bands contained in the hyperspectral image.
Further, the preprocessing step includes first setting S ═ S according to the selected scale set 1 ,s 2 ,…,s k And the resolution ratio r of the panchromatic image to the low-resolution hyperspectral image, calculating an intermediate scale set s ' ═ s ' of the target-resolution hyperspectral image and the panchromatic image ' 1 ,s′ 2 ,…s′ k Is of s' i ＝s i And/r, i is 1, 2. Carrying out fuzzy degradation on the original hyperspectral image H and the panchromatic image P by adopting a Gaussian filter with specific frequency response for s' i Performing multiple down-sampling to obtain a reference hyperspectral image during standard sharpening
Figure BDA0003599207410000021
And full color image
Figure BDA0003599207410000022
Wherein m ', n' satisfy the expression: m ═ M 'x s' i ，N＝n′×s′ i Then to H r Then r times of downsampling is carried out to obtain a low-resolution ratio hyperspectral image
Figure BDA0003599207410000023
Wherein m, n satisfy the expression: m ═ mxrxs' i ，N＝n×r×s′ i . Finally, low-resolution hyperspectral images are subjected to polynomial interpolation
Figure BDA0003599207410000024
Performing pre-interpolation of the scale r to obtain
Figure BDA0003599207410000025
Taking the original hyperspectral image as a target reference image H s Thereby obtaining a multi-scale image pair
Figure BDA0003599207410000031
s∈S。
Further, the extended sub-pixel convolution-based arbitrary resolution panchromatic sharpening network comprises N cascaded scale attention residual modules, an arbitrary scale attention up-sampling module and a light reconstruction module.
Further, the scale attention residual module specifically includes:
randomly selecting a scale s i E.g. S, input corresponding
Figure BDA0003599207410000032
Obtaining an initial feature map F by single-layer 3 x 3 convolution 0 Expressed as:
Figure BDA0003599207410000033
wherein f is 3*3 (. cndot.) denotes a 3 × 3 convolution.
Scale space attention residual block SSARB-N, for a single SSARB, input feature map F N-1 First, two layers of 3 × 3 convolution are performed to obtain an intermediate feature map F', which is specifically expressed as:
F′＝f 3*3 (ReLU(f 3*3 (F N-1 )))
then, the feature map is input into a spatial attention module, and the feature map F 'is firstly subjected to maximum pooling and average pooling along the channel dimension and is subjected to intermediate dimension s' i Splicing the generated 2-dimensional matrix, then sending the two layers of 7 multiplied by 7 convolution, finally obtaining a scale space attention diagram M by using a Sigmoid function,
M＝σ(f 7*7 (ReLU(f 7*7 ([MaxPool(F′)；AvgPool(F′)；ρ(1/s′ i )]))))
wherein MaxPool (. cndot.) represents the maximum pooling, AvgPool (. cndot.) represents the average pooling, and ρ (1/s' i ) Indicates that the total generation number is 1/s' i σ (-) denotes a sigmoid function. Then the output of the SSARB is
F N ＝F N-1 +M⊙F′
Wherein, the lines indicate point-by-point multiplication. Obtaining a hyperspectral and panchromatic fusion characteristic F by stacking N scale space attention residual blocks N 。
Further, the arbitrary scale attention upsampling module specifically includes:
input fusion feature F N And intermediate scale s' i By a layer of 3 x 3 extended sub-pixel convolution with a layer of 3 x 3 convolution and then with F by bicubic interpolation N Are added to obtain F' N Then, through the spatial attention module, obtaining the feature map F with the target resolution s . The spatial attention module of this step is the same as the scale-space attention residual block.
The concrete expression is as follows:
F′ N ＝f 3*3 (ReLU(ESPC(F N )))
F s ＝M⊙F′ N
where ESPC (·) denotes extended subpixel convolution. The extended sub-pixel convolution is first performed by coordinate transformation
Figure BDA0003599207410000034
Figure BDA0003599207410000035
Characterizing the target resolution by F HR Is projected back to its corresponding low resolution feature F LR Performing a 0-interpolation operation at sub-pixel locations based on the projection results
Figure BDA0003599207410000036
And then establishing s' i Sub-pixel map at scale, and finally to the sub-pixelThe pixel map is convolved.
The concrete expression is as follows:
Figure BDA0003599207410000041
Figure BDA0003599207410000042
wherein W SP Representing a sub-pixel convolution kernel.
Further, after the hyperspectral and panchromatic fusion feature extraction and the upsampling at any resolution, the spectral dimension of the hyperspectral and panchromatic fusion feature extraction needs to be reconstructed back to the spectral dimension of the original image, and a hyperspectral image with a predicted target resolution is obtained by adopting two layers of 1 × 1 convolution, which is specifically represented as follows:
Figure BDA0003599207410000043
a system for implementing an arbitrary resolution panchromatic sharpening method, comprising:
the image acquisition module is used for reading an original hyperspectral image and synthesizing a corresponding full-color image;
the system comprises a preprocessing module, a data acquisition module, a data analysis module and a data analysis module, wherein the preprocessing module is used for preprocessing an original hyperspectral image and a corresponding panchromatic image to obtain a multi-scale image pair, and a training data set, a verification data set and a test sample are obtained;
the network construction module is used for constructing an arbitrary resolution panchromatic sharpening network based on extended sub-pixel convolution;
the training network module is used for initializing network parameters, randomly selecting any scale data in a training data set for training, and acquiring the minimum value of network loss stability, wherein the corresponding network is a trained panchromatic sharpening network with any resolution of extended sub-pixel convolution;
and the test module is used for inputting a test sample and acquiring a sharpened target resolution hyperspectral image.
The invention has the beneficial effects that:
(1) the method comprises the steps of dividing a panchromatic sharpening process of a hyperspectral image into three stages, extracting hyperspectral and panchromatic fusion features by adopting a plurality of cascaded scale space attention residual blocks in the first stage, improving the arbitrary resolution of the fusion features by adopting an arbitrary scale attention up-sampling module based on extended sub-pixel convolution in the second stage, and reconstructing the spectrum dimension of an output image back to the spectrum dimension of an original hyperspectral image by adopting spectrum reconstruction in the last stage. The model designed by the invention can continuously predict the spatial resolution of the hyperspectral image and realize panchromatic sharpening of the hyperspectral image at any resolution.
(2) The extended sub-pixel convolution of the invention can realize up-sampling with any resolution, is different from the traditional sub-pixel convolution and deconvolution which can only realize up-sampling with integer scale, and a single model can only process single or limited integer scale. The method adaptively creates the sub-pixel images according to different sharpening scales, and convolution space modes generated under the finite scale are consistent in training and testing stages, so that the single model can be generalized to any scale through the training of the finite scale, and the practicability of hyperspectral image panchromatic sharpening is greatly improved.
(3) The scale space attention residual error feature extraction module and the arbitrary scale attention up-sampling module designed by the invention both introduce scales as prior information, so that the importance of different spatial position information of the hyperspectral image under different scales is adjusted, and the panchromatic sharpening performance of the hyperspectral image with arbitrary resolution is further improved.
Drawings
FIG. 1 is a flowchart of a method for panchromatic sharpening of a hyperspectral image with any resolution based on extended subpixel convolution according to an embodiment of the invention;
FIG. 2 is a scale space attention residual block diagram of the present invention;
FIG. 3 is a block diagram of an arbitrary scale attention upsampling module of the present invention;
FIGS. 4(a) -4 (c) are exemplary diagrams of extended sub-pixel convolution one-dimensional upsampling of the present invention;
FIG. 5 is a graph of the spatial pattern of all convolutions within the intermediate scale [1.0,2.5] of an embodiment of the present invention;
FIG. 6 is a block diagram of the extended sub-pixel convolution based neural network of the present invention;
fig. 7(a) -7 (d) are performance comparisons of different sharpening algorithms at different test scales on the PaviaC dataset.
Detailed Description
The present invention will be described in further detail with reference to examples and drawings, but the present invention is not limited to these examples.
Examples
The hyperspectral image PaviaC adopted by the embodiment is from a ROSIS satellite sensor, the image originally comprises 115 wave bands, and 102 wave bands remain after 13 noise and water absorption wave bands are removed. The ratio of spatial resolution between the panchromatic image and the hyperspectral image was r 5, the size of the panchromatic image was 1000 x 700, and the size of the hyperspectral image was 200 x 140 x 102.
As shown in fig. 1, fig. 2 and fig. 3, a method for sharpening a hyperspectral image with any resolution based on extended subpixel convolution includes the following steps:
s1, reading the original hyperspectral image H e R 1000×700×102 Synthesizing the corresponding full-color image P epsilon R 1000×700 Wherein R represents the ensemble of real numbers;
s2, preprocessing the hyperspectral image and the corresponding panchromatic image, generating a multi-scale image pair according to a selected scale set S-5 {1.0,1.1, …,2.5}, performing region division on the multi-scale image pair to generate a training data set and a verification data set, and generating a test sample according to a user-defined scale {1.0,1.5,1.7,2.5,3.2,3.6 };
s3, designing an arbitrary resolution panchromatic sharpening network based on extended sub-pixel convolution, wherein the panchromatic sharpening network comprises a scale attention residual error module, an extended sub-pixel convolution module and a spectrum reconstruction module;
s4, initializing network parameters, randomly selecting any scale data of a training set for training, updating the parameters through a self-adaptive moment estimation optimization algorithm, and acquiring the optimal parameters when the network loss is stabilized at the minimum value;
s5, inputting the trained extended sub-pixel convolution network into the test sample generated in the S2, and acquiring a sharpened target resolution hyperspectral image;
specifically, in S1, the full-color image corresponding to the acquired hyperspectral image is obtained by averaging the first 83 bands included in the hyperspectral image;
specifically, the S2 includes:
s2.1, pretreatment: firstly, an intermediate scale set S' {1.0,1.1, …,2.5} of the target resolution hyperspectral image and the panchromatic image is calculated according to the selected scale set S ═ 5 {1.0,1.1, …,2.5} and the resolution ratio r ═ 5 of the panchromatic image to the low resolution hyperspectral image. Carrying out fuzzy degradation on the original hyperspectral image H and the panchromatic image P by adopting a Gaussian filter with specific frequency response for s' i The multiple down-sampling is carried out to obtain a reference hyperspectral image H during standard sharpening r ∈R m′×n′×C And a full-color image P r ∈R m′×n′ Wherein m ', n' satisfy the expression: m ═ M 'x s' i ，N＝n′×s′ i Then to H r Then r times of downsampling is carried out to obtain a low-resolution ratio hyperspectral image
Figure BDA0003599207410000061
Wherein m, n satisfy the expression: m ═ mxrxs' i ，N＝n×r×s′ i . Finally, low-resolution hyperspectral images are subjected to polynomial interpolation
Figure BDA0003599207410000062
Performing pre-interpolation with the scale r being 5 to obtain
Figure BDA0003599207410000063
Taking the original hyperspectral image as a target reference image H s Thereby obtaining a multi-scale image pair
Figure BDA0003599207410000064
s∈S。
S2.2, image area division: fromPairs of multi-scale images
Figure BDA0003599207410000065
And the size of the training area divided by the S belongs to 1000 multiplied by 200, the size of the verification area is 300 multiplied by 400, the size of the test area is 200 multiplied by 200, and the test sample is generated according to the user-defined scale {1.0,1.5,1.7,2.5,3.2,3.6 }.
Further, as shown in fig. 6, the convolution network based on extended sub-pixel convolution in S3 mainly includes a scale attention residual module, an arbitrary scale up-sampling module based on extended sub-pixel convolution, and a spectrum reconstruction module:
the scale attention residual module: for input feature diagram F N-1 And obtaining an intermediate characteristic diagram F' by adopting two-layer 3 × 3 convolution, wherein the intermediate characteristic diagram is specifically expressed as follows:
F′＝f 3*3 (ReLU(f 3*3 (F N-1 )))
then, the feature map F ' is input into a space attention module, and the feature map F ' is subjected to maximum pooling and average pooling along the channel dimension and is subjected to intermediate dimension s ' i Splicing the generated 2-dimensional matrix, then sending the spliced 2-dimensional matrix into two layers of 7 multiplied by 7 convolutions, and finally obtaining a scale space attention diagram M by using a Sigmoid function:
M＝σ(f 7*7 (ReLU(f 7*7 ([MaxPool(F′)；AvgPool(F′)；ρ(1/s′ i )]))))
wherein MaxPool (. cndot.) represents the maximum pooling, AvgPool (. cndot.) represents the average pooling, and ρ (1/s' i ) Indicates that the total generation number is 1/s' i σ (-) denotes a sigmoid function. The output of the module is:
F N ＝F N-1 +M⊙F′
wherein, the lines indicate point-by-point multiplication. The hyperspectral and panchromatic fusion feature F is obtained by stacking 6 scale space attention residual blocks N 。
The arbitrary scale attention upsampling module: input fusion feature F N And intermediate scale s' i By a layer of 3 x 3 extended sub-pixel convolution with a layer of normal 3 x 3 convolution and then with F by bicubic interpolation N Are added to obtain F' N Then, through the same scale space attention module as in step 3.2, the feature map F with the target resolution is obtained s Specifically, it is represented as:
F′ N ＝f 3*3 (ReLU(ESPC(F N )))
F s ＝M⊙F′ N
where ESPC (·) denotes extended subpixel convolution. The extended sub-pixel convolution is first performed by coordinate transformation
Figure BDA0003599207410000071
Figure BDA0003599207410000072
Characterizing the target resolution by F HR Is projected back to its corresponding low resolution feature F LR Performing 0 interpolation operation at sub-pixel positions according to projection result
Figure BDA0003599207410000073
As shown in fig. 4(a) -4 (c), 3 one-dimensional upsampling examples are obtained, and s 'is then established' i And (4) performing convolution on the sub-pixel map under the scale.
The concrete expression is as follows:
Figure BDA0003599207410000074
Figure BDA0003599207410000075
wherein W SP Representing a sub-pixel convolution kernel. The spatial pattern resulting from convolution on sub-pixel maps of different scales is shown in fig. 5. The spatial pattern in fig. 5 already contains the spatial patterns that may be generated by all scale sub-pixel maps, and therefore it can be generalized during the test phase.
A spectrum reconstruction module: after hyperspectral and panchromatic fusion feature extraction and sampling at any resolution, the spectral dimension of the hyperspectral and panchromatic fusion feature extraction is required to be reconstructed back to the spectral dimension of an original image, and a hyperspectral image with a predicted target resolution is obtained by adopting two layers of 1 × 1 convolution, and the specific expression is as follows:
Figure BDA0003599207410000076
in addition, we evaluate the sharpening result by using 4 commonly used evaluation indexes of the sharpened image, namely Cross Correlation Coefficient (CC), Spectral Angle map (Spectral Angle Mapper), integrated global Relative error (erereal Relative estimated robust estimate De Synth se, ERGAS), and multiband universal image quality index Q2 n.
Sharpening algorithms for comparison are: adaptive Schmidt orthogonal transformation (GSA), Smoothing Filter-based Intensity Modulation (SFIM), sparse bayes (bayesian), convolutional neural network based sharpening methods PNN, HyperPNN. Since these algorithms are specifically designed for standard sharpening, we add bicubic interpolation as the last layer thereafter to achieve arbitrary resolution panchromatic sharpening for comparison.
Fig. 7(a) -7 (d) are performance comparisons of different sharpening algorithms at different test scales on the PaviaC dataset. As can be seen from the figure: in the traditional algorithm, the sharpening result obtained by the sparse Bayesian algorithm is relatively good, but the spectrum angle mapping of the sparse Bayesian algorithm is increased along with the increase of the test scale, and the distortion degree is greater than that of the other two traditional algorithms; the result obtained by the algorithm provided by the embodiment obtains the optimal result on each index, and is obviously superior to the compared advanced algorithm on the spectral angle mapping index, which shows that the designed method can obtain better spectral fidelity on the panchromatic sharpening task with any resolution.
In summary, according to the extended sub-pixel convolution-based hyperspectral image panchromatic sharpening method with any resolution, the designed extended sub-pixel convolution can realize learnable upsampling with any resolution, and the problem and the challenge that a single model is generalized to any scale under finite scale training are solved through the consistency of the extended sub-pixel convolution space mode. In addition, the importance of different spatial position information is adjusted by introducing scales as prior information, so that the panchromatic sharpening performance of the hyperspectral image with any resolution is further improved.
Another embodiment of the present invention provides an arbitrary resolution panchromatic sharpening system for hyperspectral images, including:
the image acquisition module is used for reading an original hyperspectral image and synthesizing a corresponding full-color image;
the system comprises a preprocessing module, a data acquisition module, a data analysis module and a data analysis module, wherein the preprocessing module is used for preprocessing an original hyperspectral image and a corresponding panchromatic image to obtain a multi-scale image pair, and a training data set, a verification data set and a test sample are obtained;
the network construction module is used for constructing an arbitrary resolution panchromatic sharpening network based on extended sub-pixel convolution;
the training network module is used for initializing network parameters, randomly selecting any scale data in a training data set for training, and acquiring the minimum value of network loss stability, wherein the corresponding network is a trained panchromatic sharpening network with any resolution of extended sub-pixel convolution;
and the test module is used for inputting a test sample and acquiring a sharpened target resolution hyperspectral image.
When the system is executed, the panchromatic sharpening method with any resolution ratio of the hyperspectral image can be realized.
The above embodiments are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments, and any other changes, modifications, substitutions, combinations, and simplifications which do not depart from the spirit and principle of the present invention should be construed as equivalents thereof, and all such changes, modifications, substitutions, combinations, and simplifications are intended to be included in the scope of the present invention.

Claims (10)

1. An arbitrary resolution panchromatic sharpening method for hyperspectral images is characterized by comprising the following steps:
reading an original hyperspectral image H epsilon R M×N×C And synthesizing a corresponding full-color image P eR M×N Wherein M and N represent the height and width of the hyperspectral image, C represents the number of wave bands, and R represents the whole real number set;
preprocessing the original hyperspectral image and the corresponding panchromatic image, and collecting S ═ S according to the selected training scale 1 ，s 2 ，…，s k Generating a multi-scale image pair to obtain a training data set, a verification data set and a test sample;
constructing an arbitrary resolution panchromatic sharpening network based on extended sub-pixel convolution;
initializing network parameters, randomly selecting any scale data in a training data set for training, and acquiring the minimum value of network loss stability, wherein the corresponding network is a trained panchromatic sharpening network with any resolution of extended sub-pixel convolution;
and inputting a test sample, and acquiring a sharpened target resolution hyperspectral image.
2. The method for arbitrary resolution panchromatic sharpening according to claim 1, wherein the extended sub-pixel convolution based arbitrary resolution panchromatic sharpening network comprises N cascaded scale attention residual modules, an arbitrary scale attention upsampling module and a light reconstruction module.
3. The method for panchromatic sharpening with arbitrary resolution according to claim 2, wherein the scale attention residual module is specifically:
input feature map F N-1 Adopting two layers of 3 multiplied by 3 convolution to obtain a middle characteristic diagram F';
inputting the intermediate feature map F 'into a space attention module, respectively carrying out maximum pooling and average pooling along the channel dimension, and then carrying out maximum pooling and average pooling along the channel dimension with an intermediate scale s' i Splicing the generated 2-dimensional matrix;
inputting the splicing result into two layers of 7 multiplied by 7 convolutions, and finally obtaining a scale space attention diagram M by using a Sigmoid function;
obtaining a hyperspectral and panchromatic fusion characteristic F by stacking N scale attention residual modules N 。
4. The method for arbitrary-resolution panchromatic sharpening according to claim 3, wherein the arbitrary-scale attention upsampling module is specifically:
input fusion feature F N And intermediate scale s' i By a layer of 3 x 3 extended sub-pixel convolution with a layer of 3 x 3 convolution and then with F by bicubic interpolation N Are added to obtain F' N Then, through the spatial attention module, obtaining the feature map F with the target resolution s 。
5. The method for panchromatic sharpening according to any resolution ratio of claim 3, wherein the spectrum reconstruction module acquires the hyperspectral image with the predicted target resolution ratio by adopting two-layer 1 x 1 convolution.
6. The method for sharpening any-resolution panchromatic image according to claim 1, wherein the panchromatic image is obtained by averaging all visible light wavelength bands included in the corresponding hyperspectral image.
7. The method of arbitrary resolution panchromatic sharpening of claim 1, wherein the preprocessing comprises:
first, according to the selected scale set S ═ S 1 ，s 2 ，…，s k And the resolution ratio r of the panchromatic image to the low-resolution hyperspectral image, and calculating an intermediate scale set S ' ═ S ' of the target-resolution hyperspectral image and the panchromatic image ' 1 ，s′ 2 ，…s′ k Is of s' i ＝s i /r，i＝1，2，..，k；
Carrying out fuzzy degradation on the original hyperspectral image H and the panchromatic image P by adopting a Gaussian filter with specific frequency response for s' i Performing multiple down-sampling to obtain a reference hyperspectral image H during standard sharpening r ∈R m′×n′×C And a full-color image P r ∈R m ′×n′ Wherein m ', n' satisfy the expression: m ═ M 'x s' i ，N＝n′×s′ i ；
Then to H r Then r times of downsampling is carried out to obtain a low-resolution ratio hyperspectral image
Figure FDA0003599207400000021
Wherein m, n satisfy the expression: m ═ mxrxs' i ，N＝n×r×s′ i Finally, low-resolution hyperspectral image is interpolated according to polynomial
Figure FDA0003599207400000022
Performing pre-interpolation of the scale r to obtain
Figure FDA0003599207400000023
Taking the original hyperspectral image as a target reference image H s Thereby obtaining a multi-scale image pair
Figure FDA0003599207400000024
8. The method of arbitrary resolution panchromatic sharpening according to claim 7, wherein the obtaining of the training data set, the verification data set and the test sample is performed on a multi-scale image pair
Figure FDA0003599207400000025
And carrying out region division to obtain a training data set, a verification data set and a test sample.
9. The method of arbitrary resolution panchromatic sharpening according to claim 4, wherein the extended sub-pixel convolution first projects respective coordinates of a target resolution feature to a corresponding low resolution feature by coordinate transformation;
and performing 0 interpolation operation at the sub-pixel position according to the projection result to establish s' i And (4) performing convolution on the sub-pixel map under the scale.
10. A system for implementing the method of panchromatic sharpening with arbitrary resolution of any one of claims 1 to 9, comprising:
the image acquisition module is used for reading an original hyperspectral image and synthesizing a corresponding full-color image;
the system comprises a preprocessing module, a data acquisition module, a data analysis module and a data analysis module, wherein the preprocessing module is used for preprocessing an original hyperspectral image and a corresponding panchromatic image to obtain a multi-scale image pair, and a training data set, a verification data set and a test sample are obtained;
the network construction module is used for constructing an arbitrary resolution panchromatic sharpening network based on extended sub-pixel convolution;
the training network module is used for initializing network parameters, randomly selecting any scale data in a training data set for training, and acquiring the minimum value of network loss stability, wherein the corresponding network is a trained panchromatic sharpening network with any resolution of extended sub-pixel convolution;
and the test module is used for inputting a test sample and acquiring a sharpened target resolution hyperspectral image.
CN202210396467.7A 2022-04-15 2022-04-15 Method and system for panchromatic sharpening of hyperspectral image at any resolution Pending CN114820358A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210396467.7A CN114820358A (en) 2022-04-15 2022-04-15 Method and system for panchromatic sharpening of hyperspectral image at any resolution

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210396467.7A CN114820358A (en) 2022-04-15 2022-04-15 Method and system for panchromatic sharpening of hyperspectral image at any resolution

Publications (1)

Publication Number Publication Date
CN114820358A true CN114820358A (en) 2022-07-29

Family

ID=82537175

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210396467.7A Pending CN114820358A (en) 2022-04-15 2022-04-15 Method and system for panchromatic sharpening of hyperspectral image at any resolution

Country Status (1)

Country Link
CN (1) CN114820358A (en)

Similar Documents

Publication Publication Date Title
Wang et al. A comprehensive review on deep learning based remote sensing image super-resolution methods
Jiang et al. Learning spatial-spectral prior for super-resolution of hyperspectral imagery
Loncan et al. Hyperspectral pansharpening: A review
Song et al. Spatiotemporal satellite image fusion through one-pair image learning
CN114119444B (en) Multi-source remote sensing image fusion method based on deep neural network
CN109727207B (en) Hyperspectral image sharpening method based on spectrum prediction residual convolution neural network
Sara et al. Hyperspectral and multispectral image fusion techniques for high resolution applications: A review
CN110415199B (en) Multispectral remote sensing image fusion method and device based on residual learning
US8699790B2 (en) Method for pan-sharpening panchromatic and multispectral images using wavelet dictionaries
CN110544212B (en) Convolutional neural network hyperspectral image sharpening method based on hierarchical feature fusion
CN111008936B (en) Multispectral image panchromatic sharpening method
Liu et al. Research on super-resolution reconstruction of remote sensing images: A comprehensive review
Wang et al. Enhanced image prior for unsupervised remoting sensing super-resolution
Deshpande et al. SURVEY OF SUPER RESOLUTION TECHNIQUES.
Deng et al. Multiple frame splicing and degradation learning for hyperspectral imagery super-resolution
Yu et al. Two-stage image decomposition and color regulator for low-light image enhancement
Yang et al. Hierarchical spatio-spectral fusion for hyperspectral image super resolution via sparse representation and pre-trained deep model
Li et al. Hyperspectral and Panchromatic images Fusion Based on The Dual Conditional Diffusion Models
CN114820358A (en) Method and system for panchromatic sharpening of hyperspectral image at any resolution
CN110807746B (en) Hyperspectral image sharpening method based on detail embedded injection convolutional neural network
CN114638761A (en) Hyperspectral image panchromatic sharpening method, device and medium
CN114612297A (en) Hyperspectral image super-resolution reconstruction method and device
Zhang et al. Single infrared remote sensing image super-resolution via supervised deep learning
Ramya et al. An effective super resolution image reconstruction using spatial entropy based normalized deep convolutional neural network
Wang et al. Deep hyperspectral and multispectral image fusion with inter-image variability

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination